From: Alan T. <ajt...@v-...> - 2016-04-14 23:18:53
|
The default aircraft and scenery searches may both be overridden by the user. Could the same principle be applied to apt.dat so that new airfields can be added without making local edits to apt.dat .gz in fgdata ? Alan |
From: James T. <zak...@ma...> - 2016-04-15 08:22:33
|
> On 15 Apr 2016, at 00:18, Alan Teeder <ajt...@v-...> wrote: > > The default aircraft and scenery searches may both be overridden by the user. > > Could the same principle be applied to apt.dat so that new airfields can be added without making local edits to apt.dat .gz in fgdata ? In principle, yes. Changes are slightly higher cost - we need to rebuild the nav-cache when the primary airport data changes - but that’s not terrible. It would open up quite a range of confusing behaviours if things are misconfigured (compared to missing scenery, where you generally get an ocean tile, or aircraft, where you get the glider). In particular we have no provision to override entries in the primary apt.dat, which may be what is actually wanted. We /could/ allow add-on .dat files to either replace existing items if the ident matches, or add new ones. This would need to be done with care, and as ever has the issue that some of the apt.dat is relevant to the scenery build process. If you or anyone wanted to make some experiments, I’m happy to provide info on the relevant pieces to touch. It will need a bit of C++ hacking for sure, but only in a few files. Kind regards, James |
From: Alan T. <ajt...@v-...> - 2016-04-15 08:37:29
|
I understand that it is likely to cause confusion to MP and the like, but this is the case as with private scenery and aircraft paths anyway. The worst that can happen is that a user will harm his own setup. I am probably not the one to do anything other than very basic C++ hacking, but am willing to have a go if not suitably qualified volunteer puts his head over the parapet. Alan |
From: Florent R. <f.r...@fr...> - 2016-04-15 18:31:57
|
Hello, This is a feature I am also interested in. In fact, it is on my TODO (wish)list, just a bit far away... Trying to complete what has been said, I think the option (let's call it --apt-dat for now) should have the following properties: - the first occurrence of --apt-dat=somefile on the fgfs command line should replace the use of $FG_ROOT/apt.dat.gz by 'somefile' (not just add to, which would offer less possibilities); - further occurrences on the same command line should add an apt.dat snippet to those already specified; - airport definitions found from earlier occurrences of the option on a given command line should be overridden by those found later (so as to allow a custom scenery to override data from $FG_ROOT/apt.dat.gz); - the option should at least accept gzip-compressed and uncompressed files (not a "must", but would be really nice to have). [ concerning the last point, there are examples uses of gzopen() in SimGear: io/sg_binobj.cxx:538: if ( (fp = gzopen( file.c_str(), "rb" )) == NULL ) { io/sg_binobj.cxx:540: if ( (fp = gzopen( filegz.c_str(), "rb" )) == NULL ) { io/sg_binobj.cxx:960: if ( (fp = gzopen( file.c_str(), "wb9" )) == NULL ) { misc/zfstream.cxx:113: if ( (file = gzopen(name, char_mode)) == NULL ) { sound/readwav.cxx:268: fd = gzopen(path.c_str(), "rb"); ] At least, this is how I see it, to allow easy and clean installation of custom scenery, and to allow experimenting with newer, global apt.dat.gz files without having to overwrite the one in $FG_ROOT (which is not comfortable when using the FGData Git repo). Maybe I could give a hand if needed. -- Florent |
From: <wki...@gm...> - 2016-04-15 21:17:19
|
On 04/15/2016 02:31 PM, Florent Rougon wrote: > Hello, > > This is a feature I am also interested in. In fact, it is on my TODO > (wish)list, just a bit far away... perhaps this feature should also include its own navdata.cache file? a navdata database file name taken from the apt.dat file, perhaps?? this instead of (re)building the navdata.cache every time... -- NOTE: No off-list assistance is given without prior approval. *Please keep mailing list traffic on the list* unless private contact is specifically requested and granted. |
From: Florent R. <f.r...@fr...> - 2016-05-19 17:57:29
|
Hello, I have the first chunk of this feature working and open for comments: FG being able to process several apt.dat(.gz) files. What it does: - add a --apt-dat option that can be passed as many times as you want to tell FG to use user-specified apt.dat files. Example: fgfs ... --apt-dat=/path/to/your/normal/Airports/apt.dat.gz \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmo.dat (using the custom scenery announced at <https://forum.flightgear.org/viewtopic.php?f=5&t=28784>) - as you can guess from this example, the option transparently handles gzip-compressed as well as uncompressed files (thanks to sg_gzifstream), however you *must* give exact paths: the code will not try to read /path/to/your/normal/Airports/apt.dat in the above example, even if it exists and there is no apt.dat.gz file in the same directory (this is not the normal behavior of sg_gzifstream; I added existence checks for the exact paths given to --apt-dat, otherwise things could get quite confusing as the files' realpath() and mtime are stored in the navdb to detect when it has to be rebuilt). - airports found via later --apt-dat options will override those found via earlier ones, BUT HAVING THE SAME AIRPORT DECLARED IN SEVERAL FILES IS NOT YET SUPPORTED (sorry for the caps). This will come next. The above example is safe because VRMO does not exist in FG's current apt.dat.gz. - the mechanism is generic: with it, it will be trivial to handle input from as many fix.dat(.gz) as you want, or poi.dat(.gz), or whatever. This works by adding two new columns to the stat_cache table of the navdb: group_ (with underscore because "group" is an SQL keyword) and seq_num (sequence number). For now, the group can be either CACHED_FILE_GROUP_DFLT (0) or CACHED_FILE_GROUP_APT_DAT (1). The latter value allows to identify in the stat_cache table all files that should be combined together to (virtually) form the global apt.dat we are used to. The seq_num field in the navdb reflects the order in which these files were given via the --apt-dat options the last time the navdb was rebuilt. It will allow later files to override earlier ones when some airports are declared in several apt.dat files. Other groups such as CACHED_FILE_GROUP_FIX_DAT could be added to flightgear/src/Navaids/NavDataCache.hxx where the CachedFileGroup enum is defined. The group identified as CACHED_FILE_GROUP_DFLT should be used for all files that don't need any particular "group processing". Example obtained after the above fgfs command: % sqlite3 ~/.fgfs/navdata_2016_3.cache -- Loading resources from /home/flo/.sqliterc SQLite version 3.12.2 2016-04-18 17:30:31 Enter ".help" for usage hints. sqlite> select * from stat_cache limit 10; path group_ seq_num stamp --------------------------------------------------- ---------- ---------- ---------- /home/flo/flightgear/src/fgdata/Airports/apt.dat.gz 1 0 1463481555 /mnt/mm/flightgear-data/scenery-misc/maldives-custo 1 1 1463503415 /home/flo/flightgear/src/fgdata/Airports/metar.dat. 0 0 1427032680 /home/flo/flightgear/src/fgdata/Navaids/fix.dat.gz 0 0 1463481555 /home/flo/flightgear/src/fgdata/Navaids/nav.dat.gz 0 0 1427032682 /home/flo/flightgear/src/fgdata/Navaids/poi.dat.gz 0 0 1427032682 /home/flo/flightgear/src/fgdata/Navaids/carrier_nav 0 0 1460379335 /home/flo/flightgear/src/fgdata/Navaids/awy.dat.gz 0 0 1427032682 /home/flo/flightgear/src/fgdata/Input/Event/3dconne 0 0 1427032680 /home/flo/flightgear/src/fgdata/Input/Event/FlightL 0 0 1427032680 sqlite> - when you use the --apt-dat option, you are free to include or omit $FG_ROOT/Airports/apt.dat.gz. This allows one to very easily test newer apt.dat files from <https://gateway.x-plane.com/NOTAMs> without modifying Airports/apt.dat.gz inside the FGData Git repo/your FG installation. - if you don't pass any such option, FG will use $FG_ROOT/Airports/apt.dat.gz only, as usual. - the option can already be used to test airport layouts in developement having separate apt.dat files, as long as you make sure no airport is declared in several apt.dat files. What is *not done* (yet): - ensure proper behavior when airports appear in several apt.dat files. For this, I think a sane way would be to build an std::map whose keys are the airport identifiers[1], and values: for each airport, the corresponding sequence of lines read from the latest apt.dat file passed to --apt-dat (corresponding to the order in which the --apt-dat options were passed). Then this map could be read entry by entry and processed basically in the same way as currently done in flightgear/src/Airports/apt_loader.cxx. - devise a way to make this usable with the built-in Qt launcher. This launcher initializes or rebuilds the navdb earlier than what is done when not using the launcher, and this happens *before* command line options are processed by the FG standard mechanism: ,----[ flightgear/src/Main/main.cxx ] | int fgMainInit( int argc, char **argv ) | { | | [...] | | #if defined(HAVE_QT) | bool showLauncher = flightgear::Options::checkForArg(argc, argv, "launcher"); | // an Info.plist bundle can't define command line arguments, but it can set | // environment variables. This avoids needed a wrapper shell-script on OS-X. | showLauncher |= (::getenv("FG_LAUNCHER") != 0); | | if (showLauncher) { | flightgear::initApp(argc, argv); | if (!flightgear::runLauncherDialog()) { | return EXIT_SUCCESS; | } | } | #endif | | [...] | | configResult = flightgear::Options::sharedInstance()->processOptions(); `---- As a consequence, when run from the built-in Qt launcher, the navdb code won't see the list of SGPath instances built from the --apt-dat options. It would be possible to add special code using Options::valuesForOption() in the launcher as a remedy, but AFAIK James doesn't support any option apart from --fg-root in the launcher, so some other method would have to be used. James, what do you think would be the best way (only accessible by manually editing the file storing the QSettings? Making the apt.dat files list persistent and loaded by the launcher thanks to the autosave file? New GUI controls?..)? Apart from these two points, the code should work fine and you are welcome to test it and give feedback. If you have FlightGear's Git repo and a SourceForge account, you can get the branch with this initial implementation like this: cd <FlightGear-repo-root> git remote add -f frougon \ ssh://you...@gi.../u/frougon/flightgear-flightgear git checkout -b frougon-multi-apt-dat-input-v0 \ --track frougon/multi-apt-dat-input-v0 (or just 'git checkout multi-apt-dat-input-v0' if you don't care much about the local branch name, or even 'git checkout frougon/multi-apt-dat-input-v0' in case you don't even want to create a local branch) Remaining details I intend to handle when getting closer to the merge request: - more detailed message for the (presently last) commit; - rename NavDataCache::stampCacheFile() to something such as updateMetadataForCachedFile(), because it actually replaces *all* the metadata for the specified file present in the stat_cache table: the path, the group, the sequence number and the mtime. Note for reviewers: I have little experience with databases and SQL, maybe the small SQL changes can be improved. Regards [1] Using the terminology from <http://developer.x-plane.com/?article=rfc-apt-dat-1050-extensions>. -- Florent |
From: Alan T. <ajt...@v-...> - 2016-05-19 18:28:54
|
Florent That looks great. Thanks. I will find it easier to test once James has it in the main repo. Alan |
From: Florent R. <f.r...@fr...> - 2016-05-19 20:43:47
|
Okay, if you prefer a patch file, just say so... -- Florent |
From: Alan T. <ajt...@v-...> - 2016-05-19 21:14:58
|
Florent I have worked out the git stuff now, and have pulled what appears to be the correct branch. My understanding is the your current version does not support having the same airfield existing in different apt.dat files, so I will need to split my existing apt.dat into two (or more) so that there are no repeated or duplicate entries. Will try tomorrow, as today things were too hectic. Alan |
From: Florent R. <f.r...@fr...> - 2016-05-19 21:34:03
|
"Alan Teeder" <ajt...@v-...> wrote: > I have worked out the git stuff now, and have pulled what appears to be the > correct branch. Good to hear! > My understanding is the your current version does not support having the > same airfield existing in different apt.dat files, so I will need to split > my existing apt.dat into two (or more) so that there are no repeated or > duplicate entries. That is correct. I *currently* don't guarantee anything if you have the same airport in several apt.dat files, but dealing with this is the next thing on my agenda. > Will try tomorrow, as today things were too hectic. Okay. :-) Regards -- Florent |
From: Alan T. <ajt...@v-...> - 2016-05-19 22:40:48
|
Florent Stage 1 test OK. With my existing (patched) apt.dat all is as before. Tomorrow I will test with various combinations of apt.dat files. I have one airfield that is not in the stock apt.dat as it has been abandoned since 1973. I also have two airfields that exist in the stock apt.dat but I have remodelled and have updated apt.dat entries. My modified airfields have visually different layouts from those in the stock apt.dat so it should be simple to see if things are working. Alan |
From: Alan T. <ajt...@v-...> - 2016-05-20 13:37:51
|
Florent Stage 2 test OK also. I have added this to .fgfsrc, and stopped using James' Qt launcher for these tests. --apt-dat=C:\FlightGear\fgdata\Airports\apt.dat.gz \ --apt-dat=C:\FlightGear\MyScenery\airports\apt.dat My new and updated airfields are in C:\FlightGear\MyScenery\airports\apt.dat. Looking at menu->Location->Select Airport my new airfield, that is not in the stock apt.dat ,shows up. My two updated updated ones are duplicated . If I comment out the reference to apt.dat.gz in fgdata, I only see my new and modified airfields. So all seems as expected. Alan |
From: Alan T. <ajt...@v-...> - 2016-05-20 22:20:38
|
Florent As I understand it your current plan when there are multiple entries of an airport is to overwrite the previous one. Have you considered doing the opposite, which is to ignore future entries once an airport has already been read in?. This would be compatible with the way that multiple scenery directory paths are set up. (Myscenery_dir overides Terrasync_dir usually) Is this approach more complicated in terms of software implementation? With the system that you have built so far it is already much easier to make apt.dat changes, and to synchronise with terragear builds. Thanks. By the way I am testing on Windows 10, MSVC10. Alan |
From: Florent R. <f.r...@fr...> - 2016-05-20 23:08:06
|
Hi Alan, Thanks for your testing and feedback! "Alan Teeder" <ajt...@v-...> wrote: > As I understand it your current plan when there are multiple entries of an > airport is to overwrite the previous one. > > Have you considered doing the opposite, which is to ignore future entries > once an airport has already been read in?. This would be compatible with the > way that multiple scenery directory paths are set up. (Myscenery_dir > overides Terrasync_dir usually) That's an excellent point! It makes a lot of sense to be able to use the same order for FG_SCENERY and --apt-dat options, especially for people having a lot of custom sceneries. So, I will probably do it this way. > Is this approach more complicated in terms of software implementation? No, it's almost the same. > With the system that you have built so far it is already much easier to make > apt.dat changes, and to synchronise with terragear builds. Thanks. Glad to hear that. I thought so, which is a bit why I published the code even though it is not quite finished. > By the way I am testing on Windows 10, MSVC10. Roger, thanks. TTYS Regards -- Florent |
From: Florent R. <f.r...@fr...> - 2016-05-23 11:28:39
|
Hi Alan, I have pushed a new branch: https://sourceforge.net/u/frougon/flightgear-flightgear/ci/multi-apt-dat-input-v1/tree/ (same name as the previous one except that 'v0' has been replaced by 'v1') I had to do this in a new branch in order to have a clean history despite the semantic change for --apt-dat (now: earlier overrides later, as you suggested and contrary to what I first intended to do---for the record: this is to ensure consistency with FG_SCENERY). So, the main change compared to branch 'multi-apt-dat-input-v0' as you tried it is that 'multi-apt-dat-input-v1' supports apt.dat[.gz] files that potentially declare the same airports (and, as said: for any given airport, only the definition found via the first --apt-dat=/path/to/apt.dat[.gz] option that points to this airport is used). There are also a bunch of smaller fixes and improvements (code structure for reuse from other modules, readability...). Using a command such as: fgfs --log-class=general,navaid --log-level=info \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmt.dat \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmg.dat \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmh.dat \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmk.dat \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmm.dat \ --apt-dat=/path/to/maldives-custom-scenery/data/airports/vrmo.dat \ --apt-dat=/path/to/your/normal/fgdata/Airports/apt.dat.gz \ ... you should see messages such as: /path/to/your/normal/fgdata/Airports/apt.dat.gz: skipping airport VRMG (already defined earlier) /path/to/your/normal/fgdata/Airports/apt.dat.gz: skipping airport VRMH (already defined earlier) /path/to/your/normal/fgdata/Airports/apt.dat.gz: skipping airport VRMT (already defined earlier) /path/to/your/normal/fgdata/Airports/apt.dat.gz: skipping airport VRMK (already defined earlier) /path/to/your/normal/fgdata/Airports/apt.dat.gz: skipping airport VRMM (already defined earlier) which shows that FG indeed ignores all these airport definitions found in the standard apt.dat.gz, because they are present in the *.dat files mentioned in earlier --apt-dat options. There is no such message for VRMO, because it is a new airport that is not present in the standard apt.dat.gz. Apart from a couple of commit messages I will expand if I get confirmation that this is going to be merged, the only remaining issue I am aware of is the one I already mentioned for branch 'multi-apt-dat-input-v0': the built-in Qt launcher can't see the --apt-dat option because it initializes the navdb before options are processed with the standard FG mechanism. But before spending time on a solution to this problem, I'd rather wait for James' informed opinion on the matter... Regards -- Florent |
From: Florent R. <f.r...@fr...> - 2016-05-23 11:37:46
|
I forgot to say that this requires small SimGear fixes to compile. Wow, they are already merged, so you just need an up-to-date SimGear. Thanks to whoever did this merge! :-) https://sourceforge.net/p/flightgear/simgear/merge-requests/14/ -- Florent |
From: James T. <zak...@ma...> - 2016-05-21 07:50:04
|
> On 19 May 2016, at 18:57, Florent Rougon <f.r...@fr...> wrote: > > I have the first chunk of this feature working and open for comments: FG > being able to process several apt.dat(.gz) files. Hmm, I’m really not sure this a good direction to be proceeding in :/ I would much prefer we solve this by generating new apt.dat offline and distributing them, rather than ending up with a large number of small apt.dat files, which is not what the maintainers of the main file expect. This won’t help with custom scenery-add ons but I think that really needs to be handled through a different mechanism, or just accepting that a single centralised scenery build works much better than small custom-scenery add-ons. However, given that I’m very time constrained, and hence unlikely to work on an alternative solution, I don’t mind this going in, so long as it’s sufficiently robust. In particular we somehow need to capture the state of /all/ the .dat files used by a nav-cache, so that if any change, or the set of files change, we force a rebuild. Otherwise we’ll be fighting really inexplicable bugs. Right now it’s easy because we know there is exactly one of each type of file (apt, fix, etc), but I’m not clear from the description if this change is storing the list of file paths to compare against when determining cache validity. Kind regards, James |
From: Florent R. <f.r...@fr...> - 2016-05-21 10:53:43
|
Hi James, James Turner <zak...@ma...> wrote: > Hmm, I’m really not sure this a good direction to be proceeding in :/ > > I would much prefer we solve this by generating new apt.dat offline and > distributing them, rather than ending up with a large number of small apt.dat > files, which is not what the maintainers of the main file expect. This won’t > help with custom scenery-add ons but I think that really needs to be handled > through a different mechanism, or just accepting that a single centralised > scenery build works much better than small custom-scenery add-ons. I'm not sure I follow you very well. The small apt.dat files are what people developing/improving airport layouts naturally get from WED, therefore they can't be avoided short of centrally merging them by some FG infrastructure. But in such a case, the terrain would have to match, and we have seen getting this updated takes a loooong time (not criticizing anyone---I know it is a hard problem)... The code I posted will allow users to easily install custom sceneries that add or modify airport layouts. Currently, this requires modifying apt.dat.gz by hand for every such installation, which is too difficult for most users and a maintainance nightmare for those who know how to do it (patch-unfriendly binary format, file in Git repo). Or are you proposing a separate, for-development-only, scenery system with central, frequently updated apt.dat.gz and non-seamless joints between tiles (necessary evil for reasonably fast updates)? I fear this would add burden to the infrastructure. Less importantly maybe, if that central apt.dat.gz is updated too often, the "rebuiding navdata cache" at FG startup may be annoying. But once every day or so is probably not so much of a problem for people wanting to follow scenery development. I also thought of a variant of my proposal: FG could automatically use apt.dat or apt.dat.gz files found in some defined place inside each FG_SCENERY component (files from earlier components overriding those from later ones, as expected with FG_SCENERY). That would avoid the need to pass --apt-dat in most cases, while still loading custom sceneries in a correct way (as opposed to what happens when people naively install them without updating their apt.dat.gz). > However, given that I’m very time constrained, and hence unlikely to work on > an alternative solution, I don’t mind this going in, so long as it’s > sufficiently robust. I don't intend to make things any less robust than they currently are. See my previous apt.dat parsing changes for reference, and <https://sourceforge.net/p/flightgear/flightgear/merge-requests/42/> which is fixing a few small bugs. I also wanted to add precise error reporting with the new simgear::strutils::error_string() function, but wanted to do it for both apt.dat and fix.dat at the same time for simplicity, which can't be done before this (#42) is merged. Alternatively, I could file a new request using simgear::strutils::error_string() right from the beginning, but thought that maybe Torsten wouldn't appreciate it since he has already read this one. > In particular we somehow need to capture the state of /all/ the .dat files > used by a nav-cache, so that if any change, or the set of files change, we > force a rebuild. Otherwise we’ll be fighting really inexplicable bugs. This is exactly what I did in <https://sourceforge.net/u/frougon/flightgear-flightgear/ci/5b10dc0c4d0bb3adb6d404bab7c216021cb3ff4b/>. The code is open for review... As I explained, I introduced two fields for that in the existing 'stat_cache' table of the navdb: - the 'group_' field allows to identify cached files that belong together (apt.dat pieces, possibly fix.dat pieces, etc.); - the 'seq_num' field (sequence number) allows the navdb to remember a precise order for the various files in a given group (e.g., precedence order between the various apt.dat pieces ). Relevant excerpts brought by the above commit: flightgear/src/Navaids/NavDataCache.hxx: ---------------------------------------- /** * Groups used in the database to mark cached files that should be treated * together. * * In order to somehow merge several files, they should be declared in the * same group, with appropriate sequence numbers to define the precedence * order. For instance, partial apt.dat(.gz) files should all be put in the * CACHED_FILE_GROUP_APT_DAT group. Files that are not to be grouped in any * particular way should be put in the default group, CACHED_FILE_GROUP_DFLT * (and the sequence number doesn't matter in this case). * * If codes are changed in this enum, one should make sure that the database * is rebuilt before it is used again. */ enum CachedFileGroup { CACHED_FILE_GROUP_DFLT = 0, CACHED_FILE_GROUP_APT_DAT }; // To be kept in sync with cachedFileGroupDesc() right below static const char *cachedFileGroupDesc(enum CachedFileGroup group) { static const char *desc[] = {"default", "apt.dat"}; return desc[group]; } [...] /** * Tell whether a group of cached files should be considered modified. * * @param group group identifier * @param refFilesIt input iterator yielding SGPath instances * @param refFilesEnd iterator pointing to the past-the-end element in the * sequence * @return true if the group of cached files is considered modified * * See the definition of NavDataCachePrivate::isCachedGroupModified() for * more details. */ template<class InputIter> bool isCachedGroupModified(CachedFileGroup group, InputIter refFilesIt, InputIter refFilesEnd) const; /** * Insert or replace info about a cached file in the database. * * @param path file to insert or replace info about * @param group group identifier * @param seqNum sequence number for the file inside the group * * XXX Maybe the method should be renamed, because it does a bit more than * updating the timestamp. */ void stampCacheFile(const SGPath& path, CachedFileGroup group = CACHED_FILE_GROUP_DFLT, unsigned int seqNum = 0); (I have indeed renamed this one to updateCachedFileMetadata() in my working copy) flightgear/src/Navaids/NavDataCache.cxx: ---------------------------------------- // Tell whether a group of cached files should be considered modified. // // To be considered unmodified, the two lists of files must have the same // number of elements and, when comparing each SGPath object 'p' yielded by // the 'refFilesIt' iterator to the corresponding element from the navdata // cache (according to its 'seq_num' field), p.realpath() and p.modTime() have // to exactly match the 'path' and 'stamp' fields present in the database. // // Return true if the group of cached files is considered modified. // // In short, this means that what matters is: the number and the order of the // files in 'group', their SGPath::realpath() as well as their // SGPath::modTime(). The elements yielded by the 'refFilesIt' iterator are // typically SGPath instances corresponding to on-disk files such as apt.dat. // The notion of group here allows to merge several apt.dat files (possibly // gzip-compressed and with arbitrary paths) in a deterministic way thanks to // the sequence number recorded for each file. Other groups could be used to // merge multiple fix.dat files together, etc. template<class InputIter> bool NavDataCache::NavDataCachePrivate::isCachedGroupModified( CachedFileGroup group, InputIter refFilesIt, InputIter refFilesEnd, bool verbose) [...] bool NavDataCache::isRebuildRequired() { const PathList& apt_dat_paths = globals->get_apt_dat_paths(); if (d->readOnly) { return false; } if (flightgear::Options::sharedInstance()->isOptionSet("restore-defaults")) { SG_LOG(SG_NAVCACHE, SG_INFO, "NavCache: restore-defaults requested, will rebuild cache"); return true; } if (d->isCachedGroupModified(CACHED_FILE_GROUP_APT_DAT, apt_dat_paths.begin(), apt_dat_paths.end(), true) || d->isCachedFileModified(d->metarDatPath, true) || d->isCachedFileModified(d->navDatPath, true) || d->isCachedFileModified(d->fixDatPath, true) || d->isCachedFileModified(d->carrierDatPath, true) || [...] > Right now it’s easy because we know there is exactly one of each type > of file (apt, fix, etc), but I’m not clear from the description if > this change is storing the list of file paths to compare against when > determining cache validity. I hope it is clear now... I even posted a sample sqlite3 session exploring my ~/.fgfs/navdata_2016_3.cache in the message you are replying to, that shows the paths of the two apt.dat(.gz) files I used in the example I gave, along with their group, sequence number and timestamp. Regards -- Florent |
From: James T. <zak...@ma...> - 2016-09-20 07:05:24
|
> On 19 Sep 2016, at 20:21, Alan Teeder <ajt...@v-...> wrote: > > A very disappointing response. I am sorry you think so, so I will try to justify it: I stated my opinion, if you can convince me otherwise then go ahead. I explained why I think the current patches make the codebase worse, not better. And I offered an alternate solution to fixing the problem, which keeps things straightforward in the simulator code (usually a good idea) at the expense of some offline tooling working (usually the correct trade-off). Equally, if Torsten / Stuart / Erik / Curt were saying ‘I disagree with James, this is worth adding in’, I would reconsider the proposed patches in more detail, but if the question is being asked /of me/, you have my opinion. The code being touched is already delicate, and subjecting it to more configuration scenarios which I then have to debug and support does not make me happy. We have a workflow for scene-models that works well - people submit data (models), a central server / DB / etc collates them, and periodically new versions are available to everyone (terrasync, whether online / offline / manual). The exact same flow can work for nav data, ideally in the same place. We can augment Robin’s data (or not), maintain versions in sync with scenery (indeed, I would move the data into the terrasync file tree next to Airports/ and Models/), etc. I opposed the adding of the per-airport XML data for this reason some years ago (eight I think) for essentially the same reason and it still complicates the code, with many runtime checks to see if the apt.dat data is over-riden by a particular XML file. Whereas generating apt.dat from a DB dump is a relatively straightforward text-processing operation. What’s changed since then is we have better infrastructure to update data-assets periodically. Kind regards, James |
From: Alan T. <ajt...@v-...> - 2016-09-20 10:01:11
|
James The reason that I requested this some months ago was that I found it very cumbersome to test apt.dat changes when modifying an airfield layout. The process is to decompress apt.dat.gz, insert the changed layout (overwriting an existing one if necessary) and recompress the result. With scenery it is simple, just put the new scenery at the top of the search path and that is used in preference to the default scenery. Allowing local versions of apt.dat containing a new layout or replacing an old one is a simple task. Anyone that wishes to do this just has to put the local apt.dat files in a search path. For an airfield layout developer this would be a great aid in his work. As well as helping airport developers when making new layouts this idea also allows the possibility of having historical versions of existing or long closed airfields. For example I have almost completed a version of Brooklands as it was in the 1970´s when I worked there. My apt.dat has all of the taxiways, and my model has the motor circuit as it was then, with Vickers factory built on top of sections of it. The current version in apt.dat is very corrupt with a miniature runway and no taxiways. It would be trivial for me to modify my 1970 model and layout to recreate the pre-war grass airfield, removing the post war factory and reinstating the rest of the motor racing circuit around it. This would however require the ability of switching between two apt.dat entries for the different layouts. I have two other airfields that I have nearly complete. These are Wisley which is closed and does not exist in apt.dat, and Coimbra where apt.dat has part of the runway on an adjacent hill with a deep valley between the two sections. Waiting a year or so for the Xplane apt.dat to be updated is not a good solution. Streamlining the production of new apt.dat versions will help, but is not as flexible for the reasons I have mentioned above. OK you say that there are software problems, and you are the person best placed to quantify those. However Florent´s code works fine here. Alan |
From: James T. <zak...@ma...> - 2016-09-21 19:20:40
|
> On 20 Sep 2016, at 11:00, Alan Teeder <ajt...@v-...> wrote: > > Waiting a year or so for the Xplane apt.dat to be updated is not a good > solution. Streamlining the production of new apt.dat versions will help, but > is not as flexible for the reasons I have mentioned above. > > OK you say that there are software problems, and you are the person best > placed to quantify those. However Florent´s code works fine here. The kind of problems I expect will not exhibit themselves directly, is my concern.The current code was built on the assumption that there is one source of this data, not several. Florent’s patch takes one solution to adjusting that assumption, but I believe it does so in a way that will cause problems when scenery paths are changed underneath it, which will be very problematic to debug in the wild. In addition it’s very easy for bad input data to cause very confusing results (eg, loading runways or similar for an airport twice). I think your use scenario highlights a fundamental problem with our scenery approach, in dealing with historical periods. We just have no good way to have most of the nav data be current, but some portions reflect the world as it was in 1960 or 1980. I would prefer we maintained some global historical data sets of nav data, but in principle this would also mean maintaining historical scenery and airports, and selecting / de-selecting models based on data ranges. (So buildings appear based on their construction / demolition date). Obviously that’s a colossal amount of work which is not going to happen without effort. A more immediate option would be to extend Florent’s patch to re-build the cache more aggressively, but this impacts startup performance in those times, and further complicates the code, and doesn’t deal with the number of combinations of it that can give confusing results. Kind regards, James |
From: Alan T. <ajt...@v-...> - 2016-09-21 21:28:26
|
James Thanks for your reply. I see the historical airfields and multiple versions thereof as a bonus. The prime aim is to make airfield layout development simpler, and to allow said new layouts to be used and distributed as extras (as with the various scenery projects that we already have) whilst waiting for them to be officially published in apt.dat. The finer details of system and software design are getting over my head these days - my head is firmly locked in FORTRAN land. However I am sure others will be more on the ball. Alan |
From: Florent R. <f.r...@fr...> - 2016-09-28 14:07:52
|
Question to scenery developers: Would it be convenient enough for you if FG read one apt.dat file per scenery path (e.g., $scenery_path/Airports/apt.dat[.gz]), or is it really needed to look for individual files such as $scenery_path/Airports/I/C/A/ICAO.dat[.gz]? I am asking this because I was told in private mail that WED can save multiple airports in the same apt.dat file. Looking for only one well-defined file per scenery path such as $scenery_path/Airports/apt.dat[.gz] would certainly simplify things, if acceptable. Now about the other points: James Turner <zak...@ma...> wrote: > This worries me. I would be happy with a single apt.dat (and similar) per > valid scenery path. Recursive searches are exactly what we want to avoid, > because they potentially thrash the disk (for large trees). My concern here is > we’re pessimising both the very common case (one apt.dat) and the slightly > common case (some custom scenery installs) for a developer corner case > (working on airports in WED). The idea was to look for $scenery_path/Airports/apt.dat[.gz] and only start the recursive search if it wasn't found. > Now of course, if the number of subdirs of $scenery_paths/Airports is small, > the search won’t be /that/ expensive, but I very fearful that it will > immediately to be abused to /ship/ custom scenery without a consolidated > apt.dat, and then we’re down a horrible hole. This abuse scenario could indeed happen. I am thinking about people who develop scenery on GitHub or similar platforms. It would be nice if any commit resulted in a scenery that is ready to use by FG (i.e., apt.dat files in the proper location with names that FG would find by himself). But if it's too much of a problem, probably an acceptable compromise would be: - FG only reads $scenery_path/Airports/apt.dat[.gz] and the scenery developer makes sure to ship this file in ~stable releases of his scenery; - maybe implement a scenery developer mode as you said, where the recursive search would be active (the risk being that scenery developers could abuse this by telling users to activate the scenery developer mode in their standard installation instructions...). > My concern is the ordering isn’t really anything to do with the > stat_cache - Well, I put it there because the timestamps of apt.dat.gz, fix.dat.gz, etc. were already there. > it probably belongs somewhere else. I’d prefer to do it by storing the > concatenated list of paths (as detected) in a separate table and checking for > a string comparison there. In fact, that can be done using the existing > properties table, with some expansion of the field size I think. So, everything encoded into one big string? The ordered list of apt.dat pieces as well as the timestamp of each of these files? If instead you mean one row of the table per apt.dat piece, in order to have something reliable enough, I would use a common, very specific prefix for each key, the rest of the key giving the path of the apt.dat piece, and the value containing both the timestamp and the sequence number of this file in the ordered list of apt.dat pieces. Not sure it is really simpler, and since the existing record in the 'properties' table with key 'scenery_paths' uses ';' as the path list separator even on Linux, encoding everything concerning apt.dat pieces into a single string using this separator wouldn't cause any loss in generality. Concerning the "expansion of the field size" you mentioned, I see: CREATE TABLE properties (key VARCHAR, value VARCHAR); in flightgear/src/Navaids/CacheSchema.h, and <https://www.sqlite.org/faq.html#q9> says: ,---- | SQLite does not enforce the length of a VARCHAR. You can declare a | VARCHAR(10) and SQLite will be happy to store a 500-million character | string there. And it will keep all 500-million characters intact. Your | content is never truncated. SQLite understands the column type of | "VARCHAR(N)" to be the same as "TEXT", regardless of the value of N. `---- So, what is it that would need to be expanded? Thanks, regards -- Florent |
From: James T. <zak...@ma...> - 2016-04-16 10:16:28
|
> On 15 Apr 2016, at 22:18, wki...@gm... wrote: > > perhaps this feature should also include its own navdata.cache file? a navdata > database file name taken from the apt.dat file, perhaps?? this instead of > (re)building the navdata.cache every time... That’s not possible due to the design of the system - we can have multiple input files but they all have to end up in a single cache otherwise the query logic would get very complicated. Kind regards, James |
From: Alan T. <ajt...@v-...> - 2016-05-23 14:19:55
|
Florent Confirmed, all works as advertised. There are no longer duplicates in menu->Location.>Select Airport. Alan |