Re: [Flightgear-devel] FGData splitting / assembling the base package

[IA HM <fgd...@gm...>]

>>Comments?


This is a very confusing set up in my opinion James.

It was my understanding that you guys wanted to clip Aircraft out of
FGDATA, but once done that you will stop in your interesting efforts to
make FGDATA become an atomic nightmare that requires following not now
1)monolithic and gigantic Aircraft SVN repo, but also, looking forward to 3
independent repos.

The idea, as I am understanding is to put a dynamite block in FGDATA,
explode it into pieces thrown anywhere (or a SVN repo if we can make of a
better trash can), and then make massive efforts to reconstruct it.
Sometimes even at a singular level, getting the reconstruction to happen in
everyone's computer.

Or download a zip nightly! !?! because downloads are cheap [ZIP!?]

(I feel we are back to the tarballing era in FG, but zip-style!)

I definitely conclude that no matter how chaotic the current state could
be, it seems just better to keep things the same, when the star alternative
is the one outlined above.

Do we really don't have a Plan B, Plan C, and maybe some DELTA?

================================================
To directly address your entry point:

>>some discussion is needed on how different groups of people use the system

What I do, as an end=user, connected to the git is: 1) git pull simgear &&
compile, 2) git pull flightgear && compile, 3) git pull fgdata 4) fly away!

I don't know each other particular need. Mine is to update the FGDATA on a
regular basis, in the most simple and effective manner.
Having a single repo hosting FGDATA is better than not having a repo for
FGDATA.
And having a GIT superrepository for FGDATA much better than your latest
proposal of an SVN repository.

Some arguments had been thrown about what is the percentage of people that
care where things are hosted while being developed. I wonder who is the
other 1%, but I am definitely not part of it. I do not care where thing are
stored, gitorious, github, sourceforge, etc.
What I care is how the system is optimized* to facilitate easy updating*,
and in the best of the cases, optimized to facilitate a cooperative
environment.

=================================================

>>My proposed initial solution, but this is where the discussion starts:

Do you mean, everyone else has to abide to some variant of your proposed
initial solution? or do you mean alternative approaches can be welcomed?

As an outsider this is how I see it:

A) flightgear repo: source code of the FGFS. Things here are source code
required in *compiling time*

[1] Preferences and [2] Nav databases don't really belong to this world.
But if moved here, will the sourcecode change to have these included after
compiling instead of being "looked up" in some DATA folder? If not, I
really don't see the hurry to move the lightmost part of FGDATA away into
the compiling-zone.

In my opinion: [1] and [2] Belong to the FGDATA path. Hopefully a
simplified git repository .

Having the navdatas unzipped as text files make not only easier to git
them, but also to personally optimize them. To share patches with
friends... etc.
No-one will ever these databases to be close to complete. Good design
choice!

B)[3] - core binary assets: textures, fonts, sounds.

Definitely part of the base package. In my opinion these should remain in
an available FGDATA repository.

C) [4] - shared Models / Objects (a hundred MB or so)
This is a mess. For several reasons. But one of the main ones is that
Terrasync is very unstable and prone to breaking for some time now.
I believe shared models should exist in two versions: *a minimalistic base*
required for starters, (also Known as those in KSFO area), and a *updated
set*, that is fetched via terrasync. The simples way is these being
excluyent. If it is in the base, need not to exist in Terrasync. If its in
terrasync, need not to exist in the base. FG scenery download for area
updating as was previously available here:
http://www.flightgear.org/legacy-Downloads/scenery-v2.12.html
is a very nice and simple method to update required patches in the absence
of terrasync fetching. And a method that has proven by far more stable too.
Why is this not being currently maintained are the "common sense"
alternative to terrasync is just another of the flightgear developers
decisions that totally escape my understanding.


The minimalistic base shared models belonging to FGDATA repo
All other shared models (ie, not present in the base KSFO area scenery)
need not to exist in FGDATA repo, and could be obtained by terrasync of
terrain patching as above.


[5] - KSFO scenery
FGDATA repo. Off course.


[6] - AI files, especially aircraft and traffic schedules
This becoming a git submodule! It is up to the individual developer to
recognize if he/she is, or not testing and developing this.


[7] - Aircraft
This becoming git submodules in per-aircraft repo structure. Only leaving:
C172p, ufo, and the required generic, instruments, and instruments 3d as
part of the base package.
any other aircraft being alternative.


===============
A view from this perspective

1) Not needing to immediately loose FGDATA
2) what is not needed in compilation time belongs to DATA not source (think
[1] preferences and [2] navdata)
3) THERE IS A BASIC PACKAGE WITH DATA REQUIRED TO GET FG GOING. This is the
minimalistic FGDATA. Including core binary assets, shared models ON the
basic KSFO scenery (only), and a set of 5 subfolders within Aircraft.
4) There is a expanded package with Alternative/voluntary/additional/ADDON
data. This includes the set of 500+ aircraft (and more as desired), plus
AI, plus additional scenery
Aircraft and AI could exist as submodules on a purely git system
Shared model and Scenery can be fetched via 1degreex1degree patches or
Terrasynced by preferences / stability considerations.


This model will simplify reasonably the size and management of FGDATA
It will allow an easier integration of Additional / Voluntary data
It will allow an easier removal of Additional / Voluntary data
It will maintain data separated from source code (flightgear repo)
It will facilitate and foster cooperation in parts of the development that
DO NOT require source code knowledge
It will avoid downloading nightly tarballs (or more laughably zips)
It will avoid different testers to be on different update zones (depending
on the last downloaded zip)
It will prevent the atomization of FGDATA for the purpose of reducing a
reposize, when this can be achieved in better methods.
It will prevent external re-assemblies as a way to reconstruct either
remotely or locally (with several SVNs subversion repos to be followed)


=============

*A simplified FGDATA expected size:*

*AI - Submodule; It does not add size (reduction on 451 MB)*
Aircraft - Base package (only c172p Generic Instruments Instruments-3d)
[+110MB]
*Aircraft - Submodules; It does not add size (reduction on 9.5 GB)*
*Aircraft-uuic - Submodule; It does not add size (reduction on 1M)*
Airports [uncompressed nav data]  [+50MB]
Asto [+52K]
ATC [+4M]
Docs [+24M]
Effect [+1M]
Environment [+124K]
Fonts [+3M]
gui [+1.2M]
HLA [+300K]
Huds [+104K]
Input [+800K]
Lightging [+20K]
Logs [+16K]
Materials [+484K]
Models : TO BE SEPARATED only those present physically in base KSFO area,
[current +286M]
MP: [+12K]
Nasal: [+25M]
Navaids (unzipped): [+18M]
Protocol [+68K]
Scenery [+110M]
Shaders [+1M]
Sounds [+6.2M]
Textures [+471M!!!]
Timezone [+6.3M]
Translations [+450K]
Webgui [+600K]

>From the above we can see that most of the content in FGDATA has never been
a real size concern,
separating AI + Aircraft submodules, and cleaning Models (shared) and
Scenery to solely cover the base KSFO area, the only real large directory
remaining is Textures with 471M

In total, adding the bulk: Aircraft 110M + Airports 50M + Models 286M +
Nasal 25M + Navaids 18M + Scenery (base) 110M + Texture [471M] + 10M for
every other Kb sized folder and dir == *roughly 1GB REQUIRED BASE content.*



===================

So In summary

I think you guys are taking desperate measures, and I recall everyone to
consider the steps outlined by @james above, and consider both the
complexities of that approach, the rather unconventionalisms (like nightly
zips), and the price we pay by not having a way to go to comfortably obtain
the FGDATA content in a complete and direct manner.

And thus, I invite everyone to consider other alternatives, such as that I
propose above (using git sumodules to separate large chuncks of optional
data) or any other more sensible approach


Sincerely,
Israel Hernandez












On Wed, Feb 25, 2015 at 1:52 AM, James Turner <zak...@ma...> wrote:

> Hello,
>
> To move forward with restructuring the FGData repository, some discussion
> is needed on how different groups of people use the system.
>
> In the following discussion, please continuously keep in mind, the
> difference between code repositories (where files live for development
> tracking) and downloads / installers / zips, which files can be packaged
> into for use by anyone, including developers or end-users. Up until now
> these two things have been conflated and this is going to change pretty
> much regardless.
>
> ====================================
>
> Currently FGData contains various kinds of files:
>
> [1] - preferences, effects, shaders, GUI dialogs, Nasal, translations,
> joystick configurations, autopilot XML rules, HUD descriptions
>
> These files are typically small, text-based and directly tied to the
> current FG version. I.e when as a developer I make a C++ change, and these
> files aren’t in sync, stuff breaks.
>
> [2] - the navigation databases
>
> Currently these are gzipped compressed text - they’re not small and
> currently we change them infrequently but the upstream sources does receive
> updates relatively often (monthly?). Again there is some code dependency
> here, since new versions could break things.
>
> [3] - core binary assets: textures, fonts, sounds.
>
> These occupy a few hundred megabytes, but have NO code version dependency
> - the same set of these files would ‘roughly’ work with most FG releases.
> Obviously with some missing files if you used Textures from several years
> ago, but that wouldn’t break the simulator. (And we should make that a
> rule, i.e we tolerate such missing files without crashing)
>
> [4] - shared Models / Objects (a hundred MB or so)
>
> This is a subset of the Models directory which is *also* synced by
> TerraSync as soon as it’s enabled. Unfortunately it also contains some
> files which are not part of the official Models repository, due to some
> contributors not being aware of this setup. This area is particularly
> unfortunate because we want to get the live version from TerraSync, and we
> explicitly tell non-TerraSync users to manually download the models zip
> which Martin generates (since the one in fgdata is a subset!), but it’s
> confusing for non-TerraSync users and slightly wasteful for ones who do use
> it.
>
> (One solution here would be to have terrasync update the version in place,
> but the problem is the base-package files are assumed to be read-only, so
> we can’t…. unless we clone the files into your TerraSync dir before the
> initial clone … hmmmm)
>
> [5] - KSFO scenery
>
> We include a snapshot of the KSFO scenery in FGData, periodically updated
> (as Martin or others decide)
>
> [6] - AI files, especially aircraft and traffic schedules
>
> We include many AI aircraft models (but people would like more) and
> traffic descriptions, plus other smaller pieces such as scenarios, carrier
> information and more
>
> [7] - Aircraft
>
> This one is pretty well covered I think, worth pointing out that several
> sub-dirs of Aircraft are in fact shared, i.e ‘Instruments’,
> ‘Instruments-3D’.
>
> ===================
>
> My main objectives in a restructuring would be:
>
> - only store files in one place
>
> - use Jenkins to create (once a day) a 'base package’ which assembles all
> (or most) of the pieces above into a zip, and into the Mac and Windows
> installers. So even as a developer the expectation would be you’d use a
> 'base package' to get the files above, because assembling them all by hand
> might be awkward. (But you would have the choice, of course)
>
> - get the category [1] files above into either the main ‘flightgear’ repo
> or a submodule, such that we can make single Git commits which update both
> the C++ /and/ the non-code files they depend on. I.e no more forgetting to
> push an effect, Nasal file or preferences.xml change along with the code.
>
> =======================
>
> My proposed initial solution, but this is where the discussion starts:
>
> - move the category [1] files, and potentially unzipped [2] files, into
> the main flightgear repository, under, say, ‘flightgear/data’. I don’t
> think a submodule gains us anything, all the files are tiny so it won’t
> bloat the repository. (And it avoids the complexity of forgetting to update
> the submodule - but if that issue can be solved, a submodule also works for
> me)
>
> (We need to switch to unzipping the category [2] files anyway, for another
> improvement to startup performance, and once they are text rather than
> binary, Git will be able to diff them efficiently, hence my thought they
> can maybe be treated the same way. But I’m not sure. Also there is the
> option to start getting upstream updates more often, e.g. between FG
> releases)
>
> - put the category [3] files into a ‘binary assets’ repository. I was
> thinking SVN for this on SourceForge, parallel to the aircraft repo, but it
> can be a Git repo - I just suspect over time an SVN repo will be easier for
> non-coders to work with if it contains hundreds of versions of landcover
> textures at various sizes (and probably, input artwork in XCF / PSD format)
>
> - move the ‘non-Martin’ models from [4] into the same repository as [3],
> so that we can use a pure version from Martin’s system directly
>
> - move all the AI files [6] to /another/ SVN repository on SourceForge
> called fgfs-ai or similar. This is because I have code already written to
> sync AI models and traffic files over terrasync (so the repo has to be
> SVN), and this allows the download file / base package to be considerably
> smaller for people who don’t use multiplayer or traffic. When you enable
> those features we download the traffic files and models *as needed*, and we
> can add / update AI models in the SVN repo live.
>
> =======================
>
> How this works in practice:
>
> - 3 svn repos; one for aircraft (already exists), one for ‘binary assets’,
> one for AI stuff
>
> - 2 git repos, flightgear and simgear. The flightgear repo contains the
> small text files in category [1] and potentially [2]
>
> - no need for a ‘base package version check’ when developing - you can
> update the binary assets repo whenever, and the worst you should get if
> it’s out of date is a missing wav file or texture
>
> - Jenkins assembles a base package zip or tarball each night containing:
>
>   binary assets (filtering out files / possibly even running scripts to
> rescale textures / convert image formats / check PNG compression)
>   Martin’s models
>   KSFO scenery snapshot (from where, to be decided)
>   subset of Aircraft (maybe just the C172 and UFO, or maybe the current
> aircraft selection)
>   the ‘non-terra-synced’ portions of [6] (scenarios and some other awkward
> pieces)
>
> - for day to day development, you only need to update flightgear+simgear
> and nothing will break.
>
> - periodically you grab a new base package zip from jenkins, or if working
> ‘manually’, do ‘svn update’ in the binary-assets, aircraft and AI repos.
> Again if you forget one, nothing should break catastrophically. (we could
> also use svn externals to make an svn ‘supermodule’ combining these, to
> replicate the current experience of a massive repo that contains
> everything, for people who really like that….)
>
> - nightly builds *and* release installers contain the same files: some
> KSFO scenery, the shared Models, and some subset of aircraft. This is
> perhaps the most contentious point - we could make the nightlies smaller by
> assuming such users will use TerraSync or are smart enough to grab aircraft
> / shared Models / scenery some other way. (This would more or less halve
> the size of the installers, at a rough guess)
>
> However I feel downloads are cheap, and for the sake of having the
> nightlies be close to the release setup (for testing) and simplifying the
> support experience for people testing nightly builds, it’s better just to
> keep things the same.
>
> ====================
>
> Comments?
>
> Kind regards,
> James
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Flightgear-devel mailing list
> Fli...@li...
> https://lists.sourceforge.net/lists/listinfo/flightgear-devel
>