[Flightgear-devel] Clone the HTTP scenery data with terrasync.py

[Torsten D. <to...@t3...>]

Hi all

I am currently ignoring the lack of any python skills in my portfolio and
happily hack on the script to mirror the scenery data.

First of all: you will now find it at it's new home here:
https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/scripts/python/terrasync.py
(The old place in the c++ source subtree was a bad choice)

The script now is able to provide a complete mirror of the scenery data
also known as terrasync scenery or just terrascenery.

For the impatient, before you try it out: If you run this script for the
first time, it will download more than 80GB of data and more than 1,000,000
files in 40,000 folders to your hard disk. It will run for hours, probably
days (depending on your internet connection).

If you already have at least a partial set of scenery data from
flightgear's built-in terrasync or even a full SVN checkout, existing and
up-to-date files will be reused and are not being downloaded a second time.

How does it work.
The HTTP terrascenery server holds all the well-know files required for
scenery, organized in the also well-know directory structure (/Models,
/Objects, /Terrain and /Airports) with all their subdirectories. The
content of the /Models folder for example is here:
http://flightgear.sourceforge.net/scenery/Models/

Each folder contains a (hidden) file called .dirindex (e.g.
http://flightgear.sourceforge.net/scenery/Models/.dirindex) that describes
the content of that folder by listing it's files, subdirectories and
checksums of the files and their subdirectories' .dirindex files.

The terrasync.py script does the following (simplified):
- a) download the file at the root of the scenery-url (
http://flightgear.sourceforge.net/scenery/.dirindex)
- b) walks into any subdirectory listed within the .dirindex
- c) compares the sha1sum of listed files with the local copy and downloads
them on mismatch
- d) for listed subdirectories, calls recursively step b)

Even if all scenery files exist on the server, there are initially still
approx. 40,000 .dirindex files missing on your drive and downloading will
take hours.
To speed up the procedure once you have all .dirindex files locally, try
running terrasync.py with the --quick option. This will tell terrasync.py
to compare the computed sha1sum of every .dirindex file on your disk with
that published in the entry in the parent''s directories .dirindex. If you
have an up-to-date mirror, this will be _very_ quick as only the root
.dirindex needs to be downloaded. Subsequent updates with the --quick
option will also be very fast as only the updated files will be downloaded
and also the .dirindex files of the parent folders up to the root directory.

So far, we have just added files to your disk, what about files that have
been removed on the server? The script usually does not touch them unless
you add --remove-orphan to the command-line. By doing so, you ask
terrasync.py to remove all files (not directories) that exist in a
subdirectory but are not listed in the corresponding .dirindex file.

If you don't want to download everything into your current working
directory, add --target=/some/folder to your commandline.

This describes pretty much everything terrasync.py does. Here are some
examples how to use it:

# fetch everything from the master repository server into
/home/joe/fg/scenery, keep orphan files, (re-)download every .dirindex file
terrasync.py --target=/home/joe/fg/scenery

# just pull in changes, quick and lean for subsequent updates
terrasync.py --target=/home/joe/fg/scenery --quick

# run a mirror, only keep those files listed on the server
terrasync.py --target=/home/joe/fg/scenery --quick --remove-orphan

# use another server to pull the data, not the sourceforge master
terrasync.py --target=/home/joe/fg/scenery --url=
http://someserver.org/otherscenery

Enjoy, feedback welcome.

Torsten

P.S.:
There are some items in the backlog to improve the script:
- use pipelining for HTTP request
- add an option to sync only parts of the repository
- become fool proof and check user-given options

P.P.S:
If somebody wants to turn this into a wiki page, that would be very much
appreciated.