Possible solution to SF problem

[Martin G. <gim...@gi...>]

Hi everybody,

I might have found a rather cool way to solve the problem with
updating our database at SourceForge... we can simply grab all ~3000
METARs from NWS each half hour.

I thought this would be impossible - or rather, that it could take too
long. But I tried it this today using a program called puf which can
make parallel downloads, see http://puf.sourceforge.net/.

Using 20 parallel downloads, it took about 5 minutes to download
METARs for all stations in our stations.csv file. All the METARs
combined is just 230 Kb uncompressed. By using bzip2 we end up with a
file of just 50 Kb(!).

I could set up a cronjob at my university, which would get the METARs
and upload the result to SourceForge every 30 minutes. If would then
start a PHP script at SourceForge which would update the databases.
This would mean, that there would be data available for /every/
airport at all times - people will be able to pick a random airport
and the data will be presented right away.

But before we can do this, I think we'll have to talk to the guys at
NWS. They might not agree with us, that 3000 requests every 30 minutes
is such a good idea for their server :-)

So, I've sent an email where I explain these things - my hope is, that
they will combine the 3000 files into a single file at their server,
so that we can get the data using a single transfer.

Also, since the amount of data is so small, it wouldn't be a problem
to keep a history of METAR reports for the last month (7.5 MB
uncompressed) or even the last year as packages on SourceForge (90 MB
compressed) There's plenty of fancy thing we could use that kind of
data for...

=2D-=20
Martin Geisler                       My GnuPG Key: 0xF7F6B57B

See my homepage at http://www.gimpster.com/ for:
PHP Weather =3D> Shows the current weather on your webpage.
PHP Shell   =3D> A telnet-connection (almost :-) in a PHP page.