#9 Downloading Performance

[Mark]

Quoted from the bug report emailed to me:
---
It seems to effect larger newsgroups with more than
2000 messages. I joined a larger binary group to test my
bulk download script and discovered that for the first
2000 to 2200 messages Newscoaster is pretty fast at
importing the message headers from the server. After
that point it slows to an almost unbareable crawl.
Snoopdos shows that it is opening the .index file for
reading each time it saves a header. I'm not sure what
the purpose of this is, as it would seem to me that
keeping the .index file in memory for faster parsing would
be a better method. I don't know how much data is
written to this index file for each message, but as
long as it's not an absolutely huge file, I think most
people would have the system resources available to
handle it.

Loading in the headers from a newsgroup with 31,000
articles on my Amiga 4000/060 + DSL gives roughly the
following average times.

0001 to 2200 headers => 1min 40sec (22.0 per sec)
2201 to 2300 headers => 2min 45sec (1.54 per sec)
2301 to 2500 headers => 5min 55sec (1.05 per sec)
2501 to 2600 headers => 7min 40sec (0.95 per sec)
2601 to 2800 headers => 11min 25sec (0.88 per sec)

And it just keeps getting slower from there. For 2821
headers, the .index file was 1,398,724 bytes in size,
which is about 496 bytes per article. This would have
given me an end result of a .index file of 15,376,000 for
all 31,000 articles. 15MB is possibly more than some
people wish to dedicate to an index file, so storing it in
memory may not be the best solution. There should be
some way to remove this perfomance hit when dealing
with larger newsgroups. Perhaps writing the .index file as
you import the headers, but keep the message ID's in
memory for use while checking for duplicate messages
would be something to try. This could reduce the
memory overhead in this case from 15MB to around 2MB
and still keep the speed up while reading the headers.

I've done some more testing and have discovered that
this is somewhat dependant on individual servers. On my
main news server I hit this point at about 2200 message
headers, while on a secondary server I got as high as
5300 headers before it slowed down.

I did some additional testing with Thor and Microdot2 on
my Amiga for comparisons, and they don't seem to have
any problem pulling in a full listing of the same groups
from the same server at top speed.
---

[Mark]

Logged In: YES
user_id=439017

One reason the index file is written each time is in case of a
crash. If the index isn't written each time, then all the
messages would have to be downloaded again, and the set of
messages would be stored twice on disk - the first lot of these
wouldn't get deleted, unless the user deleted all messages in
the group at some point.

A compromise I intend to release is that, when downloading in
online mode, we write messages to the index only ever x
number of messages (x=8 seems to work well.. beyond that
the performance improvements are minimal). This means that if
a crash happens, the number of messages repeatedly
downloaded and stored is fairly small. This probably shouldn't
be done in offline mode, where the user might be downloading
large messages, and wouldn't want to have these
redownloaded, or taking up wasted space on disk.

Perhaps a further improvement would be to store downloaded
messages in a temporary folder, and then move them to the
proper folder when we write to the index. This way, we never
lose track of these duplicate messages if a crash happens and
have them wasting space, and the contents in this temporary
directory can safely be deleted. Not sure if moving the
messages like this would reduce performance too much,
however.