Menu ▾ ▴

#6 Threaded file retrieval

open
Matthew Mueller
None
5
2002-10-06
2002-08-29
Anonymous
No

As nget retrieves a file it could thread all the parts
at the same time, whilst taking into account a
configured max number of connections to a news server.

eg. the base Supernews ISP account allows each
connection to be 16k/sec, but you can have several of
these.

eg. A slow supernews connection for one article could
be on the go whilst other articles from faster servers
are also dealt with.

Discussion

  • Matthew Mueller

    Matthew Mueller - 2002-10-06

    Logged In: YES
    user_id=65253

    Yep, this has been on my todo awhile, but its a large change
    so I dunno when I'll get around to it.

     

  • Matthew Mueller

    Matthew Mueller - 2002-10-06
    • assigned_to: nobody --> donut
     

  • Nobody/Anonymous

    Nobody/Anonymous - 2003-01-30

    Logged In: NO

    So many windows apps can do this... (if that don't get
    you going I don't know what will :)

     

  • Nobody/Anonymous

    Nobody/Anonymous - 2003-12-25

    Logged In: NO

    Yes please!

     

  • Arjan

    Arjan - 2004-09-27

    Logged In: YES
    user_id=1118201

    Please! We -need- it!

     

  • Arthur van Dongen

    Arthur van Dongen - 2005-07-17

    Logged In: YES
    user_id=909059

    I looked at using ngetlite for this, starting multiple
    downloads using a single listfile. But ngetlite only skips
    files that are already downloaded completely, so the
    different ngetlite instances end up downloading the same files.
    Two possible workarounds:
    1. Split the listfile into separate files before the
    downloads are started. You have to know the number of
    downloads for this and there's no dynamic balancing.
    2. Modify ngetlite to skip parts that are already being
    handled. More difficult to implement than option 1.

     

  • Martijn

    Martijn - 2005-09-20

    Logged In: YES
    user_id=305106

    I'm also very interested in a multi threaded Nget version.

    I'm going to try to understand the code this month to see if
    I can think of a solution.

    What I'm thinking of now is a nget / ngetlite combination.
    1: nget message list -> database
    2: database -> (php process) -> ngetlite workers
    3 :nget assembly and pars.

    Is there anyone else busy with the multithreaded feature ?

     

  • Martijn

    Martijn - 2005-09-25

    Logged In: YES
    user_id=305106

    I experimented a bit with a simple script running multiple
    ngetlite but to be effective and to correct error handling I
    think it has to be done from nget itself.

    My current gues is multithreading from prot_nntp.cc line
    1169 with an semafore around the administration

     

Log in to post a comment.