#6 Patch for "superservers", including article limits

open
None
5
2005-05-12
2005-04-23
No

The central problem that this patch solves, is that
both Giganews and Usenetserver have started returning
"503 Timeout" when we take too long, say, to flush
headers. Making 503 non-fatal, or reducing the
"idletimout" to below a minute is not a complete fix
for the most active groups, however, as there are
*always* more headers to flush every minute, so the
cycle repeats indefinitely. There are four parts to
this fix:

1) Make 503 non-fatal. (Closes bug 892418.)

2) Add a blank line before "Retrieving...". Without
that, we're clearing the "Flushing..." or GROUP reply
from the screen.

3) The main idea: only perform the time-consuming flush
once per session (per server, per group). The OVERVIEW
file will always be slightly out of sync on the server
itself, let alone the cached copy on the client, so
there really is no down side here.

4) While we're in here... permit the user to to limit
the number of articles requested and cached. This is an
oft-requested feature. (See bug 103887, RFE 1029939.)
My implemention is ngetrc-only. I don't see the point
to making it per-server, or permitting changing it on
the command line. A possible improvement would be to
make it per-group -- perhaps a user would like to pull
all the headers for a new group, and then trim it down
once he's caught up, or perhaps he'd like to make the
limit larger for a group that's carried on fewer
servers (to keep memory usage constant).

Discussion

  • Logged In: YES
    user_id=803104

    With the original patch, I've since found that the article
    limits are ignored for the fullxover==2 case. I have a patch
    for that, but sourceforge won't let me attach any more patches.

     
  • patch for fullxover==2, too

     
    Attachments
  • Logged In: YES
    user_id=803104

    5) Flush and don't XOVER articles before (high - maxheaders)
    in the fullxover==2 case, too. The "Retrieving...", goes
    over 100%, but you really can't avoid that unless/until the
    recommendation in the draft NNTP RFC to let LISTGROUP accept
    a range becomes widely adopted.

    6) Armor progress() with extra newlines before "Retrieving
    articles...", and in a few more places, too.

     
    • assigned_to: nobody --> donut
     
  • Logged In: YES
    user_id=65253

    Thanks for the patches. Will post more useful comment after
    I've taken a closer look.

     
  • Logged In: YES
    user_id=65253

    4,5) Committed with some modifications. (Fullxover=2 case
    properly handles holes in the article number sequence and
    prints correct progress messages, fixed a unsigned underflow
    condition if maxheaders > high, added test cases.)

    2,6) Can you give some examples of how to observe the
    problem(s)? (and tell what, if anything, you have debug= in
    .ngetrc)

    1,3) Still need to look at.

     
  • Overwritten lines (Use Wordpad on Windows.)

     
    Attachments
  • Logged In: YES
    user_id=803104

    2,6) No debug options. I'll attach a file to show what I'm
    talking about. (This is copied and pasted from a PuTTY
    terminal on Windows XP, NetBSD 2.0.2 host.) Observe that the
    response to the GROUP command and the "Flushing headers..."
    lines are missing -- they're actually output, but they're
    overwritten by the subsequent lines.

    The mystery deepens... I can observe the issue locally under
    "screen", and running a remote screen in an xterm, and under
    "PuTTY" (and under "screen" under "PuTTY"). When I ssh from
    an xterm without screen, though, the problem isn't
    exhibited, and my patch inserts ugly, extra blank lines. Any
    ideas?

     
  • Logged In: YES
    user_id=803104

    2,6) I see what the problem is, and it would only affect
    terminals that are smart enough *not* to emit a linefeed on
    printf("\r"). For those, terminfo platforms have "clear to
    beginning of line". We need to do similarly for termcap
    platforms. It looks like "clear to end of line" is the
    ticket. (Beginning, end, what's the difference?) In the
    patch, I made it so termcap is "preferred", because
    individual NetBSD setups et. al. are likely to have ncurses
    -- it's portable -- and pulling in both would make quite a
    mess, but Linux systems won't have SYSV/BSD termcap.

     
  • Do SYSV/BSD termcap like Linux terminfo.

     
    Attachments
  • Logged In: YES
    user_id=803104

    4,5) Yes, thank you! (...and I see what you mean about the
    underflow). One thing: I think it is not necessary to resort
    to fullxover==2 handling when there no stored headers to
    keep, even in the maxheaders!=-1 case. For the empty cache
    case, the other way works fine, and, on usenetserver, is
    about 10 times faster! Yes, the count may be slightly off,
    but as you say in the man page, it's only an estimate, while
    the main reason to use fullxover==2, to avoid holes in the
    cache which do not exist on the server, will not be a
    problem in this case.

    I use fullxover==2 for Usenetserver, mainly because, if I
    don't, nget will report missing articles which aren't really
    missing. I'm not sure exactly what they're doing
    wrong^Wdifferent, but the issue only comes into play on
    incremental updates. The server is evidently optimized for
    the full-blast xover: the incremental is about 10 times
    slower, even with maxstreaming set to 1024! So, if the
    cache is at all out of date, it's actually quicker to flush
    first, with "-F", before pulling down headers. However, this
    doesn't work so well after your one-line change. There's
    ways to work around this, of course, but I think the old
    principle that the special fullxover==2 handling isn't
    really needed on a fresh, clean xover still holds.

     
  • Residual patch against current CVS

     
  • Logged In: YES
    user_id=803104

    Here's the essential patch against current CVS with the
    termcap and committed parts factored out. Also, I've fixed
    an issue with my original patch, that the no-flush handling
    was not being done for the not fullxover==2 case.

    The last part of the patch is the main point: there's no
    need to flush every time we reconnect after a timeout. The
    overview file on the server itself is never perfectly in
    sync with the server's article cache, so our cached copy of
    the overview file has no hope of being correct article for
    article. In fact, it's likely that the few articles that
    this patch prevents us from flushing may still be retrieved
    by article id for some minutes. The main benefit, though, is
    that it prevents "nget" from getting stuck in a loop with
    large, active groups. (3)

    On the 503 thing (1), as of today, Giganews is still doing
    that. Without the patch, the only work-around is to set the
    idletimout very short, which exacerbates the other problem.

     
  • Logged In: YES
    user_id=65253

    Ok, I agree with your comments, I've committed the change to
    not do fullxover=2 handling on initial retrieval. I'll look
    at the other parts soon, I hope.

     
  • Logged In: YES
    user_id=803104

    Here's a new residual patch against current CVS. I;d left
    out a necessary change to a header in the last patch.
    (Oops.) As of just now, Giganews still stuffs a "503
    timeout" error into the pipe before closing it.

     
  • Residual patch - fullxover==2 + missing header

     
  • Logged In: YES
    user_id=803104

    -- I made a special case for all stored headers out of date.
    Without that servinfo-reset(), it had ended up ignoring
    "maxheaders" (in that case, only).

    -- Removed the special handling for 503 errors. Giganews
    still does it, but that's easily worked around by setting
    your timeout shorter than the server's.

     
  • LATEST - special case all stored headers out of date.