#6 Patch for "superservers", including article limits

[Frederick Bruckman]

The central problem that this patch solves, is that
both Giganews and Usenetserver have started returning
"503 Timeout" when we take too long, say, to flush
headers. Making 503 non-fatal, or reducing the
"idletimout" to below a minute is not a complete fix
for the most active groups, however, as there are
*always* more headers to flush every minute, so the
cycle repeats indefinitely. There are four parts to
this fix:

1) Make 503 non-fatal. (Closes bug 892418.)

2) Add a blank line before "Retrieving...". Without
that, we're clearing the "Flushing..." or GROUP reply
from the screen.

3) The main idea: only perform the time-consuming flush
once per session (per server, per group). The OVERVIEW
file will always be slightly out of sync on the server
itself, let alone the cached copy on the client, so
there really is no down side here.

4) While we're in here... permit the user to to limit
the number of articles requested and cached. This is an
oft-requested feature. (See bug 103887, RFE 1029939.)
My implemention is ngetrc-only. I don't see the point
to making it per-server, or permitting changing it on
the command line. A possible improvement would be to
make it per-group -- perhaps a user would like to pull
all the headers for a new group, and then trim it down
once he's caught up, or perhaps he'd like to make the
limit larger for a group that's carried on fewer
servers (to keep memory usage constant).

[Frederick Bruckman]

Logged In: YES
user_id=803104

With the original patch, I've since found that the article
limits are ignored for the fullxover==2 case. I have a patch
for that, but sourceforge won't let me attach any more patches.

[Frederick Bruckman]

patch for fullxover==2, too

[Frederick Bruckman]

Logged In: YES
user_id=803104

5) Flush and don't XOVER articles before (high - maxheaders)
in the fullxover==2 case, too. The "Retrieving...", goes
over 100%, but you really can't avoid that unless/until the
recommendation in the draft NNTP RFC to let LISTGROUP accept
a range becomes widely adopted.

6) Armor progress() with extra newlines before "Retrieving
articles...", and in a few more places, too.

[Matthew Mueller]

Logged In: YES
user_id=65253

Thanks for the patches. Will post more useful comment after
I've taken a closer look.

[Matthew Mueller]

Logged In: YES
user_id=65253

4,5) Committed with some modifications. (Fullxover=2 case
properly handles holes in the article number sequence and
prints correct progress messages, fixed a unsigned underflow
condition if maxheaders > high, added test cases.)

2,6) Can you give some examples of how to observe the
problem(s)? (and tell what, if anything, you have debug= in
.ngetrc)

1,3) Still need to look at.

[Frederick Bruckman]

Overwritten lines (Use Wordpad on Windows.)

[Frederick Bruckman]

Logged In: YES
user_id=803104

2,6) No debug options. I'll attach a file to show what I'm
talking about. (This is copied and pasted from a PuTTY
terminal on Windows XP, NetBSD 2.0.2 host.) Observe that the
response to the GROUP command and the "Flushing headers..."
lines are missing -- they're actually output, but they're
overwritten by the subsequent lines.

The mystery deepens... I can observe the issue locally under
"screen", and running a remote screen in an xterm, and under
"PuTTY" (and under "screen" under "PuTTY"). When I ssh from
an xterm without screen, though, the problem isn't
exhibited, and my patch inserts ugly, extra blank lines. Any
ideas?

[Frederick Bruckman]

Logged In: YES
user_id=803104

2,6) I see what the problem is, and it would only affect
terminals that are smart enough *not* to emit a linefeed on
printf("\r"). For those, terminfo platforms have "clear to
beginning of line". We need to do similarly for termcap
platforms. It looks like "clear to end of line" is the
ticket. (Beginning, end, what's the difference?) In the
patch, I made it so termcap is "preferred", because
individual NetBSD setups et. al. are likely to have ncurses
-- it's portable -- and pulling in both would make quite a
mess, but Linux systems won't have SYSV/BSD termcap.

[Frederick Bruckman]

Do SYSV/BSD termcap like Linux terminfo.

[Frederick Bruckman]

Logged In: YES
user_id=803104

4,5) Yes, thank you! (...and I see what you mean about the
underflow). One thing: I think it is not necessary to resort
to fullxover==2 handling when there no stored headers to
keep, even in the maxheaders!=-1 case. For the empty cache
case, the other way works fine, and, on usenetserver, is
about 10 times faster! Yes, the count may be slightly off,
but as you say in the man page, it's only an estimate, while
the main reason to use fullxover==2, to avoid holes in the
cache which do not exist on the server, will not be a
problem in this case.

I use fullxover==2 for Usenetserver, mainly because, if I
don't, nget will report missing articles which aren't really
missing. I'm not sure exactly what they're doing
wrong^Wdifferent, but the issue only comes into play on
incremental updates. The server is evidently optimized for
the full-blast xover: the incremental is about 10 times
slower, even with maxstreaming set to 1024! So, if the
cache is at all out of date, it's actually quicker to flush
first, with "-F", before pulling down headers. However, this
doesn't work so well after your one-line change. There's
ways to work around this, of course, but I think the old
principle that the special fullxover==2 handling isn't
really needed on a fresh, clean xover still holds.

[Frederick Bruckman]

Residual patch against current CVS

[Frederick Bruckman]

Logged In: YES
user_id=803104

Here's the essential patch against current CVS with the
termcap and committed parts factored out. Also, I've fixed
an issue with my original patch, that the no-flush handling
was not being done for the not fullxover==2 case.

The last part of the patch is the main point: there's no
need to flush every time we reconnect after a timeout. The
overview file on the server itself is never perfectly in
sync with the server's article cache, so our cached copy of
the overview file has no hope of being correct article for
article. In fact, it's likely that the few articles that
this patch prevents us from flushing may still be retrieved
by article id for some minutes. The main benefit, though, is
that it prevents "nget" from getting stuck in a loop with
large, active groups. (3)

On the 503 thing (1), as of today, Giganews is still doing
that. Without the patch, the only work-around is to set the
idletimout very short, which exacerbates the other problem.

[Matthew Mueller]

Logged In: YES
user_id=65253

Ok, I agree with your comments, I've committed the change to
not do fullxover=2 handling on initial retrieval. I'll look
at the other parts soon, I hope.

[Frederick Bruckman]

Logged In: YES
user_id=803104

Here's a new residual patch against current CVS. I;d left
out a necessary change to a header in the last patch.
(Oops.) As of just now, Giganews still stuffs a "503
timeout" error into the pipe before closing it.

[Frederick Bruckman]

Residual patch - fullxover==2 + missing header

[Frederick Bruckman]

Logged In: YES
user_id=803104

-- I made a special case for all stored headers out of date.
Without that servinfo-reset(), it had ended up ignoring
"maxheaders" (in that case, only).

-- Removed the special handling for 503 errors. Giganews
still does it, but that's easily worked around by setting
your timeout shorter than the server's.

[Frederick Bruckman]

LATEST - special case all stored headers out of date.