Menu
▾
▴
#178 One download failure invaildates a whole mirror
Package manager version: 0.23.1
Distribution version: 0.7.1.rsync
I can't decide if this is a bug or a "Bummer, that
sucks" situation.
While doing an update-all, Fink found that my usual
mirror (distfiles.sfo.ca.us.finkmirrors.net) was
missing a file (imagemagick-6.0.8). The mirror was
flagged as a failure and it went on to the next mirror
and the next, etc, until it finally found the file in
Norway. After a slow download, it went on to the next
file to download (libxml2). But it stuck with the same
slow Norway connection instead of reverting to
California. Eventually, I tired of the slow downloads
and Ctrl-C'd out to rerun. Happily, my fast mirror
returned (as expected).
Looking into the code, it looks like the Fink::Mirror
data structure is cached and the {failed} values are
preserved across file downloads.
Is it a bug or a feature that the CA mirror was flagged
as a failure for the whole session rather than just for
the single download?
A plausible superior behavior (but harder to implement)
would be to allow two download failures before the
whole mirror site is invalidated. This would require
us to keep a separate failure pool: current download
and whole session. I might be willing to write the
patch is this is deemed an improvement.
Logged In: YES
user_id=613165
Hello :)
Since the code does exactly what it should do, I,
personally, would consider this a feature. Whether it is a
misfeature or a well thought out feature is left to
discussion. Personally I agree with you, for the next
download after a failure , fink should default back to your
mirror preference, however this can create a lot of trouble.
Simply assume that your mirror choice has been offline for a
couple of day (for whatever reaosns) and thus is completely
out of sync still. This would mean that you get an invalid
retry for every package you try to download before fink goes
out and looks for a working mirror. Since I do not see any
"broken code" I have moved this to "Feature Request". You
are of _course_ welcoem to submit a patch that remedies the
current implementation.
Logged In: YES
user_id=535292
The "desired behavior" probably depends on what the most
common mode of failure is. If the usual case is that a whole
server is unreachable or massively out-of-sync, then should
decide to skip that server as soon as possible. If the usual
case is that packages need files before they have had a
chance to propagate, each server still has "most" files so
no reason to omit that whole server.
SF is documented to mirror itself within hours and may be
shortening that window as soon as their new hardware goes
live. CPAN mirrors hourly and has daily integrity checks.
dmalloc probably has better information about fink's own
mirror pool, both among themselves and making the jump from
the stated sources into the finkmirrors. Our package
descriptions are carried over a mirroring system also, so
the tarball-isn't-propagated-yet window is narrower than
just the tarball's mirroring.
OTOH, SF's pool is known to have dead servers from time to
time (and for long periods of time), finkmirrors are
volunteer sites who occasionally change configs and break
our scripts for a day or two, and various of the jillions
of CPAN and other servers come and go. And it's pretty easy
to get a no-long-existant server stored as a fink pref even
when we publish new mirror lists.
In my experience "dead server" is pretty common.
Logged In: YES
user_id=613165
Distfiles used to be mirrored every 24 hours. This time
frame should be down to about 2 hours by now. Some mirror
sites might choose to sync every hour, but this all depends
how often the Master Distfiles mirror goes through the info
files and collects all the sources that are mentioned within
them.
This indeed leaves us with two main situations.
1.) Sources which _cannot_ be mirroed for licensing reasons
2.) Servers which are dead, plain broken.
I think that 2 is not all that common, most of the time the
user picks a time window where the particular mirror he
chose _or_ the Master distfiles mirror have eithe rnot
picked up the file yet or it was not even collected yet.
You will catch an out of sync mirror, that is sure, mainly
because we do not require them to update in a fixed time
intervall. For once that would put massive load on the
Master as I cannot coordinate when every mirror updates and
secondly some mirrors are not taken as good care of as I
would sometimes wish for.
I hope this clarifies the current mirror situation a bit, we
could model this "Feature" around these facts.
Logged In: YES
user_id=535292
I don't think case #1 exists: fink does not try finkmirrors
for License:Restrictive packages (if it does, this is indeed
a bug since common sense and the code itself say this should
not be occurring:).
Logged In: YES
user_id=613165
Just to clarify this:
#1 _never_ happens. The Master Distfiles mirror checks and
makes sure that no package with such a restricted license
gets mirrored out to the slaves. However, if you have set
your fink to do MasterFirst it will _always_ fail when it
encounters an info that pulls restricted sources (usually
for the projects web-site). So maybe we can set a flag (in
that case) to override MasterFirst and pull the sources
directly since it will _NOT_ be present on any of the
finkmirrors anyways.