Re: [Blocks-development] V0.16 progress

[Michael S. Jr. <bl...@ms...>]

My thought is...LRU cache entry expiry (which is what we had been using,
right) is pretty universally a good thing.  LRU flushes out the things that
haven't been used in a while, by definition.  So if all blocks are weighted
equally (like in a hard disk) LRU is pretty close to optimal.

However, blocks are not evenly weighted, because of two things:

1)  if the server's user placed a file there himself, he's probably more
interested in keeping that file available than he is in using the space to
cache other files.  So those blocks are special.

2)  if a file is composed of multiple blocks, if you expire one block you've
made the whole file unavailable.  This doesn't count if you actually have no
idea that all these blocks go together...but if you have some concept that
the blocks all form one file, then you should weight the blocks together and
expire them all at once.

So I think together this yields:

Sticky files and blocks.  Already being worked on.

Different LRU behavior?  Perhaps take a weighted average of block age vs.
block-group size -- a 500 block file that's 1 day old will have similar
weight to a 100 block file that's 5 days old.  So larger files 'gain weight'
:) faster and drop off the cache sooner.

Does this scenario fail easily?

Suppose I have a 2 GB cache, filled 1.99 GB full of sticky user-uploaded
files.

I now have 160 blocks to fill with data as it comes around.  Suppose someone
transfers a 159 block file through me, finishes the file...I advertise
it...and I recognize that file to be a whole file, not a disconnected bunch
of blocks.  No other traffic is going through me while this file is being
transferred through me.

Then suppose it's a popular file.  Lots of people start requesting it from
me...so I'm servicing at least one request for at least one part of the file
at all times.  It's pretty much impossible for that file to gain any
'discard weight' -- any file size multiplied by zero age is still zero.  So
that file is effectively locked in my cache until people stop requesting it
for even one second.

So what do I have with that one-block cache?  A complete inability to
download anything.  (Right?)  I can only route blocks for other people.  And
only one block at a time -- if someone has a 1k/sec connection to me, that
will be the only block (besides the popular file and the files I'm sharing)
I can transfer until the transfer is done.  I will also not be publishing
any new files -- I would need both the header and all of the (non-zero) data
blocks...and I can't store that in one block.

Is that graceful degradation, in those extreme circumstances?  Is it
considered graceful, considering that this one popular file is being
downloaded *constantly* multiple times.  If that's all my node is good
for...is that enough?

BTW, if this seems costly to compute...you can just start purging by LRU
when the free cache hits a low water mark, and keep purging until you hit a
high water mark.

What do you think?

--Michael Spencer
bl...@ms...

----- Original Message -----
From: "Erik Moeller" <mo...@sc...>
To: <blo...@li...>
Sent: Thursday, August 24, 2000 7:51 PM
Subject: Re: [Blocks-development] V0.16 progress


> On 24 Aug 2000, at 18:41, Blat Froop wrote:
>
> > * Files are only cached if they are < 10% of the cache size.
>
> Why?
>
> Is resuming from cache implemented?
>
> Regards,
> Erik
>
> --
> Scientific Reviewer, Freelancer, Humanist -- Berlin/Germany
> Phone: +49-30-45491008 - Web: <http://www.humanist.de/erik>
> The Origins of Peace and Violence: <http://www.violence.de>
> _______________________________________________
> Blocks-development mailing list
> Blo...@li...
> http://lists.sourceforge.net/mailman/listinfo/blocks-development