[Jfs-discussion] Re: Buffers/caching

[Dave P. <da...@be...>]

Peter Grandi wrote:

 > Ha! that probably means some RAID arrangement. I would be happy
 > to know how long a full 'jfs_fsck -f' takes on that filesystem,
 > if you have ever had to do it.

Indeed; we're using a 3ware SATA RAID card.

As for the fsck time, I haven't had to do it yet, but I'm sure it will=20
take a while.  (On some other machines with similar size filesystems=20
using reiserfs, an fsck takes upwards of 12 hours and frequently=20
exhausts 4GB of RAM.)  The images are mirrored to two different servers=20
as they're uploaded, so losing data on one is not the end of the world.

 > To me this seems a very strange concern. You have a cache that
 > is at best 0.1% (1/1000th) the size of the volume of data it is
 > supposed to cache and you are =ABconcerned=BB about the difference
 > between 1/1,000th and 1/10,000th? Uhm, a bit strange.

If the images were equal in popularity, this would be true.

In my case, the images are user-uploaded photos for a social networking=20
site, and popularity is generally a factor of how recently the image was=20
uploaded and the "connectedness" of the user who uploaded it.  Likewise,=20
the relatively tiny thumbnails are served quite frequently, and the=20
relatively huge originals are *never* served (they're kept just in case=20
we redesign the site and have to re-resize things).  My guess is that=20
0.1% of our photo storage makes up a double-digit percentage of the load=20
- cutting that by a factor of 10 makes a huge difference.

 > File system caching except for the top levels of the tree and
 > some metadata is usually pointless anyhow.

This is interesting to me and deserves further study.  Given the slow=20
seek time of big 7200RPM SATA drives, it seems like being able to serve=20
the file from memory would be a huge win.

(We're advertiser-supported - barring a surge in people willing to pay=20
$20CPM rates for banner ads, the photos aren't going to be stored on 15K=20
SCSI drives any time soon. <g>)

 > Conversely one could explore storing the thumbnails in a
 > GDBM/Berkeley DB/... hash database and probably the 50-100KB
 > range images too.

A clever idea, probably worth prototyping to get some benchmarks.  As=20
with some discussions we've had internally about using Apache's=20
mod_mem_cache to cache things instead of letting the VFS layer do it,=20
the hope is that with some proper tweaking the OS will be able to "do=20
the right thing" most of the time.

 > Good ideas, but also I would try the 'deadline' elevator and the
 > 'noop' one too. They may be more appropriate (especially 'noop').

This I will definitely try and report back to the list.



--
Dave Pifke, da...@be...
Sr. System Administrator, www.bebo.com