[Openpacket-devel] Organize traces on file system

[Jacob H. <ha...@gm...>]

Hi All,

On 8/3/06, Tim Furlong <fu...@cc...> wrote:
> Hi Jake,
>
> I was wondering if you could clarify what you mean about the file system
> handling the compression.  Did you have something specific in mind?  I'm
> thinking that unless the fs is remote, any compression that it does would
> incur about the same number of CPU cycles as doing it inline.  It might work
> better on a multiprocessor system, but it wouldn't be too hard to have
> openpacket.org handle compression in a seperate process.
>

Indeed, I had ZFS in mind for a file system.  It is extremely
expandable, fast, provides data integrity, and has low CPU usage
compression.  You can read more about it here if interested,
http://www.opensolaris.org/os/community/zfs/  .  I think it really
doesn't matter now, but if we grow to hundreds of gigs of data, it
will definitely be something to think about.

Another option would be to gather meta data once uploaded, gzip it
once, and always serve the file compressed.  The only problem with
this is if we ever decide to reference captures in line (on the site
instead of having to download and open the capture in Wireshark).  Say
if someone wants to describe a capture in detail, he could reference
lines 10-29, describe them, then move to 30-45 (assuming we had a
system in place like this).

I don't know what kind of systems we have here for use, revenue model
(advertising, donations etc?), or hosting issues.  I assume Richard is
working on this.  If we need to save bandwidth and space we could do
so in the design.

> It would mainly have to store the torrent files, so
> volume of data wouldn't be as much of an issue as number of files.
> Honestly, we probably wouldn't have to store them as files, we could
> probably just store the contents of the torrent file in a DB and only dump
> it to file long enough to send it to a user, and maybe have a cache of
> frequently-requested torrents.

If we cache the most requested ones, it will be faster but then we are
back where we are now.... How would we store the cached files?!  What
if there are 1000s of popular files we cache?

Jake