Re: [Openpacket-devel] Organize traces on file system

[Richard B. <tao...@gm...>]

On 8/3/06, Tim Furlong <fu...@cc...> wrote:
> Hi Jake,
>
> I was wondering if you could clarify what you mean about the file system
> handling the compression.  Did you have something specific in mind?  I'm
> thinking that unless the fs is remote, any compression that it does would
> incur about the same number of CPU cycles as doing it inline.  It might work
> better on a multiprocessor system, but it wouldn't be too hard to have
> openpacket.org handle compression in a seperate process.
>
> One of the other ideas we've been kicking around a bit is the idea of having
> the system based around BitTorrent; in which case, openpacket.org wouldn't
> have to directly store many traces, probably just the ones awaiting
> moderator approval.  It would mainly have to store the torrent files, so
> volume of data wouldn't be as much of an issue as number of files.
> Honestly, we probably wouldn't have to store them as files, we could
> probably just store the contents of the torrent file in a DB and only dump
> it to file long enough to send it to a user, and maybe have a cache of
> frequently-requested torrents.  Anyway, those are optimizations that can be
> done transparently when needed.  The other thing is that BitTorrent uses
> hash values as an integral part of identifying a particular file, so
> integrity checking would be built-in.
>
> -Tim

Hi all,

I think we should consider BitTorrent for distributing the entire
trace collection, or perhaps large portions of it.  OpenPacket.org
should always be the "seed of last resort" for these files -- we
shouldn't hope that others can seed it.

I expect the vast majority of traces to be small, so we should serve
those without requiring BitTorrent.  I personally wouldn't want to
launch a BT client every time I want a small exploit trace or what
have you.

Thank you,

Richard