Re: [Openpacket-devel] Organize traces on file system

[Tim F. <fu...@cc...>]

Hi Jake,

I was wondering if you could clarify what you mean about the file system
handling the compression.  Did you have something specific in mind?  I'm
thinking that unless the fs is remote, any compression that it does would
incur about the same number of CPU cycles as doing it inline.  It might work
better on a multiprocessor system, but it wouldn't be too hard to have
openpacket.org handle compression in a seperate process.

One of the other ideas we've been kicking around a bit is the idea of having
the system based around BitTorrent; in which case, openpacket.org wouldn't
have to directly store many traces, probably just the ones awaiting
moderator approval.  It would mainly have to store the torrent files, so
volume of data wouldn't be as much of an issue as number of files.
Honestly, we probably wouldn't have to store them as files, we could
probably just store the contents of the torrent file in a DB and only dump
it to file long enough to send it to a user, and maybe have a cache of
frequently-requested torrents.  Anyway, those are optimizations that can be
done transparently when needed.  The other thing is that BitTorrent uses
hash values as an integral part of identifying a particular file, so
integrity checking would be built-in.

-Tim

On 8/3/06, Jacob Ham <ha...@gm...> wrote:
>
> Well, lets start out with somethings that we would like to have in a
> storage structure:
>
> - easily scalable
> - fast access
> - compression?
> - Do we want a API to directly access the files?
> - Do we want the structure to be humanly accessible?
>
> The first two are easy.  We can let the file system, and hardware
> handle all that.  We can develop Openpacket so it doesn't care if we
> use NTFS or ZFS or whatever.
>
> We could use the filesystem to compress the data, or we could compress
> the data (gzip etc). It would probably be best to have the file system
> handle the compression.  If we compress the data our selves, the cpu
> cost may be to great for the amount of traces we will be storing.
> Remember storage is cheaper than cpu time! :-)
>
> We could easily have an API to access the files.  No matter what
> framework we use, it could easily interface with the file system for
> retrieval.
>
> Do we want the structure to be humanly accessible?  I would say no.
> It would force everyone to use the API or have direct access to the
> DB.
>
> The structuring could go as follows.  Once a packet is uploaded, a
> hash/check sum can me taken of the file.  This is store in the
> database once uploaded.   From there the packet is renamed to its
> check sum, and stored on our servers.  All metadata and information
> about the capture is located in the DB on upload.  To find a capture,
> use the DB hash to access the file system and grab the file.
>
> This insures that:
> 1.  capture data can be checked against its original check sum to
> determined if anything has changed or has been damaged.
> 2.  access can be fast
> 3.  if there are 2 of the same captures uploaded, it will reference
> the same file (unless we find some collision!! haha) and won't take up
> extra space.
>
> This is just a quick sum of ideas that popped in my head.
>
> Jake
>
>
> On 8/3/06, Mark Mason <mas...@gm...> wrote:
> > How will you organize the traces on the file system?
> >
> > If you're getting thousands of traces uploaded, will you need a file
> > structure to organize the traces?
> >
> >
> -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> > opinions on IT & business topics through brief surveys -- and earn cash
> >
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Openpacket-devel mailing list
> > Ope...@li...
> > https://lists.sourceforge.net/lists/listinfo/openpacket-devel
> >
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Openpacket-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openpacket-devel
>



-- 
Tim Furlong
tim...@gm...