openpacket-devel — OpenPacket development list

Re: [Openpacket-devel] Bulk PCAP files upload

[Richard B. <tao...@gm...>]

On 7/28/06, David A. Belle-Isle <ml...@im...> wrote:
> Hi Jacob,
>
> Thanks for the input.
>
> I agree with you if we needed to upload a couple of files at the
> sametime but we are talking about thousands of files here. I doubt the
> user would be interested in clicking "Browse..." a thousand time! :)
>

David is right -- I have a lead on a source you might supply somewhere
around 1500 traces per month.

Perhaps once David or I can make David's demo accessible to others,
you will see the problem we are trying to solve.

Sincerely,

Richard

[Openpacket-devel] Fwd: Bulk PCAP files upload

[Tim F. <fu...@cc...>]

Resent from the correct email address (again). :P

---------- Forwarded message ----------
From: Tim Furlong <tim...@gm...>
Date: Jul 28, 2006 3:31 PM
Subject: Re: [Openpacket-devel] Bulk PCAP files upload
To: "David A. Belle-Isle" <ml...@im...>
Cc: ope...@li...

 One way you could manage that is to get the user to archive the traces,
tar.gz, RAR, whatever, and then upload the archive (possibly extracting
automatically on the server side, though that's a bit dicey).  Depending on
how the upload is done, you may want to make it a seperate page from the
single-trace upload.

-Tim

 On 7/28/06, David A. Belle-Isle <ml...@im...> wrote:
>
> Hi Jacob,
>
> Thanks for the input.
>
> I agree with you if we needed to upload a couple of files at the
> sametime but we are talking about thousands of files here. I doubt the
> user would be interested in clicking "Browse..." a thousand time! :)
>
> Thanks,
>
> David
>
>
>
> Jacob Ham wrote:
> > Either through the use of ajax, just some javascript, it could
> > automatically add another file field after they have selected a file
> > to be uploaded.  When they have finished, just add these to a file
> > table.  There would have to be many-to-many relationship to this file
> > table from the trace table.  Then it would be possible for a trace to
> > have many capture files.
> >
> > If you need me to elaborate more, please let me know.
> >
> > Jake
> >
> > On 7/28/06, David A. Belle-Isle < ml...@im...> wrote:
> >> Hi everyone,
> >>
> >> We decided to have the option to be able to upload a large number of
> >> files at the same time (bulk upload) to the trace files repository.
> >> (Additionally to the option of uploading one file at a time)
> >>
> >> The problem is that I can't figure out a way to do it! If you have an
> >> idea or if you have already done that I would like to know it and have
> >> some details of how this could be possible.
> >>
> >> So far I developped the project in python using the django web
> >> development framework.
> >>
> >> If you need any additional info let me know, I'll post what you need.
> >>
> >> Thanks a lot,
> >>
> >> David
> >>
> >>
> >>
> >>
> -------------------------------------------------------------------------
> >> Take Surveys. Earn Cash. Influence the Future of IT
> >> Join SourceForge.net's Techsay panel and you'll get the chance to
> >> share your
> >> opinions on IT & business topics through brief surveys -- and earn cash
> >>
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> >> _______________________________________________
> >> Openpacket-devel mailing list
> >> Ope...@li...
> >> https://lists.sourceforge.net/lists/listinfo/openpacket-devel
> >>
> >
>
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Openpacket-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openpacket-devel
>



-- 
Tim Furlong
tim...@gm...


-- 
Tim Furlong
tim...@gm...

Re: [Openpacket-devel] Bulk PCAP files upload

[David A. Belle-I. <ml...@im...>]

Hi Jacob,

Thanks for the input.

I agree with you if we needed to upload a couple of files at the 
sametime but we are talking about thousands of files here. I doubt the 
user would be interested in clicking "Browse..." a thousand time! :)

Thanks,

David



Jacob Ham wrote:
> Either through the use of ajax, just some javascript, it could
> automatically add another file field after they have selected a file
> to be uploaded.  When they have finished, just add these to a file
> table.  There would have to be many-to-many relationship to this file
> table from the trace table.  Then it would be possible for a trace to
> have many capture files.
> 
> If you need me to elaborate more, please let me know.
> 
> Jake
> 
> On 7/28/06, David A. Belle-Isle <ml...@im...> wrote:
>> Hi everyone,
>>
>> We decided to have the option to be able to upload a large number of
>> files at the same time (bulk upload) to the trace files repository.
>> (Additionally to the option of uploading one file at a time)
>>
>> The problem is that I can't figure out a way to do it! If you have an
>> idea or if you have already done that I would like to know it and have
>> some details of how this could be possible.
>>
>> So far I developped the project in python using the django web
>> development framework.
>>
>> If you need any additional info let me know, I'll post what you need.
>>
>> Thanks a lot,
>>
>> David
>>
>>
>>
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to 
>> share your
>> opinions on IT & business topics through brief surveys -- and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> Openpacket-devel mailing list
>> Ope...@li...
>> https://lists.sourceforge.net/lists/listinfo/openpacket-devel
>>
> 

[Openpacket-devel] Bulk PCAP files upload

[David A. Belle-I. <ml...@im...>]

Hi everyone,

We decided to have the option to be able to upload a large number of 
files at the same time (bulk upload) to the trace files repository. 
(Additionally to the option of uploading one file at a time)

The problem is that I can't figure out a way to do it! If you have an 
idea or if you have already done that I would like to know it and have 
some details of how this could be possible.

So far I developped the project in python using the django web 
development framework.

If you need any additional info let me know, I'll post what you need.

Thanks a lot,

David

Re: [Openpacket-devel] site requirements

[Tim F. <fu...@cc...>]

Not yet, but will take a closer look.  Just took a quick peek and it looks
like it may be more actively maintained than BNBT, which would be good.  I'm
not sure if the distributed tracker would be necessary for openpacket.org,
as I'm thinking redundancy is more of an issue if you're expecting to get
shut down by the authorities, but it would still be a good idea in general.
I'm going to try and get educated on the topic by people who run torrent
sites and who would hopefully have dealt with the issues before.

Thanks,
-Tim

On 7/23/06, Richard Bejtlich <tao...@gm...> wrote:
>
> On 7/23/06, Tim Furlong <fu...@cc...> wrote:
> > Let's try this again, this time from the address I registered with the
> list.
> > :p
> >
> > ---
> >
> > Hi,
> >
> > I'm also interested in pitching in; I've started by looking around at
> the
> > options for the BitTorrent tracker route.  I've been meaning to post
> what
> > I've found so far, so this is as good an opportunity as any.  There's
> two
> > parts to a standard BitTorrent site: the index and the tracker.  The
> index
> > lets users find .torrent files, and the tracker takes care of
> coordinating
> > the BitTorrent clients that are leeching and seeding the torrent.  These
>
> > parts are generally pretty independent, about the only coupling needed
> is
> > for when you want to display activity stats for the hosted
> torrents.  I've
> > been looking at trackers so far; there's not a lot out there for
> open-source
> > indexes, most sites seem to use custom ones.
> >
> > One of the more popular trackers, BNBT, is in the FreeBSD port tree,
> though
> > the entry in the tree seems to just be four small patch files.  Also, I
> > think it's based on version 8.1beta of BNBT, which has (at least one)
> known
> > vulnerability.  The patches will probably work on version 8.5, but I
> suspect
> > there's still some work that needs doing to harden it.  It's written in
> C++.
> >  The only other tracker software in the tree is trackerbt version 0.1.1
> > (haven't looked at it yet, sorta turned off by the low version number),
> and
> > bittornado (which is a client that includes tracker software).  I think
> we
> > can probably use BNBT, with some work.
> >
> > I'm also partial to MySQL, and I've heard good things about Ruby on
> Rails,
> > so I don't have any objections to that plan.  I've been meaning to rough
> up
> > a db schema for indexing traces; I can do that and submit it here for
> > discussion.
> >
> > Thanks,
> > -Tim
> >
>
> Hi Tim,
>
> Thanks for the update.  Any opinion on
>
> http://www.freshports.org/net-p2p/trackerbt/
>
> Thanks,
>
> Richard
>



-- 
Tim Furlong
tim...@gm...

Re: [Openpacket-devel] site requirements

[Richard B. <tao...@gm...>]

On 7/23/06, Tim Furlong <fu...@cc...> wrote:
> Let's try this again, this time from the address I registered with the list.
> :p
>
> ---
>
> Hi,
>
> I'm also interested in pitching in; I've started by looking around at the
> options for the BitTorrent tracker route.  I've been meaning to post what
> I've found so far, so this is as good an opportunity as any.  There's two
> parts to a standard BitTorrent site: the index and the tracker.  The index
> lets users find .torrent files, and the tracker takes care of coordinating
> the BitTorrent clients that are leeching and seeding the torrent.  These
> parts are generally pretty independent, about the only coupling needed is
> for when you want to display activity stats for the hosted torrents.  I've
> been looking at trackers so far; there's not a lot out there for open-source
> indexes, most sites seem to use custom ones.
>
> One of the more popular trackers, BNBT, is in the FreeBSD port tree, though
> the entry in the tree seems to just be four small patch files.  Also, I
> think it's based on version 8.1beta of BNBT, which has (at least one) known
> vulnerability.  The patches will probably work on version 8.5, but I suspect
> there's still some work that needs doing to harden it.  It's written in C++.
>  The only other tracker software in the tree is trackerbt version 0.1.1
> (haven't looked at it yet, sorta turned off by the low version number), and
> bittornado (which is a client that includes tracker software).  I think we
> can probably use BNBT, with some work.
>
> I'm also partial to MySQL, and I've heard good things about Ruby on Rails,
> so I don't have any objections to that plan.  I've been meaning to rough up
> a db schema for indexing traces; I can do that and submit it here for
> discussion.
>
> Thanks,
> -Tim
>

Hi Tim,

Thanks for the update.  Any opinion on

http://www.freshports.org/net-p2p/trackerbt/

Thanks,

Richard

Re: [Openpacket-devel] site requirements

[Richard B. <tao...@gm...>]

On 7/23/06, Mark Mason <mas...@gm...> wrote:
> I just finished reading the openpacket.org
> requirement doc found here...
> http://openpacket.blogspot.com/
> I would like to participate in building openpacket.org
>
> I know there are other people interested
> in developing this site using a variety of
> technologies.  I don't want to step on
> any toes.  If you can use my skills
> and code, great.  If not, no problem.
>
> I'm going to start building the site using
> the Ruby on Rails framework, MySQL and FreeBSD.
>
> I'll post my progress in a week.
>
> Mark

Hi Mark,

Nice to hear from you again.  Everyone, Mark is one of the people who
volunteered to start building a site.

We have another person who's started doing the same, using a different
approach.  (If you'd like to announce yourself, please do.)

At this point I'd prefer to let both development tracks run until we
can see an online demo, and then see which version people like.  Sound
ok?

Thank you,

Richard

[Openpacket-devel] site requirements

[Tim F. <fu...@cc...>]

Let's try this again, this time from the address I registered with the list.
:p

---

Hi,

I'm also interested in pitching in; I've started by looking around at the
options for the BitTorrent tracker route.  I've been meaning to post what
I've found so far, so this is as good an opportunity as any.  There's two
parts to a standard BitTorrent site: the index and the tracker.  The index
lets users find .torrent files, and the tracker takes care of coordinating
the BitTorrent clients that are leeching and seeding the torrent.  These
parts are generally pretty independent, about the only coupling needed is
for when you want to display activity stats for the hosted torrents.  I've
been looking at trackers so far; there's not a lot out there for open-source
indexes, most sites seem to use custom ones.

One of the more popular trackers, BNBT, is in the FreeBSD port tree, though
the entry in the tree seems to just be four small patch files.  Also, I
think it's based on version 8.1beta of BNBT, which has (at least one) known
vulnerability.  The patches will probably work on version 8.5, but I suspect
there's still some work that needs doing to harden it.  It's written in
C++.  The only other tracker software in the tree is trackerbt version
0.1.1(haven't looked at it yet, sorta turned off by the low version
number), and
bittornado (which is a client that includes tracker software).  I think we
can probably use BNBT, with some work.

I'm also partial to MySQL, and I've heard good things about Ruby on Rails,
so I don't have any objections to that plan.  I've been meaning to rough up
a db schema for indexing traces; I can do that and submit it here for
discussion.

Thanks,
-Tim


On 7/23/06, Mark Mason < mas...@gm...> wrote:
>
> I just finished reading the openpacket.org
> requirement doc found here...
> http://openpacket.blogspot.com/
> I would like to participate in building openpacket.org
>
> I know there are other people interested
> in developing this site using a variety of
> technologies.  I don't want to step on
> any toes.  If you can use my skills
> and code, great.  If not, no problem.
>
> I'm going to start building the site using
> the Ruby on Rails framework, MySQL and FreeBSD.
>
> I'll post my progress in a week.
>
> Mark
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Openpacket-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openpacket-devel
>



-- 
Tim Furlong
tim...@gm...


-- 
Tim Furlong
tim...@gm...

[Openpacket-devel] site requirements

[Mark M. <mas...@gm...>]

I just finished reading the openpacket.org
requirement doc found here...
http://openpacket.blogspot.com/
I would like to participate in building openpacket.org

I know there are other people interested
in developing this site using a variety of
technologies.  I don't want to step on
any toes.  If you can use my skills
and code, great.  If not, no problem.

I'm going to start building the site using
the Ruby on Rails framework, MySQL and FreeBSD.

I'll post my progress in a week.

Mark

Re: [Openpacket-devel] openpacket.org draft proposal

[Richard B. <tao...@gm...>]

On 7/22/06, Tim Furlong <fu...@cc...> wrote:
> Thanks for posting the proposal, it definately helps fill out the picture of
> what you're shooting for.
>
> I had some comments on the document itself:
> - 'Required features'
>   - You might want to elaborate on what you have in mind for being able to
> search packet traces; are you thinking they should just be searchable on the
> categories described later, or are you thinking of more advanced searching
> features?
>   - There are two references to RSS/Atom feeds which I think was just a
> duplication
> - 'Resources' could include volunteers:
>   - analysts and moderators
>   - web developers (even if it is based on an existing software package or
> set of packages, it'll probably require extensive customization)
>   - programmers (to build and improve tools for the manipulation and
> anonymization of traces)
>   - lawyers (to verify that all appropriate backsides are appropriately
> covered, and to advise on ownership/IP issues and user agreements)
>   - etc.
> - 'Caveats'
>   - I think you mean 'discreet' rather than 'discrete' advertisements, if
> you mean that they should be small and not prominent
>
> One possibility for searching would be a Flickr-style keyword labeling
> scheme, so that a Slammer trace could be tagged as "worm, windows, mssql,
> malware, slammer, buffer overflow, udp/1434"; alternately, a fixed-field
> scheme (port, type, OS, ...) could be used instead (less powerful, but
> easier to implement and probably easier to search for general users).
>
> Another useful feature would be the ability to associate analyses with the
> traces; this could be as simple as just using comments, or you might want to
> seperate out comments from full-blown analyses.  It would also be nice for
> users to be able to 'digg' the analyses.
>
>  I think this is an awesome idea.  I'm looking forward to seeing it take
> off. :-)
>
>
> -Tim
>

Hi Tim,

Regarding searches, for now I don't see us searching packet contents.
That would be a nice feature but I think it might be too
resource-intensive.

I agree with your other comments... let's see what we can do.

Thank you,

Richard

[Openpacket-devel] openpacket.org draft proposal

[Tim F. <fu...@cc...>]

Thanks for posting the proposal, it definately helps fill out the picture of
what you're shooting for.

I had some comments on the document itself:
- 'Required features'
  - You might want to elaborate on what you have in mind for being able to
search packet traces; are you thinking they should just be searchable on the
categories described later, or are you thinking of more advanced searching
features?
  - There are two references to RSS/Atom feeds which I think was just a
duplication
- 'Resources' could include volunteers:
  - analysts and moderators
  - web developers (even if it is based on an existing software package or
set of packages, it'll probably require extensive customization)
  - programmers (to build and improve tools for the manipulation and
anonymization of traces)
  - lawyers (to verify that all appropriate backsides are appropriately
covered, and to advise on ownership/IP issues and user agreements)
  - etc.
- 'Caveats'
  - I think you mean 'discreet' rather than 'discrete' advertisements, if
you mean that they should be small and not prominent

One possibility for searching would be a Flickr-style keyword labeling
scheme, so that a Slammer trace could be tagged as "worm, windows, mssql,
malware, slammer, buffer overflow, udp/1434"; alternately, a fixed-field
scheme (port, type, OS, ...) could be used instead (less powerful, but
easier to implement and probably easier to search for general users).

Another useful feature would be the ability to associate analyses with the
traces; this could be as simple as just using comments, or you might want to
seperate out comments from full-blown analyses.  It would also be nice for
users to be able to 'digg' the analyses.

I think this is an awesome idea.  I'm looking forward to seeing it take off.
:-)

-Tim

[Openpacket-devel] OpenPacket.org Draft Proposal posted

[Richard B. <tao...@gm...>]

Hello,

I just posted the first draft of a document describing OpenPacket.org.

http://openpacket.sourceforge.net/openpacket_req_doc_draft_21jul06.pdf

Please take a look and provide comments and suggestions.  Those of you
who want to help might want to announce yourselves on this list to
facilitate collaboration.

I did not assign any timelines because I am not sure of the level of
effort required to make this a reality.  At the very least this
document is a starting point.

Thank you,

Richard

[Openpacket-devel] First post

[Richard B. <tao...@gm...>]

First post -- test.

Richard