Thread: SV: [Secureideas-base-devel] Ideas
Brought to you by:
secureideas,
sinukas
From: Christian S. <Chr...@ti...> - 2005-01-04 06:31:13
|
Hi Now it sounds like we are getting somewhere the last month(s) the = project been at a standstill and i hope we can get it moving fast again = and in the right direction. In my last mail I address the performance = issue but did not get any respond to that so please have a look at it = and any comments are welcome. Chris just drop me a line when you want = that stuff test I will setup a larger database (1 million + alerts) on = my test system and use that as reference.=20 "I'd like to see how fast the app will run if we take the part of the=20 code that moves the records into the database off the webapp" Is this = the part when It puts the cached alerts and moved those in to the DB or = ? "IMO we already have a list open to the users, and the users who are=20 genuinely interested in the development portion appear to have hooked=20 up on this team." I agree totally and since I personally don't like this = hole mailing list way of communicate I read the users list directly on = SF.=20 And Kevin yes i managed to take em down ;) Regards /Christian -----Ursprungligt meddelande----- Fr=E5n: sec...@li... = [mailto:sec...@li...] F=F6r Kevin = Johnson Skickat: den 4 januari 2005 06:07 Till: BASE Developers =C4mne: Re: [Secureideas-base-devel] Ideas On Mon, 2005-01-03 at 14:08, Chris Shepherd wrote: > Kevin Johnson wrote: > > There are a number of reasons for this. The main one being I do not = > > think the current code base is capable of handling the majority of=20 > > features that we would like to include as we move forward to an=20 > > enterprise level security analysis tool. Every time we try to add a = feature or fix a bug, we run into limitations to the way that ACID was = designed. This has proven itself during the performance discussions and = work. >=20 > Over the holidays I gave our performance issue some thought, did a bit = > of looking at the code, and I came up with an idea. For future=20 > endeavours -- and something I'd like to test out on the current=20 > codebase, perhaps before we start working on the entirely new version = -- > I'd like to see how fast the app will run if we take the part of the = > code that moves the records into the database off the webapp and say,=20 > toss in a Perl or PHP script that people can run via cron to handle=20 > copying the records into the BASE format and tables. I agree that the cache seems to be a huge part of the performance issue. = If your idea works, maybe we could give people the option in the = base_conf file to turn on cache in the php or use the perl. > It would seem that the majority of the issues we are having=20 > performance wise are due to this very process. We could probably=20 > vastly speed the application up by farming out this task to a Perl=20 > script, command line PHP script, or even a lightweight piece of C. My=20 > preference is to keep it as an interpreted language. For that a perl script is probably best. > I think we should give this a run in the next week or so if that's=20 > possible, and since I've mentioned the idea, I'll work on getting such = > a script written and added to CVS. I'll also need Christian to test it = > out, and anyone who has access to a really large snort log. >=20 I have a semi-large database that I could run it against. Or I could = send you a dump of it for you to use. > > To further this, I am going to create a base project in CVS. This = will replace the base-php4 project that we have been using so far.=20 > > I think we should follow the same type of directory structure and so = I will try to populate it today and tomorrow since I am off work.=20 > > I will be building a /doc directory where we can keep the design = docs and any wish list/TODO that we have. >=20 > I'd also like top propose a directory structure similar to what you=20 > might see in GNU apps (/ is the BASE root dir, ^ =3D dir above in = list): > /admin - directory for the admin scripts > /bin - where we'd stick any command-line binaries we want > For this dir, we would want to prevent access from > the web. > /contrib - user/developer contributed extras > /doc - documentation (in HTML format?) > /include - BASE includes > /templates - where we put the templates (maybe call it themes?) > /^/images - images related to templates > /^/pages - html code related to templates (might call it html) >=20 I was thinking we should also add /scripts and /js. The first for = command line programs and the second for any client side scripting. > > I am also thinking that we could open the developers list to the = public to get more feedback from our users, but I am not sure of this. > > It seems to work for Ethereal. >=20 > IMO we already have a list open to the users, and the users who are=20 > genuinely interested in the development portion appear to have hooked=20 > up on this team. :) I believe you are correct. One question is, has everyone of you guys = signed up for the user list? =20 >=20 > > Chris S, you had a number of great ideas when we started on how to=20 > > design the system and you had also proposed changing the database=20 > > schema. Well, go for it<grin> My idea is to continue to run in the = > > snort database but we can do it how ever we think is best. I would = also like to build the entire thing using PEAR::DB to start. We can = also use the PEAR::Graph stuff if you think it is ready. >=20 > I'm all for it! :) I'll check out the PEAR::Image_Graph stuff and get=20 > back to you on its readiness. Cool.... > I think we should definitely collaborate and develop a sane design=20 > document. This should include input from everyone, and would have to=20 > dictate the structure of the app (no arguing with it later on unless=20 > it makes absolute sense to). I agree and will put the start of a doc into CVS for everyone to add = too... does that work? I also think that having a design will help keep = us on track. > I think I can safely refer to my previous posts about features, since=20 > we haven't managed to add a large portion of them. :) Sorry..... They have all been good ideas.<g> > One thing I think we should look at is allowing the BASE DB to be=20 > housed on a separate server than the snort DB. Will that just put us in the same position with the cache issue? Or = would we move forward with the script idea? =20 One thing to remember is that a lot of our users(at least the ones that = ask questions) are not that technical when it comes to setting up = programs and a lot run on windows where cron doesn't exist the same way = it does for most of us.( I know you can schedule jobs <g> ) Thanks for the feedback and I think that works.... Kevin |
From: Chris S. <ch...@co...> - 2005-01-04 08:52:46
|
Christian Svensson wrote: > Now it sounds like we are getting somewhere the last month(s) the project been at a standstill and i hope we can get it moving fast again and in the right direction. In my last mail I address the performance issue but did not get any respond to that so please have a look at it and any comments are welcome. You didn't get any responses because by and large your statements were what we were probably all thinking. :) > Chris just drop me a line when you want that stuff test I will setup a larger database (1 million + alerts) on my test system and use that as reference. Cool > "I'd like to see how fast the app will run if we take the part of the > code that moves the records into the database off the webapp" > Is this the part when It puts the cached alerts and moved those in to the DB or ? To be ultimately clear, I'm talking about the process of pulling the records out of the snort logging tables, and dumping them into the BASE tables. >>Kevin Johnson wrote: > For that a perl script is probably best. My only thinking with the PHP script idea is that it way simplifies grabbing the current configuration and so forth. We can simply include as if it was a web page, but run as though it were command line. > I have a semi-large database that I could run it against. Or I could send you a dump of it for you to use. Getting everyone to test it out is probably a better idea. :) > I was thinking we should also add /scripts and /js. The first for command line programs and the second for any client side scripting. Scripts is a better name than bin, /js is good, and one we missed -- /db. We could stick the SQL scripts to create the tables/databases in there. > I believe you are correct. One question is, has everyone of you guys signed up for the user list? I have. > I agree and will put the start of a doc into CVS for everyone to add too... does that work? I also think that having a design will help keep us on track. I'll check it out later as it's rapidly approaching 4 AM. ;) > Will that just put us in the same position with the cache issue? Or would we move forward with the script idea? Conceptually, they could be independent ideas. My thought process with this is that people may want to back up one database without backing up some needless data (ie: back up the BASE DB but not the Snort log DB, for instance). -- Chris Shepherd Wise man say, chemist who falls into acid, absorbed in his work. |
From: Kevin J. <kjo...@se...> - 2005-01-04 23:45:00
|
On Tue, 2005-01-04 at 03:53, Chris Shepherd wrote: > >>Kevin Johnson wrote: > > For that a perl script is probably best. > My only thinking with the PHP script idea is that it way simplifies=20 > grabbing the current configuration and so forth. We can simply include=20 > as if it was a web page, but run as though it were command line. >=20 That makes sense and is probably a better idea... Does PHP on windows support command line execution? > > I have a semi-large database that I could run it against. Or I could s= end you a dump of it for you to use. > Getting everyone to test it out is probably a better idea. :) >=20 Usually... > > I was thinking we should also add /scripts and /js. The first for comm= and line programs and the second for any client side scripting. > Scripts is a better name than bin, /js is good, and one we missed --=20 > /db. We could stick the SQL scripts to create the tables/databases in the= re. >=20 > > I believe you are correct. One question is, has everyone of you guys s= igned up for the user list? =20 > I have. >=20 > > I agree and will put the start of a doc into CVS for everyone to add to= o... does that work? I also think that having a design will help keep us o= n track. > I'll check it out later as it's rapidly approaching 4 AM. ;) >=20 I didn't get to it last night so hopefully I will tonight.... sorry. > > Will that just put us in the same position with the cache issue? Or wo= uld we move forward with the script idea? =20 > Conceptually, they could be independent ideas. My thought process with=20 > this is that people may want to back up one database without backing up=20 > some needless data (ie: back up the BASE DB but not the Snort log DB,=20 > for instance). This appear to be the right direction for this... let us know.... Kevin p.s. anyone else? |
From: Tim R. <TAR...@in...> - 2005-01-05 01:21:56
|
Kevin Johnson wrote: >On Tue, 2005-01-04 at 03:53, Chris Shepherd wrote: > > >>>>Kevin Johnson wrote: >>>> >>>> >>>For that a perl script is probably best. >>> >>> >>My only thinking with the PHP script idea is that it way simplifies >>grabbing the current configuration and so forth. We can simply include >>as if it was a web page, but run as though it were command line. >> >> >> > >That makes sense and is probably a better idea... Does PHP on windows >support command line execution? > > > Yes but I dont believe it's installed (by default) anywhere in the Windows $PATH so this may be something that we should point out to people unless we give them the option to specify the path to their php executable. This might be one of those config options that people get fed up with though. Tim |
From: Chris S. <ch...@co...> - 2005-01-07 04:15:38
|
Hi all, Sorry about the delay in getting that script together, I've had a migraine for a couple days and have just generally been feeling under the weather -- which is easy since where I am just got another 1/2 foot of snow this afternoon. ;) I hope to have something whipped up sometime before I start school on Monday, but if anyone else thinks they could do it faster feel free. :) -- Chris Shepherd Wise man say, chemist who falls into acid, absorbed in his work. |
From: Kevin J. <kjo...@se...> - 2005-01-04 23:42:48
|
On Tue, 2005-01-04 at 01:30, Christian Svensson wrote: > Hi >=20 > Now it sounds like we are getting somewhere the last month(s) the project= been at a standstill and i hope we can get it moving=20 > fast again and in the right direction. In my last mail I address the perf= ormance issue but did not get any respond to that so please > have a look at it and any comments are welcome. Chris just drop me a lin= e when you want that stuff test I will setup a larger database=20 > (1 million + alerts) on my test system and use that as reference.=20 >=20 I apologize for not responding. I agree that people need to tune their alerts, but we can't make them, and in some cases they can't. So we should design something that will handle the load. At least in my opinion. > "I'd like to see how fast the app will run if we take the part of the=20 > code that moves the records into the database off the webapp" Is this the= part when It puts the cached alerts and moved those in to the DB or ? >=20 > "IMO we already have a list open to the users, and the users who are=20 > genuinely interested in the development portion appear to have hooked=20 > up on this team." I agree totally and since I personally don't like this = hole mailing list way of communicate I read the users list directly on SF.=20 >=20 Is there a better way to communicate in your opinion... for example I have thought about setting up an irc channel but didn't know if it made sense. > And Kevin yes i managed to take em down ;) >=20 Cool.... Good work, now of course everyone else is curious.....<g> > Regards > /Christian >=20 Enjoy Kevin |
From: Chris S. <ch...@co...> - 2005-01-23 16:43:59
|
Kevin Johnson wrote: > On Thu, 2005-01-06 at 23:16, Chris Shepherd wrote: > >>Hi all, >> >>Sorry about the delay in getting that script together, I've had a >>migraine for a couple days and have just generally been feeling under >>the weather -- which is easy since where I am just got another 1/2 foot >>of snow this afternoon. ;) >> >>I hope to have something whipped up sometime before I start school on >>Monday, but if anyone else thinks they could do it faster feel free. :) > > > Any more work on this done? I am thinking this should be the way we > handle it in the next version.... Hello all, I'm sorry I wasn't able to get back to anyone (the list, you directly Kevin, etc.) on this. I've had a bit of a rough month personally-speaking, and between school and my personal life I just haven't had the time to participate in the development process at all. My problem is that I really don't foresee this changing before summer. I took on too many obligations at school, and I just no longer have the time to work on BASE. I'm going to have to resign as a developer for the time being. Again, I'm sorry I didn't get back to you all sooner about this. I just haven't really been near my computer at home all that much, and when I have been, I just haven't had the energy to devote to the project. If my situation changes, I hope you'll accept me back as a developer in the future, but until then, I'm glad I was able to participate (in a rather minor role) in continuing the development of the project, and I'll keep an eye on it as much as I can. -- Chris Shepherd Wise man say, chemist who falls into acid, absorbed in his work. |
From: Kevin J. <kjo...@se...> - 2005-01-23 21:07:34
|
On Sun, 2005-01-23 at 11:44, Chris Shepherd wrote: > Hello all, > I'm sorry I wasn't able to get back to anyone (the list, you directly > Kevin, etc.) on this. I've had a bit of a rough month > personally-speaking, and between school and my personal life I just > haven't had the time to participate in the development process at all. > My problem is that I really don't foresee this changing before summer. I > took on too many obligations at school, and I just no longer have the > time to work on BASE. I'm going to have to resign as a developer for the > time being. > Chris, I am sorry to hear this. You have been a great help and if you need anything to help with what ever is going on, all you have to do is ask. > Again, I'm sorry I didn't get back to you all sooner about this. I just > haven't really been near my computer at home all that much, and when I > have been, I just haven't had the energy to devote to the project. If my > situation changes, I hope you'll accept me back as a developer in the > future, but until then, I'm glad I was able to participate (in a rather > minor role) in continuing the development of the project, and I'll keep > an eye on it as much as I can. > Not a problem, when you are ready and if you would like, all you have to do is let us know and we will re-add you to the group. Thanks for all that you have contributed, Kevin |