Re: [Os-sim-devel] OSIM arquitecture: too many components?

[<dk...@ip...>]

Hi, sorry for the late reply.

On Sun, Sep 28, 2003 at 06:48:34PM -0400, Jose Vicente Nunez Zuleta wrote=
:
> Hi,
>=20
> Je je, this is getting lengthy but lets talk :)
>=20
> On Sun, 2003-09-28 at 10:03, DK wrote:
> > Hi,
> >=20
> > El 9/27/03 16:46, "Jose Vicente Nunez Zuleta" <jo...@ne...>
> > escribi=F3:
> >=20
> > Both issues are going to be solved when we release the roadmap (I don=
't know
> > when, hopefully next week) so we can provide the following:
> >=20
> > - In depth system architecture.
> > - What to expect from updates until 1.0 is released. What we want to =
do
> > until 2.0. When to expect all of this.
> > - Tasks for ourselves as well as for anyone wishing to help with the
> > project.
> >=20
>=20
>=20
> That sounds great, this will help a lot of people to cooperate with the
> project.
>=20
>=20
> > Interesting indeed. Why do you think it would be useful to integrate =
jabber
> > ?
>=20
> Jabber is platform neutral, OpenSource and can be used for:
>=20
> 1) Monitoring in real time. Just send the messages to Jabber and with
> any client you can have them displayed in real time. Instead of getting
> hundred of nasty emails address (that you will delete after wards) you
> can keep watching the messages on a window.=20
> 2) You don have to reinvent the wheel for a messaging platform, it is
> already there.
> 3) Everybody has a IM client and a web browser. This will cut developin=
g
> costs.

I think it's and interesting addon but far away from our main focus. As o=
f today we have enough work to maintain the web interface ;)

>=20
> > Besides, our intention is to move as many things we can away from sys=
log
> > because syslog (or syslog-ng, as you want) slows things down. We have=
 to
> > speed everything up, speed it a lot up so that's our main focus as of=
 today.
> >=20
>=20
> What do you mean  with Syslog slowing things down? Syslog can become th=
e
> main entry point of messages to this system and because is already ther=
e
> it means that you don have to deploy a custom event forwarder on every
> machine. just think about this:
>=20
> 1) IPTables has the option to send the reject or discard messages to
> Syslog. Is easier to configure than the ulog module (wich can send
> messages to a MySQL database for example).
>=20
> 2) Snort can send messages to Syslog instead of emails, database
> connections or even SNMP traps (SNMP takes a lot of work to get working
> by the way).
>=20
> 3) Syslog is available =B4as is=B4 on every Cisco, Nokia Firewall, Sola=
ris
> machine, Linux machine, etc. You only need to add the following to the
> /etc/syslog.conf to make it happen (i mean on a Unix box):
>=20
> *.*	@loghost
>=20
> I agree that Syslog is not secure and can be tampered but i also believ=
e
> than is an error to ignore it as a data source of events. On a
> enterprise, you will have many data sources and Syslog is the standard
> on Unix.

We don't ignore it and we'll always use it but if it's possible we want t=
o get the data from other sources.

>=20
>=20
>=20
> > I have to draw a new architecture map, hopefully this week I can get =
two
> > hours to redraw it.
> >=20
>=20
> Cool. I know is not easy as it sounds :)

Yeah, and when you have to do your daily work too it gets even harder.

>=20
> > >=20
> > > Ok. Again, i think PostgreSQL is better suited for this task (this =
can be
> > > discussed in detail).
> >=20
> > We have to decide this issue soon. I think it is going to be easier t=
o
> > rewrite our C code to make use of postgresql rather than move OpenNMS=
 to
> > MySQL. No decision taken as of today.
> >=20
>=20
> ok.
>=20
> > >=20
> > >> - Of course nessus-opennms integration would be done after talking=
 with
> > >> Opennms's creators.
> > >=20
> > > Great.
> > >=20
> > >> - We have to write a data consolidator which accepts input from ma=
ny
> > >> more
> > >> devices.
> > >=20
> > > Syslog could be a first option, and then more could be added as 'pl=
ugins'?
> > >=20
> >=20
> > As stated before, we want to move away from syslog for speed reasons,=
 but of
> > course not everything because some products are better integrated wit=
hin
> > syslog. And we want to correlate system events too (both unix & windo=
ws).
> >=20
>=20
> I=B4m not a big fan of Syslog, but hey they even have a version on Wind=
ows
> :)
>=20
> > =20
> > > You could still use a language like Python to do the scripting part=
; Is easier
> > > to extend in C than Perl (you don't need
> > > Swig for that). Supports objects better than Perl, etc. Also if you=
 wanna use
> > > Jython instead of Python that makes it easier
> > > to glue it together with Java.
> > >=20
> > > I would use C only for tasks that require the speed, i think Java i=
s better
> > > suited as the main languaje of the application.
> >=20
> > Personally I don=B9t like Java although this opinion is changing late=
ly. The
> > scripting part is only a temporary solution because every single comp=
onent
> > needs to run as fast as possible so C is going to be used for the mai=
n core.
> > Around that we can build over every language that suits our needs. Ja=
va is
> > going to be an alternative, that for sure, but we will also use PHP, =
Perl
> > and why not, python is also an alternative (I love python).
> >=20
>=20
> I think Java is a strong languaje on the server side. Is portable (so
> the system can be run even on Windows, for the benefit of the poor
> Windoooze server out there :)). The (lack of) speed of Java is a myth
> that can give us a lot to talk for days, but here are some things for
> you to think about it:
>=20
> 1) Java offers more facilities for rapid development. You have to admit
> than working on C/C++ will require a lot of third party libraries that
> already come with Java. Database connectivity (JDBC), XML management
> 2) This application will be doing SQL queries and showing reports to th=
e
> user, rigth? The data capture is left to sensors like Snort, the networ=
k
> discovery to tools like OpenNMS (written almost entirely in Java) so i
> don see how writing the whole app in C will give you more speed while
> the database engine and probably PHP (or JSP) will be showing the data
> back to the users.
>=20
> I think other aspects should be considered besides speed, like how fast
> you can develop the application, reusable components, scalability, etc.
>=20
> Let the flame war begins :D.

Nah, no flame war. Your opinions make sense and our development is tendin=
g to a similar architecture:
- C core.
- Php (or anything else) for the presentation layer.
- Few misc scripts written in whatever (python, sh, perl, etc...)

>=20
>=20
> > > But again, what sense it makes to have two applications discovering=
 nodes at
> > > the same time? OpenNMS discovery capabilitites
> > > can be extended easily using Java plugins and the Assest database i=
s fairly
> > > complete; I think the goal of this project
> > > should be focus on how to analize all the data gathered by OpenNMS,=
 Snort,
> > > Syslog instead of replicate
> > > the polling and discovery functionality.
> >=20
> > Perhaps my lack of knowledge of the inner workings of OpenNMS mislead=
s me,
> > but I think for some small specific functions we should better rely o=
n
> > specific programs, as with Ntop.
> >=20
>=20
> Don=B4t worry, lets go one step at the time.
>=20
> > How does opennms's service detection work ? Does it rely on port numb=
er or
> > does it make some checks to ensure port 53 is DNS indeed (for example=
) ? Is
> > this work based on existing applications like amap or is it a complet=
e new
> > write up in Java ?
> >=20
>=20
> OpenNMS has two daemons: capsd and polld. Capsd detects if a host has a
> given service running, while polld checks a service previously detected.
>=20
> And here is when OpenNMS kicks nmap ass, because OpenNMS can run
> synthetic transactions so it can really interact with an application to
> see if it really leaves there, is not just a plain portscan (which can
> be misleading).

Do you know amap ? (I think it's from THC)

>=20
> For example, the JDBC poller i wrote for OpenNMS does that: talks with
> the database using JDBC and ask for the database metadata, so is a real
> query going on there. Also i wrote a custom plugin for my company that
> parses an XML and based on that detects is we are running a web service
> on a given box.
>=20
> So far, the best article i=B4ve seen on OpenNMS is here:
>=20
> http://www-106.ibm.com/developerworks/java/library/j-jmx3/

I'll read it.

>=20
> Read it and you will master the basics.
>=20
>=20
> > What about host and port discovery ? Does it use nmap or scanrand or
> > something similar or is it written up from 0 ?
> >=20
>=20
> Nope, you tell it what you expect to look on the servers and thats it.
> Also, doing a portscan on services that can crash (because they are
> poorly written) is not a good idea. The plugin concept of OpenNMS doesn
> use nmap nor scanrand at all.
>=20
> > If you have tried the program out you'll have noticed that OpenNMS
> > integration at this point is minimal. Only one link from the main pag=
e but
> > not as tight integrated as ntop or rrd for example.
> >=20
>=20
> OpenNMS can use rrdtool (just check the SNMP graphics for example). You
> can customize it too (check on the OpenNMS page a short tutorial i wrot=
e
> about how to generate custom graphics on OpenNMS with rrdtool).
>=20
>=20
>=20
> > One of the first items in my TODO list is to dissect OpenNMS's code s=
o to
> > know exactly what can be used where, but as always, time, time, time.=
..
> >=20
>=20
> Check the archives of the list or even better, why you don=B4t ask some
> questions on the developers list?

When I get to 100% OpenNMS integration I surely will do.

At this time I can't tell you much more, I haven't done almost anything t=
owards getting a decent roadmap because I'm introducing heavy parser modi=
fications, I'll release them on friday probably. But I promise to get tha=
t damn roadmap written up as soon as possible so we can discuss real deve=
lopment issues.

Greetings,

DK