Re: [Os-sim-devel] OSIM arquitecture: too many components?

[Jose V. N. Z. <jo...@us...>]

Hi,

Je je, this is getting lengthy but lets talk :)

On Sun, 2003-09-28 at 10:03, DK wrote:
> Hi,
>=20
> El 9/27/03 16:46, "Jose Vicente Nunez Zuleta" <jo...@ne...>
> escribi=F3:
>=20
> Both issues are going to be solved when we release the roadmap (I don't=
 know
> when, hopefully next week) so we can provide the following:
>=20
> - In depth system architecture.
> - What to expect from updates until 1.0 is released. What we want to do
> until 2.0. When to expect all of this.
> - Tasks for ourselves as well as for anyone wishing to help with the
> project.
>=20


That sounds great, this will help a lot of people to cooperate with the
project.


> Interesting indeed. Why do you think it would be useful to integrate ja=
bber
> ?

Jabber is platform neutral, OpenSource and can be used for:

1) Monitoring in real time. Just send the messages to Jabber and with
any client you can have them displayed in real time. Instead of getting
hundred of nasty emails address (that you will delete after wards) you
can keep watching the messages on a window.=20
2) You don have to reinvent the wheel for a messaging platform, it is
already there.
3) Everybody has a IM client and a web browser. This will cut developing
costs.

> Besides, our intention is to move as many things we can away from syslo=
g
> because syslog (or syslog-ng, as you want) slows things down. We have t=
o
> speed everything up, speed it a lot up so that's our main focus as of t=
oday.
>=20

What do you mean  with Syslog slowing things down? Syslog can become the
main entry point of messages to this system and because is already there
it means that you don have to deploy a custom event forwarder on every
machine. just think about this:

1) IPTables has the option to send the reject or discard messages to
Syslog. Is easier to configure than the ulog module (wich can send
messages to a MySQL database for example).

2) Snort can send messages to Syslog instead of emails, database
connections or even SNMP traps (SNMP takes a lot of work to get working
by the way).

3) Syslog is available =B4as is=B4 on every Cisco, Nokia Firewall, Solari=
s
machine, Linux machine, etc. You only need to add the following to the
/etc/syslog.conf to make it happen (i mean on a Unix box):

*.*	@loghost

I agree that Syslog is not secure and can be tampered but i also believe
than is an error to ignore it as a data source of events. On a
enterprise, you will have many data sources and Syslog is the standard
on Unix.



> I have to draw a new architecture map, hopefully this week I can get tw=
o
> hours to redraw it.
>=20

Cool. I know is not easy as it sounds :)

> >=20
> > Ok. Again, i think PostgreSQL is better suited for this task (this ca=
n be
> > discussed in detail).
>=20
> We have to decide this issue soon. I think it is going to be easier to
> rewrite our C code to make use of postgresql rather than move OpenNMS t=
o
> MySQL. No decision taken as of today.
>=20

ok.

> >=20
> >> - Of course nessus-opennms integration would be done after talking w=
ith
> >> Opennms's creators.
> >=20
> > Great.
> >=20
> >> - We have to write a data consolidator which accepts input from many
> >> more
> >> devices.
> >=20
> > Syslog could be a first option, and then more could be added as 'plug=
ins'?
> >=20
>=20
> As stated before, we want to move away from syslog for speed reasons, b=
ut of
> course not everything because some products are better integrated withi=
n
> syslog. And we want to correlate system events too (both unix & windows=
).
>=20

I=B4m not a big fan of Syslog, but hey they even have a version on Window=
s
:)

> =20
> > You could still use a language like Python to do the scripting part; =
Is easier
> > to extend in C than Perl (you don't need
> > Swig for that). Supports objects better than Perl, etc. Also if you w=
anna use
> > Jython instead of Python that makes it easier
> > to glue it together with Java.
> >=20
> > I would use C only for tasks that require the speed, i think Java is =
better
> > suited as the main languaje of the application.
>=20
> Personally I don=B9t like Java although this opinion is changing lately=
. The
> scripting part is only a temporary solution because every single compon=
ent
> needs to run as fast as possible so C is going to be used for the main =
core.
> Around that we can build over every language that suits our needs. Java=
 is
> going to be an alternative, that for sure, but we will also use PHP, Pe=
rl
> and why not, python is also an alternative (I love python).
>=20

I think Java is a strong languaje on the server side. Is portable (so
the system can be run even on Windows, for the benefit of the poor
Windoooze server out there :)). The (lack of) speed of Java is a myth
that can give us a lot to talk for days, but here are some things for
you to think about it:

1) Java offers more facilities for rapid development. You have to admit
than working on C/C++ will require a lot of third party libraries that
already come with Java. Database connectivity (JDBC), XML management
2) This application will be doing SQL queries and showing reports to the
user, rigth? The data capture is left to sensors like Snort, the network
discovery to tools like OpenNMS (written almost entirely in Java) so i
don see how writing the whole app in C will give you more speed while
the database engine and probably PHP (or JSP) will be showing the data
back to the users.

I think other aspects should be considered besides speed, like how fast
you can develop the application, reusable components, scalability, etc.

Let the flame war begins :D.


> > But again, what sense it makes to have two applications discovering n=
odes at
> > the same time? OpenNMS discovery capabilitites
> > can be extended easily using Java plugins and the Assest database is =
fairly
> > complete; I think the goal of this project
> > should be focus on how to analize all the data gathered by OpenNMS, S=
nort,
> > Syslog instead of replicate
> > the polling and discovery functionality.
>=20
> Perhaps my lack of knowledge of the inner workings of OpenNMS misleads =
me,
> but I think for some small specific functions we should better rely on
> specific programs, as with Ntop.
>=20

Don=B4t worry, lets go one step at the time.

> How does opennms's service detection work ? Does it rely on port number=
 or
> does it make some checks to ensure port 53 is DNS indeed (for example) =
? Is
> this work based on existing applications like amap or is it a complete =
new
> write up in Java ?
>=20

OpenNMS has two daemons: capsd and polld. Capsd detects if a host has a
given service running, while polld checks a service previously detected.

And here is when OpenNMS kicks nmap ass, because OpenNMS can run
synthetic transactions so it can really interact with an application to
see if it really leaves there, is not just a plain portscan (which can
be misleading).

For example, the JDBC poller i wrote for OpenNMS does that: talks with
the database using JDBC and ask for the database metadata, so is a real
query going on there. Also i wrote a custom plugin for my company that
parses an XML and based on that detects is we are running a web service
on a given box.

So far, the best article i=B4ve seen on OpenNMS is here:

http://www-106.ibm.com/developerworks/java/library/j-jmx3/

Read it and you will master the basics.


> What about host and port discovery ? Does it use nmap or scanrand or
> something similar or is it written up from 0 ?
>=20

Nope, you tell it what you expect to look on the servers and thats it.
Also, doing a portscan on services that can crash (because they are
poorly written) is not a good idea. The plugin concept of OpenNMS doesn
use nmap nor scanrand at all.

> If you have tried the program out you'll have noticed that OpenNMS
> integration at this point is minimal. Only one link from the main page =
but
> not as tight integrated as ntop or rrd for example.
>=20

OpenNMS can use rrdtool (just check the SNMP graphics for example). You
can customize it too (check on the OpenNMS page a short tutorial i wrote
about how to generate custom graphics on OpenNMS with rrdtool).



> One of the first items in my TODO list is to dissect OpenNMS's code so =
to
> know exactly what can be used where, but as always, time, time, time...
>=20

Check the archives of the list or even better, why you don=B4t ask some
questions on the developers list?


Hope to hear some news soon.

JV.

> >>=20
> >> Again thank you very much.
> >=20
> > Let me known if you're looking for developers.
>=20
> As soon as our roadmap is complete.
>=20
> Thanks for the input.
>=20
> DK
--=20
Jose Vicente Nunez Zuleta <jo...@us...>