shoki-users — A generic mailing list for the shoki NIDS.

Re: [Shoki-users] chroot - make db

[Jaliya B. <ja...@sl...>]

Dear Stephen,

Thank you very much. Actually at the moment I was able to run the 'lexer'
and log some data to database. And also I'm working on the Snort2Shoki and
CVE2shoki scripts to edit those.

Actually I'm working on an IDS benchmarking tool and trying to use your
shoki ids as the underlying IDS. Therefore it is necessary for me to have a
reference to a vulnerability database for interoperability. CVE is the best
one available according to my understanding. 
I'm planning to use Shoki in parallel to the RealSecure which has been
installed from long ago in our data center. 

Thank you once again for the info. And now after doing much I think the best
way to try as much on my own first. 

For the moment I found that the 'importer' is not logging to the database
due to the -D option in 'lexer' inside 'importer'. It says that it can not
find the dump file. So will try to resolve this first.

Best Regards,
Jaliya




-----Original Message-----
From: Stephen P. Berry [mailto:sp...@me...] 
Sent: Thursday, October 04, 2007 1:24 PM
To: Jaliya Bandara
Cc: sho...@li...
Subject: Re: [Shoki-users] chroot - make db 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>Actually I did my best first, but due to little knowledge I'm having on
Unix
>environment I could not even try running it manually.

Assuming you installed in the default location, all you need to do
to run the importer script by hand is to use:

	# /usr/local/shoki/bin/importer


>Actually before going for implementation I want to test whether it is
>possible to get Shoki logs with the vulnerability references: that is if an
>attack is going on, will it be possible for Shoki to log that event
>referring to some vulnerability number such as (CVE, Bugtraq etc.) without
>having the data in 'Vulnerabilities' table since Snort rules having those
>references or else will Shoki work without data in this table? This is the
>problem I am having.

The vulnerabilities table isn't necessary for any of the default behaviour
of shoki 0.3.0.  It's there so any applications you might implement
around shoki have access to the vulnerability information---i.e., to
provide a link to the detailed CVE data.  If you're not planning on
implementing anything like this, it isn't needed.

This, incidentally, is generally true of a lot of the stuff in the
shoki distribution.  It's intended to be more of a toolkit than a
turnkey IDS.

When 1.0 is finally released, it (should) have a lot more happy/friendly
features like this as part of the default install.


>Ok let me give a try, if I want to re-write the script to load cve data,
>from where should I start?

It's just a perl script.  Pretty much all it does is parse a CVE file
and import the results into the vulnerabilities table.  The main
logic of the parsing is in the read_file() subroutine.  It calls
the add_vuln() subroutine to update the database.  It's pretty
straightforward.



- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFHBJwhP32VcPQQS7wRAvTFAKCFqShxHFXqfy1jEUxjfCxNoQNlpwCdHv43
kgyOBw6jcjUqobNoM4/YYD8=
=a+ca
-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Stephen P. B. <sp...@me...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>Actually I did my best first, but due to little knowledge I'm having on Unix
>environment I could not even try running it manually.

Assuming you installed in the default location, all you need to do
to run the importer script by hand is to use:

	# /usr/local/shoki/bin/importer


>Actually before going for implementation I want to test whether it is
>possible to get Shoki logs with the vulnerability references: that is if an
>attack is going on, will it be possible for Shoki to log that event
>referring to some vulnerability number such as (CVE, Bugtraq etc.) without
>having the data in 'Vulnerabilities' table since Snort rules having those
>references or else will Shoki work without data in this table? This is the
>problem I am having.

The vulnerabilities table isn't necessary for any of the default behaviour
of shoki 0.3.0.  It's there so any applications you might implement
around shoki have access to the vulnerability information---i.e., to
provide a link to the detailed CVE data.  If you're not planning on
implementing anything like this, it isn't needed.

This, incidentally, is generally true of a lot of the stuff in the
shoki distribution.  It's intended to be more of a toolkit than a
turnkey IDS.

When 1.0 is finally released, it (should) have a lot more happy/friendly
features like this as part of the default install.


>Ok let me give a try, if I want to re-write the script to load cve data,
>from where should I start?

It's just a perl script.  Pretty much all it does is parse a CVE file
and import the results into the vulnerabilities table.  The main
logic of the parsing is in the read_file() subroutine.  It calls
the add_vuln() subroutine to update the database.  It's pretty
straightforward.



- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFHBJwhP32VcPQQS7wRAvTFAKCFqShxHFXqfy1jEUxjfCxNoQNlpwCdHv43
kgyOBw6jcjUqobNoM4/YYD8=
=a+ca
-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Jaliya B. <ja...@sl...>]

Dear Stephen,

Thank you very much for the infos.

Pl forgive me for troubling you. I can understand how busy you are.

Actually I did my best first, but due to little knowledge I'm having on Unix
environment I could not even try running it manually.

So I'll try it first and let you know if I can not fix it only (I'm grabbing
Unix environment fast). 

Actually before going for implementation I want to test whether it is
possible to get Shoki logs with the vulnerability references: that is if an
attack is going on, will it be possible for Shoki to log that event
referring to some vulnerability number such as (CVE, Bugtraq etc.) without
having the data in 'Vulnerabilities' table since Snort rules having those
references or else will Shoki work without data in this table? This is the
problem I am having.



Ok let me give a try, if I want to re-write the script to load cve data,
from where should I start?


Regards,
Jaliya


===============
Jaliya Bandara
Engineer - SAIII
Internet Data Center,
Sri Lanka Telecom PLC.
94 716 816 425


-----Original Message-----
From: Stephen P. Berry [mailto:sp...@me...] 
Sent: Tuesday, October 02, 2007 6:29 AM
To: Jaliya Bandara
Cc: sho...@li...
Subject: Re: [Shoki-users] chroot - make db 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>.         'importer' does not populate the db tables.
>o    All the shoki*.gz go to
>'/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be
>the problem?

Have you tried running the importer script by hand, and if so does it
report any errors.

If it doesn't, try running lexer(1) by hand on or more of the dump files.

Without any additional information, it's difficult to guess what the problem
might be.


>.         it's not possible to insert all the CVE entries using 'cve2shoki
>-f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more.
Only
>'allitems.csv' with both 'cve' & 'can' together. In this file there are
more
>than 40k entries, but only about 1762 loaded into the 'vulnerabilities'
>table. It says other records are having syntax errors such as '. Replacing
>those with spaces did not work.

This probably won't be fixed for shoki 0.3.0 .  In the soon-to-be-released
shoki 1.0 importing foreign data formats (like CVE and CAN) are handled
by their own loadable modules.  In 0.3.0, it's just hard-coded into a
script.  So the only way to make it work in 0.3.0 would be to re-write
the script itself.

If you want to undertake a re-write of the script, I'd be glad to
incorporate it into the 0.3.0 source and make the update available via
sourceforge.  Otherwise, it may or may not get fixed in 0.3.0, depending on
how busy I am.




- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFHAZfQP32VcPQQS7wRAgtgAKCQP4mTptSfnVhGuDH5P77ylCJylQCaA9zo
1gyrKdXzRfjFwzn0iZnY3eo=
=36KF
-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Stephen P. B. <sp...@me...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>.         'importer' does not populate the db tables.
>o    All the shoki*.gz go to
>'/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be
>the problem?

Have you tried running the importer script by hand, and if so does it
report any errors.

If it doesn't, try running lexer(1) by hand on or more of the dump files.

Without any additional information, it's difficult to guess what the problem
might be.


>.         it's not possible to insert all the CVE entries using 'cve2shoki
>-f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only
>'allitems.csv' with both 'cve' & 'can' together. In this file there are more
>than 40k entries, but only about 1762 loaded into the 'vulnerabilities'
>table. It says other records are having syntax errors such as '. Replacing
>those with spaces did not work.

This probably won't be fixed for shoki 0.3.0 .  In the soon-to-be-released
shoki 1.0 importing foreign data formats (like CVE and CAN) are handled
by their own loadable modules.  In 0.3.0, it's just hard-coded into a
script.  So the only way to make it work in 0.3.0 would be to re-write
the script itself.

If you want to undertake a re-write of the script, I'd be glad to
incorporate it into the 0.3.0 source and make the update available via
sourceforge.  Otherwise, it may or may not get fixed in 0.3.0, depending on
how busy I am.




- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFHAZfQP32VcPQQS7wRAgtgAKCQP4mTptSfnVhGuDH5P77ylCJylQCaA9zo
1gyrKdXzRfjFwzn0iZnY3eo=
=36KF
-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Jaliya B. <ja...@sl...>]

Dear All,

 

Can someone help me to resolve the following problems I'm having with Shoki?

 

.         'importer' does not populate the db tables.

o    All the shoki*.gz go to
'/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be
the problem?

               

.         it's not possible to insert all the CVE entries using 'cve2shoki
-f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only
'allitems.csv' with both 'cve' & 'can' together. In this file there are more
than 40k entries, but only about 1762 loaded into the 'vulnerabilities'
table. It says other records are having syntax errors such as '. Replacing
those with spaces did not work.

 

 

Regards,

Jaliya

 

 

Re: [Shoki-users] chroot - make db

[Jaliya B. <ja...@sl...>]

Dear Stephen,

 

First of all I would like to thank you for the support given.

 

Yes, the problem was with the socket, after following your steps, now I am
able to run 'make db'. It's working fine.

 

But now I have some new issues:

 

.         'importer' does not populate the db tables.

o    All the shoki*.gz go to
'/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be
the problem?

               

.         it's not possible to insert all the CVE entries using 'cve2shoki
-f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only
'allitems.csv' with both 'cve' & 'can' together. In this file there are more
than 40k entries, but only about 1762 loaded into the 'vulnerabilities'
table.

o    So could you pl help me to update the table or could you pl send me a
previous full-cve.csv and full-can.csv files that you might be having.

 

 

Regards,

Jaliya

 

 

 

 

 

 

 

 

 

-----Original Message-----
From: Stephen P. Berry [mailto:sp...@me...] 
Sent: Tuesday, September 25, 2007 11:39 AM
To: Jaliya Bandara
Cc: 'Stephen P. Berry'; sho...@li...
Subject: Re: [Shoki-users] chroot - make db 

 

-----BEGIN PGP SIGNED MESSAGE-----

Hash: SHA1

 

 

>But I was ABLE to start PgSQL after changing Unix_Socket_directory in

>postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a 

>"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the shoki/chroot/tmp

>created by "make chroot" first)

 

I think I see the problem.  Instead of:

 

      # ln -s /usr/local/shoki/choot/tmp /tmp

 

(creating a symlink for the entire directory), you just want to create

a symlink for the PostgreSQL socket:

 

      # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp

 

The longish explanation:

 

This is because by default postgres widgets (like createdb and so forth)

will look for the socket in /tmp (regardless of what postgres.conf says).

Since the shoki widgets run (by default) chroot'd and a chroot'd

process can't see outside the jail---which also means they can't follow

symlinks---that means the socket itself needs to be inside the chroot.

Everything else (which isn't running chroot'd) therefore needs the symlink

to find the socket in the non-default location.

 

Some applications (like syslog-ng) allow you to specify multiple socket

locations for just this sort of thing.  PostgreSQL unfortunately does not.

 

 

Anyway, if I understand your mail correctly, you want to:

 

      -Remove the /usr/local/shoki/chroot/tmp you created by hand

      -Re-run `make chroot' to re-recreate /usr/local/shoki/chroot/tmp

       (these two steps are just to make sure the permissions on the tmp

       directory are set correctly)

      -Stop and restart PostgreSQL (to re-create the socket itself)

      -As root:

 

       # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp

 

      -Then try `make db' again.

 

Let me know if this fixes the problems you're seeing.  I'll probably

update the documentation to be clearer on this tomorrow.

 

 

 

 

- -spb

 

-----BEGIN PGP SIGNATURE-----

Version: GnuPG v1.4.6 (OpenBSD)

 

iD8DBQFG+KX8P32VcPQQS7wRAtfpAJ9eqEss3/kTzfFaTLNAmC6WaX56mgCeNV1m

fm+FHQOFOdw4YNAds4GPD+A=

=DYtV

-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Jaliya B. <ja...@sl...>]

Dear Stephen,

First of all I would like to thank you for the support given.

Yes, the problem was with the socket, after following your steps, now 
I am able to run 'make db'. It's working fine.

But now I have some new issues:

•	‘importer’ (possibly this) does not populate the db tables.
       o	All the shoki*.gz go  to 
‘/usr/local/shoki/central/localhost.localdomain/corrupt’. where can it 
be the problem?
		
•	it's not possible to insert all the CVE entries using ‘cve2shoki 
–f’. http://cve.mitre.org/ does not have CVE & CAN .csv files any 
more. Only ‘allitems.csv’ with both ‘cve’ & ‘can’ together. In this 
file there are more than 40k entries, but only about 1762 loaded into 
the ‘vulnerabilities’ table.
       o	So could you pl help me to update the table or could you pl 
send me a previous full-cve.csv and full-can.csv files that you might 
be having.

•	If I run ‘nessus against a target with a selected attack scripts’ 
and then want to check whether ‘shoki’ has detected those selected 
attacks, how can I do that? Do I need to upload nessus reports (.nbe) 
first or using cve values (references) is it possible?
•	Is it possible to get txt based alerts with cve values without using 
database tables?

•	Only very few Snort signatures can be loaded to shoki. So I had to 
comment many snort rules in snort_converted.conf file. Is this a 
limitation of shoki or is there any way to load more snort rules?

Your further help is highly appreciated.

(pl bear with me the formating error of this mail for today)

Best Regards,
Jaliya






On Mon, 24 Sep 2007 23:09:14 -0700
  "Stephen P. Berry" <sp...@me...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
>>But I was ABLE to start PgSQL after changing Unix_Socket_directory in
>>postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a 
>>"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the 
>>shoki/chroot/tmp
>>created by "make chroot" first)
> 
> I think I see the problem.  Instead of:
> 
> 	# ln -s /usr/local/shoki/choot/tmp /tmp
> 
> (creating a symlink for the entire directory), you just want to 
>create
> a symlink for the PostgreSQL socket:
> 
> 	# ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp
> 
> The longish explanation:
> 
> This is because by default postgres widgets (like createdb and so 
>forth)
> will look for the socket in /tmp (regardless of what postgres.conf 
>says).
> Since the shoki widgets run (by default) chroot'd and a chroot'd
> process can't see outside the jail---which also means they can't 
>follow
> symlinks---that means the socket itself needs to be inside the 
>chroot.
> Everything else (which isn't running chroot'd) therefore needs the 
>symlink
> to find the socket in the non-default location.
> 
> Some applications (like syslog-ng) allow you to specify multiple 
>socket
> locations for just this sort of thing.  PostgreSQL unfortunately 
>does not.
> 
> 
> Anyway, if I understand your mail correctly, you want to:
> 
> 	-Remove the /usr/local/shoki/chroot/tmp you created by hand
> 	-Re-run `make chroot' to re-recreate /usr/local/shoki/chroot/tmp
> 	 (these two steps are just to make sure the permissions on the tmp
> 	 directory are set correctly)
> 	-Stop and restart PostgreSQL (to re-create the socket itself)
> 	-As root:
> 
> 	 # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp
> 
> 	-Then try `make db' again.
> 
> Let me know if this fixes the problems you're seeing.  I'll probably
> update the documentation to be clearer on this tomorrow.
> 
> 
> 
> 
> - -spb
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (OpenBSD)
> 
> iD8DBQFG+KX8P32VcPQQS7wRAtfpAJ9eqEss3/kTzfFaTLNAmC6WaX56mgCeNV1m
> fm+FHQOFOdw4YNAds4GPD+A=
> =DYtV
> -----END PGP SIGNATURE-----

*******************************************************************************
The information contained in this email is confidential and intended solely for
the intended recipient. Any use, distribution, transmittal or retransmittal of
information contained in this email by persons who are not intended recipients
may be a violation of law and is strictly prohibited.
If you are not the intended recipient, please contact the sender and delete all
copies.
************SLTiDC Security System scanned this email for viruses.*************
*******************************************************************************
INTERNET DATA CENTER,SRI LANKA TELECOM LIMITED,LOTUS ROAD,COLOMBO-1,SRI LANKA
*******************************************************************************

Re: [Shoki-users] chroot - make db

[Stephen P. B. <sp...@me...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>But I was ABLE to start PgSQL after changing Unix_Socket_directory in
>postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a 
>"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the shoki/chroot/tmp
>created by "make chroot" first)

I think I see the problem.  Instead of:

	# ln -s /usr/local/shoki/choot/tmp /tmp

(creating a symlink for the entire directory), you just want to create
a symlink for the PostgreSQL socket:

	# ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp

The longish explanation:

This is because by default postgres widgets (like createdb and so forth)
will look for the socket in /tmp (regardless of what postgres.conf says).
Since the shoki widgets run (by default) chroot'd and a chroot'd
process can't see outside the jail---which also means they can't follow
symlinks---that means the socket itself needs to be inside the chroot.
Everything else (which isn't running chroot'd) therefore needs the symlink
to find the socket in the non-default location.

Some applications (like syslog-ng) allow you to specify multiple socket
locations for just this sort of thing.  PostgreSQL unfortunately does not.


Anyway, if I understand your mail correctly, you want to:

	-Remove the /usr/local/shoki/chroot/tmp you created by hand
	-Re-run `make chroot' to re-recreate /usr/local/shoki/chroot/tmp
	 (these two steps are just to make sure the permissions on the tmp
	 directory are set correctly)
	-Stop and restart PostgreSQL (to re-create the socket itself)
	-As root:

	 # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp

	-Then try `make db' again.

Let me know if this fixes the problems you're seeing.  I'll probably
update the documentation to be clearer on this tomorrow.




- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFG+KX8P32VcPQQS7wRAtfpAJ9eqEss3/kTzfFaTLNAmC6WaX56mgCeNV1m
fm+FHQOFOdw4YNAds4GPD+A=
=DYtV
-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Jaliya B. <ja...@sl...>]

Dear Stephen,

I'm using RH Linux 9. I installed PostgreSQL by building it from source.
PgSQL is working fine. I can create DBs/Tables etc. 
But PgSQL is installed outside /shoki/chroot directory. 

Initially I installed Shoki without running "make chroot". Then "make db"
was successful but finally I was unable to get logs from the sensor
according to the procedure given in Shoki User manual (it checks
/shoki/chroot directory).

That's why I'm installing Shoki from the scratch with all the steps given in
the Manual. But now I have got stuck at "make db" 

I'm having a very little knowledge on jailing/chrooting. 
I think the problem occurred after I did "make chroot".

But I was ABLE to start PgSQL after changing Unix_Socket_directory in
postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a 
"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the shoki/chroot/tmp
created by "make chroot" first)

So I guessed my PgSQL is working fine and only the chrooted shoki can not
find the necessary commands. Do I need to do some more "ln -s/link"
commands.


Your help is highly appreciated.

Regards,
Jaliya


-----Original Message-----
From: Stephen P. Berry [mailto:sp...@me...] 
Sent: Tuesday, September 25, 2007 7:03 AM
To: Jaliya Bandara
Cc: sho...@li...
Subject: Re: [Shoki-users] chroot - make db 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In message <007e01c7fe9f$7cbf8590$763e90b0$@lk>, "Jaliya Bandara" writes:

>I'm new to shoki, I googled and tried several methods. But was unable to do
>"make db " after doing "make chroot".
>Log says can not find:
>created
>createlang
>psql 
>commands.

It sounds like either PostgreSQL isn't installed, or it's installed in
someplace the database creation script can't find it.

What OS/distribution/version are you trying to install under, and did
you install PostgreSQL using the OS's package management tool or by
building it from source?



- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFG+GVLP32VcPQQS7wRAhs0AJ40Hahjcyk9B5+ls4K/Sj3yT+okrwCcDq8P
EWQTnhsUnvcZToPz3kzc8DQ=
=K1Ku
-----END PGP SIGNATURE-----

Re: [Shoki-users] chroot - make db

[Stephen P. B. <sp...@me...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In message <007e01c7fe9f$7cbf8590$763e90b0$@lk>, "Jaliya Bandara" writes:

>I'm new to shoki, I googled and tried several methods. But was unable to do
>"make db " after doing "make chroot".
>Log says can not find:
>created
>createlang
>psql 
>commands.

It sounds like either PostgreSQL isn't installed, or it's installed in
someplace the database creation script can't find it.

What OS/distribution/version are you trying to install under, and did
you install PostgreSQL using the OS's package management tool or by
building it from source?



- -spb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (OpenBSD)

iD8DBQFG+GVLP32VcPQQS7wRAhs0AJ40Hahjcyk9B5+ls4K/Sj3yT+okrwCcDq8P
EWQTnhsUnvcZToPz3kzc8DQ=
=K1Ku
-----END PGP SIGNATURE-----