shoki-users Mailing List for shoki
Status: Beta
Brought to you by:
spb
You can subscribe to this list here.
2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(6) |
Oct
(5) |
Nov
|
Dec
|
---|
From: Jaliya B. <ja...@sl...> - 2007-10-04 08:55:01
|
Dear Stephen, Thank you very much. Actually at the moment I was able to run the 'lexer' and log some data to database. And also I'm working on the Snort2Shoki and CVE2shoki scripts to edit those. Actually I'm working on an IDS benchmarking tool and trying to use your shoki ids as the underlying IDS. Therefore it is necessary for me to have a reference to a vulnerability database for interoperability. CVE is the best one available according to my understanding. I'm planning to use Shoki in parallel to the RealSecure which has been installed from long ago in our data center. Thank you once again for the info. And now after doing much I think the best way to try as much on my own first. For the moment I found that the 'importer' is not logging to the database due to the -D option in 'lexer' inside 'importer'. It says that it can not find the dump file. So will try to resolve this first. Best Regards, Jaliya -----Original Message----- From: Stephen P. Berry [mailto:sp...@me...] Sent: Thursday, October 04, 2007 1:24 PM To: Jaliya Bandara Cc: sho...@li... Subject: Re: [Shoki-users] chroot - make db -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >Actually I did my best first, but due to little knowledge I'm having on Unix >environment I could not even try running it manually. Assuming you installed in the default location, all you need to do to run the importer script by hand is to use: # /usr/local/shoki/bin/importer >Actually before going for implementation I want to test whether it is >possible to get Shoki logs with the vulnerability references: that is if an >attack is going on, will it be possible for Shoki to log that event >referring to some vulnerability number such as (CVE, Bugtraq etc.) without >having the data in 'Vulnerabilities' table since Snort rules having those >references or else will Shoki work without data in this table? This is the >problem I am having. The vulnerabilities table isn't necessary for any of the default behaviour of shoki 0.3.0. It's there so any applications you might implement around shoki have access to the vulnerability information---i.e., to provide a link to the detailed CVE data. If you're not planning on implementing anything like this, it isn't needed. This, incidentally, is generally true of a lot of the stuff in the shoki distribution. It's intended to be more of a toolkit than a turnkey IDS. When 1.0 is finally released, it (should) have a lot more happy/friendly features like this as part of the default install. >Ok let me give a try, if I want to re-write the script to load cve data, >from where should I start? It's just a perl script. Pretty much all it does is parse a CVE file and import the results into the vulnerabilities table. The main logic of the parsing is in the read_file() subroutine. It calls the add_vuln() subroutine to update the database. It's pretty straightforward. - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFHBJwhP32VcPQQS7wRAvTFAKCFqShxHFXqfy1jEUxjfCxNoQNlpwCdHv43 kgyOBw6jcjUqobNoM4/YYD8= =a+ca -----END PGP SIGNATURE----- |
From: Stephen P. B. <sp...@me...> - 2007-10-04 07:54:31
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >Actually I did my best first, but due to little knowledge I'm having on Unix >environment I could not even try running it manually. Assuming you installed in the default location, all you need to do to run the importer script by hand is to use: # /usr/local/shoki/bin/importer >Actually before going for implementation I want to test whether it is >possible to get Shoki logs with the vulnerability references: that is if an >attack is going on, will it be possible for Shoki to log that event >referring to some vulnerability number such as (CVE, Bugtraq etc.) without >having the data in 'Vulnerabilities' table since Snort rules having those >references or else will Shoki work without data in this table? This is the >problem I am having. The vulnerabilities table isn't necessary for any of the default behaviour of shoki 0.3.0. It's there so any applications you might implement around shoki have access to the vulnerability information---i.e., to provide a link to the detailed CVE data. If you're not planning on implementing anything like this, it isn't needed. This, incidentally, is generally true of a lot of the stuff in the shoki distribution. It's intended to be more of a toolkit than a turnkey IDS. When 1.0 is finally released, it (should) have a lot more happy/friendly features like this as part of the default install. >Ok let me give a try, if I want to re-write the script to load cve data, >from where should I start? It's just a perl script. Pretty much all it does is parse a CVE file and import the results into the vulnerabilities table. The main logic of the parsing is in the read_file() subroutine. It calls the add_vuln() subroutine to update the database. It's pretty straightforward. - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFHBJwhP32VcPQQS7wRAvTFAKCFqShxHFXqfy1jEUxjfCxNoQNlpwCdHv43 kgyOBw6jcjUqobNoM4/YYD8= =a+ca -----END PGP SIGNATURE----- |
From: Jaliya B. <ja...@sl...> - 2007-10-02 04:04:14
|
Dear Stephen, Thank you very much for the infos. Pl forgive me for troubling you. I can understand how busy you are. Actually I did my best first, but due to little knowledge I'm having on Unix environment I could not even try running it manually. So I'll try it first and let you know if I can not fix it only (I'm grabbing Unix environment fast). Actually before going for implementation I want to test whether it is possible to get Shoki logs with the vulnerability references: that is if an attack is going on, will it be possible for Shoki to log that event referring to some vulnerability number such as (CVE, Bugtraq etc.) without having the data in 'Vulnerabilities' table since Snort rules having those references or else will Shoki work without data in this table? This is the problem I am having. Ok let me give a try, if I want to re-write the script to load cve data, from where should I start? Regards, Jaliya =============== Jaliya Bandara Engineer - SAIII Internet Data Center, Sri Lanka Telecom PLC. 94 716 816 425 -----Original Message----- From: Stephen P. Berry [mailto:sp...@me...] Sent: Tuesday, October 02, 2007 6:29 AM To: Jaliya Bandara Cc: sho...@li... Subject: Re: [Shoki-users] chroot - make db -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >. 'importer' does not populate the db tables. >o All the shoki*.gz go to >'/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be >the problem? Have you tried running the importer script by hand, and if so does it report any errors. If it doesn't, try running lexer(1) by hand on or more of the dump files. Without any additional information, it's difficult to guess what the problem might be. >. it's not possible to insert all the CVE entries using 'cve2shoki >-f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only >'allitems.csv' with both 'cve' & 'can' together. In this file there are more >than 40k entries, but only about 1762 loaded into the 'vulnerabilities' >table. It says other records are having syntax errors such as '. Replacing >those with spaces did not work. This probably won't be fixed for shoki 0.3.0 . In the soon-to-be-released shoki 1.0 importing foreign data formats (like CVE and CAN) are handled by their own loadable modules. In 0.3.0, it's just hard-coded into a script. So the only way to make it work in 0.3.0 would be to re-write the script itself. If you want to undertake a re-write of the script, I'd be glad to incorporate it into the 0.3.0 source and make the update available via sourceforge. Otherwise, it may or may not get fixed in 0.3.0, depending on how busy I am. - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFHAZfQP32VcPQQS7wRAgtgAKCQP4mTptSfnVhGuDH5P77ylCJylQCaA9zo 1gyrKdXzRfjFwzn0iZnY3eo= =36KF -----END PGP SIGNATURE----- |
From: Stephen P. B. <sp...@me...> - 2007-10-02 00:59:26
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >. 'importer' does not populate the db tables. >o All the shoki*.gz go to >'/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be >the problem? Have you tried running the importer script by hand, and if so does it report any errors. If it doesn't, try running lexer(1) by hand on or more of the dump files. Without any additional information, it's difficult to guess what the problem might be. >. it's not possible to insert all the CVE entries using 'cve2shoki >-f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only >'allitems.csv' with both 'cve' & 'can' together. In this file there are more >than 40k entries, but only about 1762 loaded into the 'vulnerabilities' >table. It says other records are having syntax errors such as '. Replacing >those with spaces did not work. This probably won't be fixed for shoki 0.3.0 . In the soon-to-be-released shoki 1.0 importing foreign data formats (like CVE and CAN) are handled by their own loadable modules. In 0.3.0, it's just hard-coded into a script. So the only way to make it work in 0.3.0 would be to re-write the script itself. If you want to undertake a re-write of the script, I'd be glad to incorporate it into the 0.3.0 source and make the update available via sourceforge. Otherwise, it may or may not get fixed in 0.3.0, depending on how busy I am. - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFHAZfQP32VcPQQS7wRAgtgAKCQP4mTptSfnVhGuDH5P77ylCJylQCaA9zo 1gyrKdXzRfjFwzn0iZnY3eo= =36KF -----END PGP SIGNATURE----- |
From: Jaliya B. <ja...@sl...> - 2007-10-01 03:10:20
|
Dear All, Can someone help me to resolve the following problems I'm having with Shoki? . 'importer' does not populate the db tables. o All the shoki*.gz go to '/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be the problem? . it's not possible to insert all the CVE entries using 'cve2shoki -f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only 'allitems.csv' with both 'cve' & 'can' together. In this file there are more than 40k entries, but only about 1762 loaded into the 'vulnerabilities' table. It says other records are having syntax errors such as '. Replacing those with spaces did not work. Regards, Jaliya |
From: Jaliya B. <ja...@sl...> - 2007-09-28 09:57:48
|
Dear Stephen, First of all I would like to thank you for the support given. Yes, the problem was with the socket, after following your steps, now I am able to run 'make db'. It's working fine. But now I have some new issues: . 'importer' does not populate the db tables. o All the shoki*.gz go to '/usr/local/shoki/central/localhost.localdomain/corrupt'. where can it be the problem? . it's not possible to insert all the CVE entries using 'cve2shoki -f'. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only 'allitems.csv' with both 'cve' & 'can' together. In this file there are more than 40k entries, but only about 1762 loaded into the 'vulnerabilities' table. o So could you pl help me to update the table or could you pl send me a previous full-cve.csv and full-can.csv files that you might be having. Regards, Jaliya -----Original Message----- From: Stephen P. Berry [mailto:sp...@me...] Sent: Tuesday, September 25, 2007 11:39 AM To: Jaliya Bandara Cc: 'Stephen P. Berry'; sho...@li... Subject: Re: [Shoki-users] chroot - make db -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >But I was ABLE to start PgSQL after changing Unix_Socket_directory in >postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a >"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the shoki/chroot/tmp >created by "make chroot" first) I think I see the problem. Instead of: # ln -s /usr/local/shoki/choot/tmp /tmp (creating a symlink for the entire directory), you just want to create a symlink for the PostgreSQL socket: # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp The longish explanation: This is because by default postgres widgets (like createdb and so forth) will look for the socket in /tmp (regardless of what postgres.conf says). Since the shoki widgets run (by default) chroot'd and a chroot'd process can't see outside the jail---which also means they can't follow symlinks---that means the socket itself needs to be inside the chroot. Everything else (which isn't running chroot'd) therefore needs the symlink to find the socket in the non-default location. Some applications (like syslog-ng) allow you to specify multiple socket locations for just this sort of thing. PostgreSQL unfortunately does not. Anyway, if I understand your mail correctly, you want to: -Remove the /usr/local/shoki/chroot/tmp you created by hand -Re-run `make chroot' to re-recreate /usr/local/shoki/chroot/tmp (these two steps are just to make sure the permissions on the tmp directory are set correctly) -Stop and restart PostgreSQL (to re-create the socket itself) -As root: # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp -Then try `make db' again. Let me know if this fixes the problems you're seeing. I'll probably update the documentation to be clearer on this tomorrow. - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFG+KX8P32VcPQQS7wRAtfpAJ9eqEss3/kTzfFaTLNAmC6WaX56mgCeNV1m fm+FHQOFOdw4YNAds4GPD+A= =DYtV -----END PGP SIGNATURE----- |
From: Jaliya B. <ja...@sl...> - 2007-09-27 05:46:07
|
Dear Stephen, First of all I would like to thank you for the support given. Yes, the problem was with the socket, after following your steps, now I am able to run 'make db'. It's working fine. But now I have some new issues: • ‘importer’ (possibly this) does not populate the db tables. o All the shoki*.gz go to ‘/usr/local/shoki/central/localhost.localdomain/corrupt’. where can it be the problem? • it's not possible to insert all the CVE entries using ‘cve2shoki –f’. http://cve.mitre.org/ does not have CVE & CAN .csv files any more. Only ‘allitems.csv’ with both ‘cve’ & ‘can’ together. In this file there are more than 40k entries, but only about 1762 loaded into the ‘vulnerabilities’ table. o So could you pl help me to update the table or could you pl send me a previous full-cve.csv and full-can.csv files that you might be having. • If I run ‘nessus against a target with a selected attack scripts’ and then want to check whether ‘shoki’ has detected those selected attacks, how can I do that? Do I need to upload nessus reports (.nbe) first or using cve values (references) is it possible? • Is it possible to get txt based alerts with cve values without using database tables? • Only very few Snort signatures can be loaded to shoki. So I had to comment many snort rules in snort_converted.conf file. Is this a limitation of shoki or is there any way to load more snort rules? Your further help is highly appreciated. (pl bear with me the formating error of this mail for today) Best Regards, Jaliya On Mon, 24 Sep 2007 23:09:14 -0700 "Stephen P. Berry" <sp...@me...> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > >>But I was ABLE to start PgSQL after changing Unix_Socket_directory in >>postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a >>"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the >>shoki/chroot/tmp >>created by "make chroot" first) > > I think I see the problem. Instead of: > > # ln -s /usr/local/shoki/choot/tmp /tmp > > (creating a symlink for the entire directory), you just want to >create > a symlink for the PostgreSQL socket: > > # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp > > The longish explanation: > > This is because by default postgres widgets (like createdb and so >forth) > will look for the socket in /tmp (regardless of what postgres.conf >says). > Since the shoki widgets run (by default) chroot'd and a chroot'd > process can't see outside the jail---which also means they can't >follow > symlinks---that means the socket itself needs to be inside the >chroot. > Everything else (which isn't running chroot'd) therefore needs the >symlink > to find the socket in the non-default location. > > Some applications (like syslog-ng) allow you to specify multiple >socket > locations for just this sort of thing. PostgreSQL unfortunately >does not. > > > Anyway, if I understand your mail correctly, you want to: > > -Remove the /usr/local/shoki/chroot/tmp you created by hand > -Re-run `make chroot' to re-recreate /usr/local/shoki/chroot/tmp > (these two steps are just to make sure the permissions on the tmp > directory are set correctly) > -Stop and restart PostgreSQL (to re-create the socket itself) > -As root: > > # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp > > -Then try `make db' again. > > Let me know if this fixes the problems you're seeing. I'll probably > update the documentation to be clearer on this tomorrow. > > > > > - -spb > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (OpenBSD) > > iD8DBQFG+KX8P32VcPQQS7wRAtfpAJ9eqEss3/kTzfFaTLNAmC6WaX56mgCeNV1m > fm+FHQOFOdw4YNAds4GPD+A= > =DYtV > -----END PGP SIGNATURE----- ******************************************************************************* The information contained in this email is confidential and intended solely for the intended recipient. Any use, distribution, transmittal or retransmittal of information contained in this email by persons who are not intended recipients may be a violation of law and is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ************SLTiDC Security System scanned this email for viruses.************* ******************************************************************************* INTERNET DATA CENTER,SRI LANKA TELECOM LIMITED,LOTUS ROAD,COLOMBO-1,SRI LANKA ******************************************************************************* |
From: Stephen P. B. <sp...@me...> - 2007-09-25 06:09:28
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >But I was ABLE to start PgSQL after changing Unix_Socket_directory in >postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a >"ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the shoki/chroot/tmp >created by "make chroot" first) I think I see the problem. Instead of: # ln -s /usr/local/shoki/choot/tmp /tmp (creating a symlink for the entire directory), you just want to create a symlink for the PostgreSQL socket: # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp The longish explanation: This is because by default postgres widgets (like createdb and so forth) will look for the socket in /tmp (regardless of what postgres.conf says). Since the shoki widgets run (by default) chroot'd and a chroot'd process can't see outside the jail---which also means they can't follow symlinks---that means the socket itself needs to be inside the chroot. Everything else (which isn't running chroot'd) therefore needs the symlink to find the socket in the non-default location. Some applications (like syslog-ng) allow you to specify multiple socket locations for just this sort of thing. PostgreSQL unfortunately does not. Anyway, if I understand your mail correctly, you want to: -Remove the /usr/local/shoki/chroot/tmp you created by hand -Re-run `make chroot' to re-recreate /usr/local/shoki/chroot/tmp (these two steps are just to make sure the permissions on the tmp directory are set correctly) -Stop and restart PostgreSQL (to re-create the socket itself) -As root: # ln -s /usr/local/shoki/chroot/tmp/.s.PGSQL.5432 /tmp -Then try `make db' again. Let me know if this fixes the problems you're seeing. I'll probably update the documentation to be clearer on this tomorrow. - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFG+KX8P32VcPQQS7wRAtfpAJ9eqEss3/kTzfFaTLNAmC6WaX56mgCeNV1m fm+FHQOFOdw4YNAds4GPD+A= =DYtV -----END PGP SIGNATURE----- |
From: Jaliya B. <ja...@sl...> - 2007-09-25 03:51:03
|
Dear Stephen, I'm using RH Linux 9. I installed PostgreSQL by building it from source. PgSQL is working fine. I can create DBs/Tables etc. But PgSQL is installed outside /shoki/chroot directory. Initially I installed Shoki without running "make chroot". Then "make db" was successful but finally I was unable to get logs from the sensor according to the procedure given in Shoki User manual (it checks /shoki/chroot directory). That's why I'm installing Shoki from the scratch with all the steps given in the Manual. But now I have got stuck at "make db" I'm having a very little knowledge on jailing/chrooting. I think the problem occurred after I did "make chroot". But I was ABLE to start PgSQL after changing Unix_Socket_directory in postgres.conf to "/usr/local/shoki/chroot/tmp" and doing a "ln -s /usr/local/shoki/chroot/tmp /tmp" (removed the shoki/chroot/tmp created by "make chroot" first) So I guessed my PgSQL is working fine and only the chrooted shoki can not find the necessary commands. Do I need to do some more "ln -s/link" commands. Your help is highly appreciated. Regards, Jaliya -----Original Message----- From: Stephen P. Berry [mailto:sp...@me...] Sent: Tuesday, September 25, 2007 7:03 AM To: Jaliya Bandara Cc: sho...@li... Subject: Re: [Shoki-users] chroot - make db -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 In message <007e01c7fe9f$7cbf8590$763e90b0$@lk>, "Jaliya Bandara" writes: >I'm new to shoki, I googled and tried several methods. But was unable to do >"make db " after doing "make chroot". >Log says can not find: >created >createlang >psql >commands. It sounds like either PostgreSQL isn't installed, or it's installed in someplace the database creation script can't find it. What OS/distribution/version are you trying to install under, and did you install PostgreSQL using the OS's package management tool or by building it from source? - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFG+GVLP32VcPQQS7wRAhs0AJ40Hahjcyk9B5+ls4K/Sj3yT+okrwCcDq8P EWQTnhsUnvcZToPz3kzc8DQ= =K1Ku -----END PGP SIGNATURE----- |
From: Stephen P. B. <sp...@me...> - 2007-09-25 01:33:16
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 In message <007e01c7fe9f$7cbf8590$763e90b0$@lk>, "Jaliya Bandara" writes: >I'm new to shoki, I googled and tried several methods. But was unable to do >"make db " after doing "make chroot". >Log says can not find: >created >createlang >psql >commands. It sounds like either PostgreSQL isn't installed, or it's installed in someplace the database creation script can't find it. What OS/distribution/version are you trying to install under, and did you install PostgreSQL using the OS's package management tool or by building it from source? - -spb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (OpenBSD) iD8DBQFG+GVLP32VcPQQS7wRAhs0AJ40Hahjcyk9B5+ls4K/Sj3yT+okrwCcDq8P EWQTnhsUnvcZToPz3kzc8DQ= =K1Ku -----END PGP SIGNATURE----- |