Re: [Nfsen-discuss] memory problem

[Milenko S. <mic...@gm...>]

Hi Ryan,

I hope I didn't forget something but these are steps that I did when installing botents:

*1.* I copied backend file "Botnets.pm" to nfsen backend folder in my case "/usr/local/libexec/nfsen/plugins/"

*2.* I created folder /usr/local/libexec/botnets/ where I stored all required files
botnet.cron, botnet_filter, get_botnets_emerging-botcc

*3.1* content of botnet.cron (I changed it a little bit)

#!/bin/sh

/usr/local/bin/wget -P /tmp/ "http://rules.emergingthreats.net/blockrules/emerging-botcc.rules"  
if [ -f /tmp/emerging-botcc.rules ];
then
        /usr/local/libexec/botnets/get_botnets_emerging-botcc /tmp/emerging-botcc.rules > /usr/local/libexec/botnets/filterrules
        /usr/local/libexec/botnets/botnet_filter < /usr/local/libexec/botnets/filterrules > /usr/local/libexec/botnets/alert-filter
        rm /tmp/emerging-botcc.rules

else
        echo "$(date "+%b %d %H:%M:%S") botnets File emerging-botcc.rules was not downloaded. Check!!!" >> /var/log/nfsen.log
fi


*3.2* contet of botnet_filter

#!/usr/bin/perl

print "ip in [";
my $first = 1;
foreach my $line (<STDIN>) {
        next if ($line =~ /^\s*\#.*$/ || $line =~ /^\s*$/); # skip empty and comment lines

        my ($ip, $a, $b) = split(/\|/, $line);
        print $ip." ";
}
print "]";


*3.3* content of get_botnets_emerging-botcc

#!/usr/bin/perl

sub get_version ($) {
        my ($filename) = @_;
        return `stat -t $filename | cut -f 13 -d" "`;
}

$filename = $ARGV[0];

my ($stamp) = get_version($filename);
chomp($stamp);

# c&c's are timed out one week after the lists age. This means that we don't accept c&c's from a list that is older than a week
my $expire = $stamp + 86400*7; 

open(BOTNETS, "<$filename");

foreach my $line (<BOTNETS>) {
        next if ($line =~ /^\s*\#.*$/ || $line =~ /^\s*$/); # skip empty and comment lines

        my ($ip_list) = $line=~m/alert tcp \$HOME_NET any \-\> \[([^\]]+)\]/;
        foreach my $ip (split(/,/,$ip_list)) {
                print join('|',$ip, "", "", "emergingthreads", $stamp, $expire, "")."\n";
        }
}


In this file a replaced "alert any" to "alert tcp". I got solution for this from this discussion:
http://sourceforge.net/mailarchive/forum.php?thread_name=5081AA44.9080006%40unsane.co.uk&forum_name=nfsen-discuss


*4.* Crontab
0 * * * * /usr/local/libexec/botnets/botnet.cron

Running botnet.cron script will create files filterrules and alert-filter.

*5.* my nfsen.conf for Botnets

@plugins = (
    [ "live", 'Events' ],
    [ "!", 'Botnets' ],
);

%PluginConf = (

periodic_queries => [
        { # Botnet events are timed out 8 hour after the last update
                "Type"=>"[eq]botnet",
                "UpdateTime"=>['[le]#$unix_time-28800#','#$unix_time#'],
                "StopTime"=>["[null]",'#$unix_time#'],
        },
        { # Botnet events are upgraded if the bot contacted the controller more than 20 times
                "Type"=>"[eq]botnet",
                "Times"=>"[ge]20",
                "UpdateTime"=>'#$unix_time#',
                "Level"=>["[eq]notify","alarm"],
        },
    ],

events  => {
        db_connection_string    => "DBI:mysql:database=events;host=localhost;port=3306",
        db_user                 => "user",
        db_passwd               => "pass",
        },

delete_queries          => [
                { # Delete events after 14 days (14 days*24 hour*60 minute*60 seconds=1209600 seconds)    
                        "StopTime"=>['[le]#$unix_time-1209600#'],
                }
        ],
        
botnets => {
        import_cmd => "/bin/cat /usr/local/libexec/botnets/filterrules",
        },
);

*6. *I installed events plugin as described in README file.

I hope that it will help you with your installation.

Regards


On Thursday 11 of April 2013 16:39:52 Ryan West wrote:


Hey Mitja,
 
I’m looking to get Botnets with Events configured for my nfsen installation.  I was wondering if you can share of the steps you used?
 
Thanks,
 
-ryan
 
*From:* Mitja Podlogar [mailto:mic...@gm...] 

*Sent:* Thursday, April 11, 2013 9:30 AM

*To:* nfs...@li...

*Subject:* Re: [Nfsen-discuss] memory problem
 
Update:
I have tested a little and found out that RAM/SWAP consumption is done by plugin "nfsight". Even though I configured nfsight to use 512M of RAM it goes beyond that. 
Problem is with networks that I added, two /16 and I guess this is to much for server to handle it. I now added only one /16 as an internal network and  it works but most of the traffic is generated form other /16. 
I am using nfsight-beta-2013-03-23.
I don't have much experience with this so I don't really know what is normal behavior for this plugin. Maybe someone who have used it for a while can tell theirs statistics.

 
 
On Wed, Apr 10, 2013 at 1:00 PM, Mitja Podlogar <mic...@gm...[1]> wrote:
I installed Nfsen 1.3.6p1 and nfdump 1.6.9 on FreeBSD 9.1. I'm using these plugins: SURFmap, PortTracker, Botnets (with Events) and nfsight.
Everything was working great for more then a month. Today I noticed strange behavior of process "nfsend-comm" which started using 2 CPU and entire RAM + entire SWAP. At that time nothing really special was done except watching graphs.
I tried restarting nfsen service and it was working for a minute and then again nfsend-comm starts using RAM and CPU.



Yesterday I added plugin PortTracker and upgraded nfdump from 1.6.8 to 1.6.9. I don't really know if it has anything to do with this problem since everything was working fine for 24 hours.
I tried and disabled PortTracker but it didn't help. 
 
If logs are needed I can provide them but I didn't see anything interesting in messages and nfsen log.
I would be very grateful if anyone could tell me how to proceed with debugging this or if anyone already had this problem and knows a solution.
Thank you in advance. 
 

--------
[1] mailto:mic...@gm...