Re: [Nfsen-discuss] Nfsen profiles are not collecting data
Netflow visualisation and investigation tool
Brought to you by:
phaag
Menu
▾
▴
Re: [Nfsen-discuss] Nfsen profiles are not collecting data
|
From: cedric.delaunay <ced...@gm...> - 2012-07-31 08:59:39
|
Hi all,
Same problem here after upgrade yesterday from nfsen 1.3.5/nfdump 1.6.1
to lastest releases :1.3.6p1 and 1.6.6.
Nfcapd capture flows.
Live profile looks good (I guess rrd works) but alerts and other
profiles (continuous or continus/shadow on my box) don't create graphs.
plugins installed : ddd, SSHCURE, SURFmap, Porttracker(doesn't work,
but long), Events and a home made one.
=> Profiles :
- nfcapd files are no longer copied in continuous profiles directory's
but they were before update.
- tried to create a new/fresh one withous success
You have to know that some profile filters are edited nightly by cronjob
and are long. Example :
wc /data/nfsen/profiles-stat/protocoles/dns/sortant-filter.txt
1 1032 6818 /data/nfsen/profiles-stat/protocoles/dns/sortant-filter.txt
here are my logs for profiles :
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Run periodic at Tue Jul
31 10:35:00 2012
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'./Traffic_interne'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling './live'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'./testucop1'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'./traffic_ucop'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'protocoles/dns'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'protocoles/dns_vers_serveurs_inconnu'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'protocoles/ntpAncien'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'protocoles/partage_windows'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'protocoles/smtp'
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Prepare profiling
'protocoles/web'
Jul 31 10:35:15 dionysos user.notice nfsen[28507]: 17 channels/alerts to
profile
Jul 31 10:35:15 dionysos user.info nfsen[28508]: comm child[28739]
terminated with no exit value
Jul 31 10:35:15 dionysos user.info nfsen[28740]: nfsend: exit
child[28741] Exit: 255, Signal: 0, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28744]: nfsend: exit
child[28746] Exit: 255, Signal: 0, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28745]: nfsend: exit
child[28747] Exit: 255, Signal: 0, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28748]: nfsend: exit
child[28749] Exit: 255, Signal: 0, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28742]: nfsend: exit
child[28743] Exit: 255, Signal: 0, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28742]: profile opts:
.#Traffic_interne#2#cs6513#cs6513 for profiler 1
Jul 31 10:35:15 dionysos user.info nfsen[28742]: profile opts:
protocoles#dns#6#sortant#ucop1-V4|ucop2-V4 for profiler 1
Jul 31 10:35:15 dionysos user.info nfsen[28740]: profile opts:
protocoles#dns#6#entrant#ucop1-V4|ucop2-V4 for profiler 0
Jul 31 10:35:15 dionysos user.info nfsen[28740]: profile opts:
protocoles#smtp#6#cs7204#ucop1-V4|ucop2-V4 for profiler 0
Jul 31 10:35:15 dionysos user.info nfsen[28742]: profile opts:
protocoles#web#6#sortant#ucop1-V4|ucop2-V4 for profiler 1
Jul 31 10:35:15 dionysos user.info nfsen[28745]: profile opts:
.#traffic_ucop#2#ucop1-V4#ucop1-V4 for profiler 3
Jul 31 10:35:15 dionysos user.info nfsen[28745]: profile opts:
protocoles#ntpAncien#6#VersCoeur#cs6513 for profiler 3
Jul 31 10:35:15 dionysos user.info nfsen[28745]: profile opts:
.#~CherchePics#8#CherchePics#ucop1-V4|ucop2-V4 for profiler 3
Jul 31 10:35:15 dionysos user.info nfsen[28740]: profiler 0 started
Jul 31 10:35:15 dionysos user.info nfsen[28742]: profiler 1 started
Jul 31 10:35:15 dionysos user.info nfsen[28507]: nfsend: exit
child[28742] Exit: 0, Signal: 13, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28507]: nfsend: exit
child[28740] Exit: 0, Signal: 13, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28744]: profile opts:
.#testucop1#6#ucop1-V4#ucop1-V4 for profiler 2
Jul 31 10:35:15 dionysos user.info nfsen[28744]: profile opts:
protocoles#dns_vers_serveurs_inconnu#6#cs6513#cs6513 for profiler 2
Jul 31 10:35:15 dionysos user.info nfsen[28744]: profile opts:
.#~botdetect#8#botdetect#ucop1-V4|ucop2-V4 for profiler 2
Jul 31 10:35:15 dionysos user.info nfsen[28744]: profiler 2 started
Jul 31 10:35:15 dionysos user.info nfsen[28745]: profiler 3 started
Jul 31 10:35:15 dionysos user.info nfsen[28748]: profile opts:
.#traffic_ucop#2#ucop2-V4#ucop2-V4 for profiler 4
Jul 31 10:35:15 dionysos user.info nfsen[28748]: profile opts:
protocoles#ntpAncien#6#versVntp#cs6513 for profiler 4
Jul 31 10:35:15 dionysos user.info nfsen[28748]: profile opts:
.#~DNSCchangerMalwareVictims#8#DNSCchangerMalwareVictims#ucop1-V4|ucop2-V4
for profiler 4
Jul 31 10:35:15 dionysos user.info nfsen[28748]: profiler 4 started
Jul 31 10:35:15 dionysos user.info nfsen[28750]: nfsend: exit
child[28751] Exit: 255, Signal: 0, Core: 0
Jul 31 10:35:15 dionysos user.info nfsen[28750]: profile opts:
protocoles#dns#6#interne#cs6513 for profiler 5
Jul 31 10:35:15 dionysos user.info nfsen[28750]: profile opts:
protocoles#partage_windows#6#cs6513#cs6513 for profiler 5
Jul 31 10:35:15 dionysos user.info nfsen[28750]: profile opts:
.#~test#8#test#cs6513|ucop1-V4|ucop2-V4 for profiler 5
Jul 31 10:35:15 dionysos user.info nfsen[28750]: profiler 5 started
Jul 31 10:35:15 dionysos user.notice nfsen[28507]: Update profile
Traffic_interne in group .
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Add channel size
30285926400
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Set new profile size:
30285926400
Jul 31 10:35:15 dionysos user.info nfsen[28507]: Add
.:Traffic_interne:201207311030 for plugin processing
Jul 31 10:35:16 dionysos user.notice nfsen[28507]: Update profile live
in group .
Jul 31 10:35:16 dionysos user.info nfsen[28507]: Add channel size
544696852480
Jul 31 10:35:16 dionysos user.info nfsen[28507]: Add channel size
232831922176
Jul 31 10:35:16 dionysos user.info nfsen[28507]: Add channel size 1008914432
Jul 31 10:35:16 dionysos user.info nfsen[28507]: Set new profile size:
778537689088
Jul 31 10:35:16 dionysos user.info nfsen[28507]: Add .:live:201207311030
for plugin processing
Jul 31 10:35:17 dionysos user.notice nfsen[28507]: Update profile
testucop1 in group .
Jul 31 10:35:18 dionysos user.notice nfsen[28507]: Update profile
traffic_ucop in group .
Jul 31 10:35:18 dionysos user.info nfsen[28507]: Add channel size
30558482432
Jul 31 10:35:18 dionysos user.info nfsen[28507]: Add channel size 87322624
Jul 31 10:35:18 dionysos user.info nfsen[28507]: Set new profile size:
30645805056
Jul 31 10:35:18 dionysos user.info nfsen[28507]: Add
.:traffic_ucop:201207311030 for plugin processing
Jul 31 10:35:19 dionysos user.notice nfsen[28507]: Update profile dns in
group protocoles
Jul 31 10:35:19 dionysos user.notice nfsen[28507]: Update profile
dns_vers_serveurs_inconnu in group protocoles
Jul 31 10:35:20 dionysos user.notice nfsen[28507]: Update profile
ntpAncien in group protocoles
Jul 31 10:35:21 dionysos user.notice nfsen[28507]: Update profile
partage_windows in group protocoles
Jul 31 10:35:22 dionysos user.notice nfsen[28507]: Update profile smtp
in group protocoles
Jul 31 10:35:23 dionysos user.notice nfsen[28507]: Update profile web in
group protocoles
Jul 31 10:35:24 dionysos user.info nfsen[28507]: Run plugins for
201207311030
=> Alerts :
Trying to understand, I patched file /.../.../nfsen/libexec/NfAlert.pm
like this :
$err = "No flow file for requested time slot : $file";
here are my logs for alerts :
Jul 31 10:35:33 dionysos user.info nfsen[28507]: Check alerts for Tue
Jul 31 10:30:00 2012
Jul 31 10:35:33 dionysos user.notice nfsen[28507]: Process alert 'botdetect'
Jul 31 10:35:33 dionysos user.info nfsen[28507]: alert 'botdetect':
conditions based on plugin
Jul 31 10:35:33 dionysos user.crit nfsen[28507]: Error reading statinfo
of 'botdetect': No flow file for requested time slot :
/data/nfsen/profiles-data/~botdetect/botdetect/nfcapd.201207311030
Jul 31 10:35:33 dionysos user.info nfsen[28507]: Alert 'botdetect' done.
Jul 31 10:35:33 dionysos user.notice nfsen[28507]: Process alert
'CherchePics'
Jul 31 10:35:33 dionysos user.info nfsen[28507]: alert 'CherchePics':
conditions based on total flow summary
Jul 31 10:35:33 dionysos user.crit nfsen[28507]: Error reading statinfo
of 'CherchePics': No flow file for requested time slot :
/data/nfsen/profiles-data/~CherchePics/CherchePics/nfcapd.201207311030
Jul 31 10:35:33 dionysos user.info nfsen[28507]: Alert 'CherchePics' done.
Jul 31 10:35:33 dionysos user.notice nfsen[28507]: Process alert
'DNSCchangerMalwareVictims'
Jul 31 10:35:33 dionysos user.info nfsen[28507]: alert
'DNSCchangerMalwareVictims': conditions based on total flow summary
Jul 31 10:35:33 dionysos user.crit nfsen[28507]: Error reading statinfo
of 'DNSCchangerMalwareVictims': No flow file for requested time slot :
/data/nfsen/profiles-data/~DNSCchangerMalwareVictims/DNSCchangerMalwareVictims/nfcapd.201207311030
Jul 31 10:35:33 dionysos user.info nfsen[28507]: Alert
'DNSCchangerMalwareVictims' done.
Jul 31 10:35:33 dionysos user.notice nfsen[28507]: Process alert 'test'
Jul 31 10:35:33 dionysos user.info nfsen[28507]: alert 'test':
conditions based on total flow summary
Jul 31 10:35:33 dionysos user.crit nfsen[28507]: Error reading statinfo
of 'test': No flow file for requested time slot :
/data/nfsen/profiles-data/~test/test/nfcapd.201207311030
Jul 31 10:35:33 dionysos user.info nfsen[28507]: Alert 'test' done.
Jul 31 10:35:33 dionysos user.info nfsen[28507]: Check alerts done.
Any Idea ?
Thanks a lot
Cédric
Le 30/07/2012 10:39, tes...@gm... a écrit :
> Hello Peter,
>> Make sure your rrd library and Perl module for RRD are linked correctly,
>> as well as the loader will find them.
> The live profile works fine. The graphs looks good.
> I am not use empy filters for subprofile. I use like 'proto udp' or 'as
> ASnum'
> Now i set filter 'any' and type 1:1, but results the same.
>
> I am not use nfcad. I use flow-capture, and
> pile up data in folder for profile 'live' after converting data by the
> utility ft2nfdump .
> The data processing occurs before the nfsen process is run.
> I repeat again -- the profile 'live' works fine.
> Maybe I am not understand correctly how must works a sub-profile.
> I think the nfprofile process apply filter for file which was stored in
> folder for profile 'live'
> and after that it is copying result file in to sub-profile directory.
>
> Is it right?
>
>
>
>
> 29.07.2012 13:27, Peter Haag пишет:
>> Make sure your rrd library and Perl module for RRD are linked correctly,
>> as well as the loader will find them.
>> for a 1:1 profile use filer 'any' an empty filter means 'not any'
>>
>> - Peter
>>
>>
>> On 23/7/12 1:19 PM, Anrey Teslenko wrote:
>>> Hello all,
>>>
>>> Let me to raise this issue once again. I've seen a lot of
>>> correspondences on this topic but has not found the answer to resolve
>>> it
>>>
>>> My nfdump: Version: 1.6.6 $Date: 2012-03-11 11:57:45 +0100 (Sun, 11 Mar 2012)
>>> My nfsen: Version: 1.3.6p1 $Id: nfsen 53 2012-01-23 16:36:02Z peter
>>>
>>> I have a problem when nftsen trying to collect the data from "live" profile into
>>> other new profile. All nfcapd files for all profiles are empty with a
>>> size of 312 bytes.
>>>
>>>
>>> This is a output of nfdump for this files.
>>>
>>> Date flow start Duration Proto Src IP Addr:Port
>>> Dst IP Addr:Port Packets Bytes Flows
>>> Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0,
>>> avg pps: 0, avg bpp: 0
>>> Time window: <unknown>
>>> Total flows processed: 0, Blocks skipped: 0, Bytes read: 36
>>> Sys: 0.000s flows/second: 0.0 Wall: 0.000s flows/second: 0.0
>>>
>>> I see zeros in rrd file.
>>> For appropriate file in profile live all looks good .
>>> I see next in log file
>>>
>>> Run periodic at Mon Jul 23 13:40:00 2012
>>> Prepare profiling './FlowStatistics'
>>> Prepare profiling './live'
>>> 1 channels/alerts to profile
>>> nfprofile run: /usr/local/bin/nfprofile -I -t 1343039700 -p
>>> /home/netflow/flows -P /var/www/nfsen/profiles-stat -L local3 -M
>>> /home/netflow/flows/live/upstreams -r nfcapd.201207231335
>>> Limit profilers: 1
>>> profile opts: .#FlowStatistics#2#AS6703#upstreams for profiler 0
>>> profiler 0 started
>>> comm child[16690] terminated with no exit value
>>> Process line '.#FlowStatistics#2#AS6703#upstreams#012'
>>> Setup channel 'AS6703' in profile 'FlowStatistics' group '.',
>>> channellist 'upstreams'
>>> profiler 0 finished
>>> Update profile FlowStatistics in group .
>>> Add channel size 20480
>>> Set new profile size: 20480
>>> Add .:FlowStatistics:201207231335 for plugin processing
>>>
>>>
>>> When i trying to start from command line
>>> /usr/local/bin/nfprofile -I -t 1343041440 -p /home/netflow/flows -P
>>> /var/www/php-cisco/profiles-stat -L local3 -M
>>> /home/netflow/flows/live/upstreams -r nfcapd.201207231402
>>>
>>> Process just sleep and do nothing (may be hung)
>>>
>>> www-data 22884 0.0 0.1 9312 1648 pts/1 S+ 14:06 0:00
>>> /usr/local/bin/nfprofile -I -t 1343039700 -p /home/netflow/flows -P
>>> /var/www/php-cisco/profiles-stat -L local3 -M
>>> /home/netflow/flows/live/upstreams -r nfcapd.201207231335
>>>
>>> I see next in syslog when i try push ENTER
>>>
>>> Jul 23 14:10:16 nettools nfprofile[22884]: Process line '#012'
>>> Jul 23 14:10:16 nettools nfprofile[22884]: Incomplete line - channel skipped.
>>> Jul 23 14:10:16 nettools nfprofile[22884]: Process line '#012'
>>> Jul 23 14:10:16 nettools nfprofile[22884]: Incomplete line - channel skipped.
>>> Jul 23 14:10:16 nettools nfprofile[22884]: Process line '#012'
>>> Jul 23 14:10:16 nettools nfprofile[22884]: Incomplete line - channel skipped.
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Nfsen-discuss mailing list
>>> Nfs...@li...
>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Nfsen-discuss mailing list
> Nfs...@li...
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
|
