Re: [Nfsen-discuss] Add asn number to existing data
Netflow visualisation and investigation tool
Brought to you by:
phaag
Menu
▾
▴
Re: [Nfsen-discuss] Add asn number to existing data
|
From: Adrian P. <adr...@gm...> - 2018-02-28 07:46:39
|
See man nfdump for output format options. You should add something like %srcas %dstas to display AS numbers. On Tue, Feb 20, 2018 at 10:17 PM, Leandro <ing...@gm...> wrote: > Dear Alan ... I would like to share my results. > So far can not get it work. > > What I did: > 1 Downloaded and installed "Net-NfDump-1.25" using cpan. ok > 2 Copy a nfdump generated file (this file is being used by nfsen for > graphics) ok > 3 Verify file content: ok > [root@AR-LXNF01 bin]$ /usr/local/bin/nfdump -r /usr/local/nfsen/profiles- > data/asn_updated/nfcapd.201802201310 | more > Date first seen Duration Proto Src IP Addr:Port Dst > IP Addr:Port Packets Bytes Flows > 2018-02-20 12:54:35.741 0.000 TCP 187.118.18.149:46938 -> > 169.60.79.184:5222 500 155500 1 > 2018-02-20 12:54:35.741 0.000 TCP 186.183.22.205:443 -> > 187.118.11.18:54343 500 750000 1 > 2018-02-20 12:54:35.742 0.000 UDP 187.174.42.129:3658 -> > 187.12.187.134:3658 500 33500 1 > 2018-02-20 12:54:35.745 0.000 UDP 187.118.10.47:55796 -> > 186.183.22.206:443 500 32000 1 > .....trunked > > 4 Apply perl script for asn update: ok > > /usr/src/Net-NfDump-1.25/bin/nfasnupd nfcapd.201802201310 > Subroutine main::pack_sockaddr_in6 redefined at > /usr/share/perl5/vendor_perl/Exporter.pm line 66. > at /usr/src/Net-NfDump-1.25/bin/nfasnupd line 8. > Subroutine main::unpack_sockaddr_in6 redefined at > /usr/share/perl5/vendor_perl/Exporter.pm line 66. > at /usr/src/Net-NfDump-1.25/bin/nfasnupd line 8. > Subroutine main::sockaddr_in6 redefined at /usr/share/perl5/vendor_perl/Exporter.pm > line 66. > at /usr/src/Net-NfDump-1.25/bin/nfasnupd line 8. > 2018-02-20.13:23:28[18171]: Checking BGP database for new version. > 2018-02-20.13:23:28[18171]: Loading AS database. > 2018-02-20.13:23:34[18171]: Updating records. > 2018-02-20.13:23:39[18171]: Processed 394412 flows in 5 secs. > > 4 Verify converted file content NOK > [root@AR-LXNF01 asn_updated]$ /usr/local/bin/nfdump -r > nfcapd.201802201310 > > Skip unknown record type 10 > > Skip unknown record type 10 > > Skip unknown record type 10 > ...trunked > > ############################After that I did: > 1 compiled nfdump ver 1.6.16 > 2 Read the file again: > > [leo@arch nfdump]$ nfdump -r nfcapd.201802201310 | more > Date first seen Duration Proto Src IP Addr:Port Dst > IP Addr:Port Packets Bytes Flows > 2018-02-20 13:09:03.614 27.624 UDP 187.183.47.57:56084 -> > 187.183.22.205:443 1500 105500 1 > 2018-02-20 13:09:10.518 28.752 TCP 198.38.124.206:80 -> > 181.118.2.92:50632 4500 6.6 M 1 > 2018-02-20 13:09:31.239 0.000 TCP 187.183.37.208:49822 -> > 172.217.28.163:80 500 26000 1 > 2018-02-20 13:09:21.275 19.923 TCP 187.183.22.204:443 -> > 187.183.21.174:6036 1500 2.2 M 1 > > As you can see with this newer version of nfdump I can read the file but > don't see the asn number. > > ############################ some debug > While reading the nfasnupd script found the following important paths: > > my $DBDIR = "/var/db/flowtools"; > my $CASNSDB = "/usr/local/etc/asns.txt"; # cuscom ASN DB > #This file was not here, I copied it from > /usr/src/Net-NfDump-1.25/bin/asns.txt > my $BGPDB = "/var/tmp/asns-$>.gz"; > #this file does not exist even while script is > working. > my $BGPDB_MD5 = "/var/tmp/asns-$>.gz.md5"; > #this file does not exist even while script is working. > my $GEO_DB4 = $DBDIR.'/geo/GeoIP.dat'; > #this file does not exist even while script is > working. > my $GEO_DB6 = $DBDIR.'/geo/GeoIPv6.dat'; > #this file does not exist even while script is working. > > my $FNAME_TMP = ".nffile_$$.tmp"; > my $DEBUG = 1; > my $FNAME; > my $VERSION = "1.10"; > > ###############So ... > I think those files are very important for the script to work propperly. > You you confirm if you can see those files ? > Also can you share a few lines of the nfdump output when displays properly > the asn number ? > btw ... my generated file is bigger than original. > > > Ok .... thats all ... thanks for the help , any idea would be preciated. > Leandro. > > > > > > > > > > On 09/02/18 03:58, Alan Whinery wrote: > > On 1/26/2018 4:52 AM, Leandro wrote: > > > Hi, Alan , thanks for the response. > It think the most difficult part of this is how to insert asn number > on the flow data structure. > First I need to read the data file , then iterate over all flows > (perhaps using pcap library ?) then take the ips involved in the > flow and insert their asn. > The idea is that this modification allow us to keep using nfsen > graphics and filter for further analisys. > I think it is necessary to have a very deep knowledge of the necessary > development tools. > Do you know some tool / api to analize and edit the flow data file at > a higher level ? > > After an uncanny sequence of events, I do, in fact have an answer to this. > > We have a flow analysis project going and not long after your question, > I was tasked with doing the very same thing here. It turns out that two > of our border routers are Juniper QFX, which appear only to export > SFlow, and they don't write ASNs in the records, even though they have > BGP tables. > > I found a Perl module https://metacpan.org/pod/Net::NfDump which has the > necessary capabilities. While I was in the process of getting something > working to insert AS numbers into flow files, I found that the example > program, "nfasnupd" not only inserts AS numbers, but also geo info, > which I had thought about as a next step! > > > Command updates nfdump file and adds AS and geoIP information > > Usage: > > nfasnupd [ -d <level> ] -b -g [ -a -5 -4 -6 ] [ -c <ASN_db_file> ] > <nfdump_file> > > Options: > > -d <level> : debug level (dafault: 1) > > -B do NOT update AS numbers (srcas, dstas) > -g update country code (*xsrcport, *xdstport) > > -a <file> : path to BGPDB file (default: /var/tmp/asns-0.gz) > -5 <file> : path to BGPDB MD5 (default: /var/tmp/asns-0.gz.md5) > -c <file> : path to additional textfile with ASN mapping (default: > /usr/local/etc/asns.txt) > -4 <file> : path to IPv4 GeoIP database (default: > /var/db/flowtools/geo/GeoIP.dat) > -6 <file> : path to IPv6 GeoIP database (default: > /var/db/flowtools/geo/GeoIPv6.dat) > > Part of libnf.net project, version: 1.10 > > > I am only at the stage of running it on sample files, not yet committing > to bulk operations, but it is promising. > > It does make the flow file smaller. I haven't yet figured out why. > > If you hand it a flow file, it effectively rewrites it in place > (probably writes a temp and then moves it to the original). > > the Geo stuff doesn't happen by default, so one only has to complete > dependencies and run > > ./nfasnupd flowfile.nfdump > > If you're not Perl-proficient, send me an off list message. I'm going to > try to move toward bulk conversion, with appropriate backups first. > > -Alan > > > Thanks, > Leo. > > > > > > On 26/01/18 00:08, Alan Whinery wrote: > > Of course, generally when you export flows from a BGP router with a full > table, it should already have ASNs populated. > > If you have flow data with no ASN, probably the easiest way to fill it > in would be to script something with MaxMind's open source ASN data: > https://www.maxmind.com/en/open-source-data-and-api-for-ip-geolocation > > I don't know off-hand of software that updates fields in nfdump files, > but there must be something out there, or some Perl or Python modules to > do so. > > In the past, I've rolled my own ASN-to-prefix cross-ref by grabbing the > global routing table from a BGP router and then annotating it with the > asn lists from cidr-report.org: > > http://www.cidr-report.org/as2.0/autnums.html > > which is linked from:http://www.cidr-report.org/as2.0/ > > > On 1/25/2018 5:37 AM, Leandro wrote: > > Hi guys , Im trying to analyze incoming traffic from an specific asn , > I can not filter this using source ip since this operator uses a lot > of subnets (about 7k). > My idea is to grab a flow file and insert the asn for further > analysis. Is there something about this ? > Any idea would help , > Regards , > Leo. > > > ------------------------------------------------------------------------------ > > > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfsen-discuss mailing lis...@li...://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfsen-discuss mailing lis...@li...://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfsen-discuss mailing lis...@li...://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfsen-discuss mailing lis...@li...://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfsen-discuss mailing list > Nfs...@li... > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > |
