Menu
▾
▴
openfpc-users — OpenFPC Discussion
You can subscribe to this list here.
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(6) |
Jun
|
Jul
|
Aug
(18) |
Sep
(17) |
Oct
(1) |
Nov
(10) |
Dec
(2) |
|---|
|
From: Leon W. <leo...@gm...> - 2015-12-03 17:37:21
|
Nice, thanks. I'll merge it in. -L On Thu, Dec 3, 2015 at 5:26 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Sorry for delaying the reply. > > > > I have tested the hard_time_window version and found it works fine. > > > > I have tested it with –last and also using curl with starttime and > endtime options. > > It returns the correct time window. > > > > Thanks so much for this fix!!! > > > > -Igor > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Monday, November 23, 2015 12:03 PM > > *To:* Igor Kaplan > *Cc:* <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Yup, > > > > On Mon, Nov 23, 2015 at 4:57 PM, Igor Kaplan <igo...@gm...> wrote: > > Leon, > > > > Thanks so much. I’ll take a look at this as soon as I can. Sounds really > good! > > > > So now, if I specify the exact time window it will return only packets > from that time frame?. > > > > Yup, that's the plan. All of my testing was using "--last", so please test > other options. For example --last 3 would only return 3 seconds of traffic. > > > > -L > > > > > > Many thanks. > > > > -Igor > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Monday, November 23, 2015 11:12 AM > > > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > Yeah, I did find time to poke at this over the weekend while recovering > from jetlag. > > > > It's in a branch called hard_time_window, and I've decided to enforce the > behaviour by default because it works pretty well. > > > > Before I merge it for everyone into the main branch, I'd appreciate it if > you could test it out. Any chance you could test the branch, and if it > functions I'll merge it. It is also configurable (but to exposed by > default) in the node config file, so if you need to go back to the old > behaviour you can set STRICT_TIME=0. The log will reflect if it is > enforcing it or not. > > > > It's working for me here. > > > > Cheers > > > > -L > > > > > > > > On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Just would like to ask you, if you had a chance to take a look at the > fix, which we talked below, regarding retrieving only data for the time > frame? > > > > Now I am facing the issue when the retrieved pcap file is so large,, so > the tshark is having problems to process it. > > > > Is there any way to reduce the size of pcap file which is fetched? > > > > Many thanks. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 12:25 PM > *To:* Igor Kaplan > *Cc:* ope...@li... > > > *Subject:* Re: fetch query time format. > > > > I don't think it's hard to do - I'll take a look sometime... > > > > -L > > > > > > On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > > > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-12-03 17:26:43
|
Hi Leon, Sorry for delaying the reply. I have tested the hard_time_window version and found it works fine. I have tested it with –last and also using curl with starttime and endtime options. It returns the correct time window. Thanks so much for this fix!!! -Igor From: Leon Ward [mailto:leo...@gm...] Sent: Monday, November 23, 2015 12:03 PM To: Igor Kaplan Cc: <ope...@li...> Subject: Re: fetch query time format. Yup, On Mon, Nov 23, 2015 at 4:57 PM, Igor Kaplan <igo...@gm...> wrote: Leon, Thanks so much. I’ll take a look at this as soon as I can. Sounds really good! So now, if I specify the exact time window it will return only packets from that time frame?. Yup, that's the plan. All of my testing was using "--last", so please test other options. For example --last 3 would only return 3 seconds of traffic. -L Many thanks. -Igor From: Leon Ward [mailto:leo...@gm...] Sent: Monday, November 23, 2015 11:12 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, Yeah, I did find time to poke at this over the weekend while recovering from jetlag. It's in a branch called hard_time_window, and I've decided to enforce the behaviour by default because it works pretty well. Before I merge it for everyone into the main branch, I'd appreciate it if you could test it out. Any chance you could test the branch, and if it functions I'll merge it. It is also configurable (but to exposed by default) in the node config file, so if you need to go back to the old behaviour you can set STRICT_TIME=0. The log will reflect if it is enforcing it or not. It's working for me here. Cheers -L On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Just would like to ask you, if you had a chance to take a look at the fix, which we talked below, regarding retrieving only data for the time frame? Now I am facing the issue when the retrieved pcap file is so large,, so the tshark is having problems to process it. Is there any way to reduce the size of pcap file which is fetched? Many thanks. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Jeremy H. <jt...@gm...> - 2015-11-23 20:06:21
|
Leon, did you have a chance to take a look at the chrome plugin? On Tue, Oct 27, 2015 at 6:41 AM, Leon Ward (leonward) <leo...@ci...> wrote: > Will take a look at the weekend… > > -L > > On 25 Oct 2015, at 22:03, Jeremy Hoel <jt...@gm...> wrote: > > So, I grabbed the chrome plugin from the git hub, turned on the advanced > extensions options and installed the plugin. I configured the settings and > I can generate searches, but I seem to be getting errors. > > 10.167 is the chrome browser making the request. > > Oct 25 15:57:16 razor OpenfpcQ[14968]: Default_Node COMMS: Accepted new > connection from 192.168.10.167 > Oct 25 15:57:16 razor OpenfpcQ[14968]: Default_Node COMMS: 192.168.10.167 > : Bad Request > > > and if I inspect the pop-up, I see these errors. > > Loading Data! > popup.js:52 Asking if there are any results to show.. > popup.js:67 Asking for Search Constraints > popup.js:36 Asking if there is an error to show.. > popup.js:57 And the response was... > popup.js:58 Object > popup.js:72 And the constraints are... > popup.js:73 Object > popup.js:41 And the error response was... > popup.js:42 Object > popup.js:123 Searching packets from popup > popup.js:146 The URL for request is > http://192.168.10.249:4242/api/1/search?apikey=removed&limit=20 > popup.js:153 Data is OFPC READY > > popup.html:1 Uncaught SyntaxError: Unexpected token > OxmlHttp.onreadystatechange @ popup.js:154 > > On the openfpc server side, I see the request packet, > > ........X....P.@)....GET > /api/1/search?apikey=1661402E-7B5D-11E5-B027-F6037414871C&sip=192.168.10.167&limit=20 > HTTP/1.1 > Host: 192.168.10.249:4242 > Connection: keep-alive > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, > like Gecko) Chrome/46.0.2490.71 Safari/537.36 > Accept: */* > Accept-Encoding: gzip, deflate, sdch > Accept-Language: en-US,en;q=0.8 > > > And then > > .............P....*..OFPC READY > > > I've restarted the restapi part, and checked that starman is running and > listening (it is) and the error log for that just shows the server starting > up, 4222 and 4223 and hat's about it. > > Any ideas or how I can help troubelshoot? > > ------------------------------------------------------------------------------ > _______________________________________________ > Openfpc-users mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > > |
|
From: Leon W. <leo...@gm...> - 2015-11-23 17:03:22
|
Yup, On Mon, Nov 23, 2015 at 4:57 PM, Igor Kaplan <igo...@gm...> wrote: > Leon, > > > > Thanks so much. I’ll take a look at this as soon as I can. Sounds really > good! > > > > So now, if I specify the exact time window it will return only packets > from that time frame?. > Yup, that's the plan. All of my testing was using "--last", so please test other options. For example --last 3 would only return 3 seconds of traffic. -L > > > Many thanks. > > > > -Igor > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Monday, November 23, 2015 11:12 AM > > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > Yeah, I did find time to poke at this over the weekend while recovering > from jetlag. > > > > It's in a branch called hard_time_window, and I've decided to enforce the > behaviour by default because it works pretty well. > > > > Before I merge it for everyone into the main branch, I'd appreciate it if > you could test it out. Any chance you could test the branch, and if it > functions I'll merge it. It is also configurable (but to exposed by > default) in the node config file, so if you need to go back to the old > behaviour you can set STRICT_TIME=0. The log will reflect if it is > enforcing it or not. > > > > It's working for me here. > > > > Cheers > > > > -L > > > > > > > > On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Just would like to ask you, if you had a chance to take a look at the > fix, which we talked below, regarding retrieving only data for the time > frame? > > > > Now I am facing the issue when the retrieved pcap file is so large,, so > the tshark is having problems to process it. > > > > Is there any way to reduce the size of pcap file which is fetched? > > > > Many thanks. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 12:25 PM > *To:* Igor Kaplan > *Cc:* ope...@li... > > > *Subject:* Re: fetch query time format. > > > > I don't think it's hard to do - I'll take a look sometime... > > > > -L > > > > > > On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-11-23 16:57:14
|
Leon, Thanks so much. I’ll take a look at this as soon as I can. Sounds really good! So now, if I specify the exact time window it will return only packets from that time frame?. Many thanks. -Igor From: Leon Ward [mailto:leo...@gm...] Sent: Monday, November 23, 2015 11:12 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, Yeah, I did find time to poke at this over the weekend while recovering from jetlag. It's in a branch called hard_time_window, and I've decided to enforce the behaviour by default because it works pretty well. Before I merge it for everyone into the main branch, I'd appreciate it if you could test it out. Any chance you could test the branch, and if it functions I'll merge it. It is also configurable (but to exposed by default) in the node config file, so if you need to go back to the old behaviour you can set STRICT_TIME=0. The log will reflect if it is enforcing it or not. It's working for me here. Cheers -L On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Just would like to ask you, if you had a chance to take a look at the fix, which we talked below, regarding retrieving only data for the time frame? Now I am facing the issue when the retrieved pcap file is so large,, so the tshark is having problems to process it. Is there any way to reduce the size of pcap file which is fetched? Many thanks. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-11-23 16:12:23
|
Hi, Yeah, I did find time to poke at this over the weekend while recovering from jetlag. It's in a branch called hard_time_window, and I've decided to enforce the behaviour by default because it works pretty well. Before I merge it for everyone into the main branch, I'd appreciate it if you could test it out. Any chance you could test the branch, and if it functions I'll merge it. It is also configurable (but to exposed by default) in the node config file, so if you need to go back to the old behaviour you can set STRICT_TIME=0. The log will reflect if it is enforcing it or not. It's working for me here. Cheers -L On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Just would like to ask you, if you had a chance to take a look at the > fix, which we talked below, regarding retrieving only data for the time > frame? > > > > Now I am facing the issue when the retrieved pcap file is so large,, so > the tshark is having problems to process it. > > > > Is there any way to reduce the size of pcap file which is fetched? > > > > Many thanks. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 12:25 PM > *To:* Igor Kaplan > *Cc:* ope...@li... > > *Subject:* Re: fetch query time format. > > > > I don't think it's hard to do - I'll take a look sometime... > > > > -L > > > > > > On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > > > |
|
From: Stefano P. <spi...@gm...> - 2015-11-23 00:42:33
|
Thanks for the confirmation Leon. Really appreciate it. On Sun, Nov 22, 2015 at 1:51 PM Leon Ward <le...@le...> wrote: > Hi, > > Jeremy is correct. If you've got the opportunity to use Ubuntu on the > version I work on you'll have much more luck. That's the current LTS > release. > > Cheers > > -L > > > On Wed, Nov 18, 2015 at 1:17 PM, Stefano Pirrello <spi...@gm...> > wrote: > >> Thanks for the feedback Jeremey. I'll have my guys reimage our server >> with Ubuntu instead. >> >> On Tue, Nov 17, 2015 at 11:57 PM, Jeremy Hoel <jt...@gm...> wrote: >> >>> You might get there.. you'll have to do a lot of compiling yourself, but >>> some of the new API features, the SOAP interface, it won't work. The >>> modules and applications to make it work don't exist in the repos and after >>> fighting it for two days I went and installed it on ubuntu. >>> >>> you can get the daemons and servers working, but you'll have to walk >>> through the process. The install script doesn't have the proper >>> recommendations to fix missing packages. It takes time. >>> >>> On Tue, Nov 17, 2015 at 10:29 PM, Stefano Pirrello <spi...@gm...> >>> wrote: >>> >>>> Hi, >>>> >>>> Does anyone have any experience with installing Openfpc on Red Hat and >>>> if so an you point me to a good site with instructions? >>>> >>>> Thanks! >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> Openfpc-users mailing list >>>> Ope...@li... >>>> https://lists.sourceforge.net/lists/listinfo/openfpc-users >>>> >>>> >>> >> >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> Openfpc-users mailing list >> Ope...@li... >> https://lists.sourceforge.net/lists/listinfo/openfpc-users >> >> > |
|
From: Leon W. <le...@le...> - 2015-11-22 18:51:51
|
Hi, Jeremy is correct. If you've got the opportunity to use Ubuntu on the version I work on you'll have much more luck. That's the current LTS release. Cheers -L On Wed, Nov 18, 2015 at 1:17 PM, Stefano Pirrello <spi...@gm...> wrote: > Thanks for the feedback Jeremey. I'll have my guys reimage our server > with Ubuntu instead. > > On Tue, Nov 17, 2015 at 11:57 PM, Jeremy Hoel <jt...@gm...> wrote: > >> You might get there.. you'll have to do a lot of compiling yourself, but >> some of the new API features, the SOAP interface, it won't work. The >> modules and applications to make it work don't exist in the repos and after >> fighting it for two days I went and installed it on ubuntu. >> >> you can get the daemons and servers working, but you'll have to walk >> through the process. The install script doesn't have the proper >> recommendations to fix missing packages. It takes time. >> >> On Tue, Nov 17, 2015 at 10:29 PM, Stefano Pirrello <spi...@gm...> >> wrote: >> >>> Hi, >>> >>> Does anyone have any experience with installing Openfpc on Red Hat and >>> if so an you point me to a good site with instructions? >>> >>> Thanks! >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Openfpc-users mailing list >>> Ope...@li... >>> https://lists.sourceforge.net/lists/listinfo/openfpc-users >>> >>> >> > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Openfpc-users mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > |
|
From: Stefano P. <spi...@gm...> - 2015-11-18 13:17:56
|
Thanks for the feedback Jeremey. I'll have my guys reimage our server with Ubuntu instead. On Tue, Nov 17, 2015 at 11:57 PM, Jeremy Hoel <jt...@gm...> wrote: > You might get there.. you'll have to do a lot of compiling yourself, but > some of the new API features, the SOAP interface, it won't work. The > modules and applications to make it work don't exist in the repos and after > fighting it for two days I went and installed it on ubuntu. > > you can get the daemons and servers working, but you'll have to walk > through the process. The install script doesn't have the proper > recommendations to fix missing packages. It takes time. > > On Tue, Nov 17, 2015 at 10:29 PM, Stefano Pirrello <spi...@gm...> > wrote: > >> Hi, >> >> Does anyone have any experience with installing Openfpc on Red Hat and if >> so an you point me to a good site with instructions? >> >> Thanks! >> >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> Openfpc-users mailing list >> Ope...@li... >> https://lists.sourceforge.net/lists/listinfo/openfpc-users >> >> > |
|
From: Jeremy H. <jt...@gm...> - 2015-11-18 04:57:43
|
You might get there.. you'll have to do a lot of compiling yourself, but some of the new API features, the SOAP interface, it won't work. The modules and applications to make it work don't exist in the repos and after fighting it for two days I went and installed it on ubuntu. you can get the daemons and servers working, but you'll have to walk through the process. The install script doesn't have the proper recommendations to fix missing packages. It takes time. On Tue, Nov 17, 2015 at 10:29 PM, Stefano Pirrello <spi...@gm...> wrote: > Hi, > > Does anyone have any experience with installing Openfpc on Red Hat and if > so an you point me to a good site with instructions? > > Thanks! > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Openfpc-users mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > |
|
From: Stefano P. <spi...@gm...> - 2015-11-18 03:29:41
|
Hi, Does anyone have any experience with installing Openfpc on Red Hat and if so an you point me to a good site with instructions? Thanks! |
|
From: Igor K. <igo...@gm...> - 2015-11-11 16:30:26
|
Hi Leon, Just would like to ask you, if you had a chance to take a look at the fix, which we talked below, regarding retrieving only data for the time frame? Now I am facing the issue when the retrieved pcap file is so large,, so the tshark is having problems to process it. Is there any way to reduce the size of pcap file which is fetched? Many thanks. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Jeremy H. <jt...@gm...> - 2015-10-25 22:03:52
|
So, I grabbed the chrome plugin from the git hub, turned on the advanced extensions options and installed the plugin. I configured the settings and I can generate searches, but I seem to be getting errors. 10.167 is the chrome browser making the request. Oct 25 15:57:16 razor OpenfpcQ[14968]: Default_Node COMMS: Accepted new connection from 192.168.10.167 Oct 25 15:57:16 razor OpenfpcQ[14968]: Default_Node COMMS: 192.168.10.167 : Bad Request and if I inspect the pop-up, I see these errors. Loading Data! popup.js:52 Asking if there are any results to show.. popup.js:67 Asking for Search Constraints popup.js:36 Asking if there is an error to show.. popup.js:57 And the response was... popup.js:58 Object popup.js:72 And the constraints are... popup.js:73 Object popup.js:41 And the error response was... popup.js:42 Object popup.js:123 Searching packets from popup popup.js:146 The URL for request is http://192.168.10.249:4242/api/1/search?apikey=removed&limit=20 popup.js:153 Data is OFPC READY popup.html:1 Uncaught SyntaxError: Unexpected token OxmlHttp.onreadystatechange @ popup.js:154 On the openfpc server side, I see the request packet, ........X....P.@)....GET /api/1/search?apikey=1661402E-7B5D-11E5-B027-F6037414871C&sip=192.168.10.167&limit=20 HTTP/1.1 Host: 192.168.10.249:4242 Connection: keep-alive User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36 Accept: */* Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8 And then .............P....*..OFPC READY I've restarted the restapi part, and checked that starman is running and listening (it is) and the error log for that just shows the server starting up, 4222 and 4223 and hat's about it. Any ideas or how I can help troubelshoot? |
|
From: Igor K. <igo...@gm...> - 2015-09-29 20:13:38
|
Leon, Many many thanks! From: Leon Ward [mailto:leo...@gm...] Sent: Tuesday, September 29, 2015 4:09 PM To: Igor Kaplan Cc: <ope...@li...> Subject: Re: fetch query time format. Sure, Use openfpc-dbmaint. Something /like/ the below should do the trick. 1) Stop OpenFPC sudo openfpc -a stop 2) Drop the session database openfpc-dbmaint -a drop -t session -c /etc/openfpc/openfpc-<nodename>.conf 3) recreate the session db openfpc-dbmaint -a create -t session -c /etc/openfpc/openfpc-<nodename>.conf 4) Start again sudo openfpc -a start If you wanted to remove all of the pcap files, you could purge your install and re-install from scratch instead of option #3 sudo ./openfpc-install.sh purge sudo ./openfpc-install.sh install -Leon On Tue, Sep 29, 2015 at 12:32 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Is there any easy way to clean up mysql database tables and start everything fresh? I am having some strange problems, so would like to start everything from beginning. Thanks so much. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-09-29 20:08:59
|
Sure, Use openfpc-dbmaint. Something /like/ the below should do the trick. 1) Stop OpenFPC sudo openfpc -a stop 2) Drop the session database openfpc-dbmaint -a drop -t session -c /etc/openfpc/openfpc-<nodename>.conf 3) recreate the session db openfpc-dbmaint -a create -t session -c /etc/openfpc/openfpc-<nodename>.conf 4) Start again sudo openfpc -a start If you wanted to remove all of the pcap files, you could purge your install and re-install from scratch instead of option #3 sudo ./openfpc-install.sh purge sudo ./openfpc-install.sh install -Leon On Tue, Sep 29, 2015 at 12:32 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Is there any easy way to clean up mysql database tables and start > everything fresh? > > I am having some strange problems, so would like to start everything > from beginning. > > > > Thanks so much. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-29 19:33:11
|
Hi Leon, Is there any easy way to clean up mysql database tables and start everything fresh? I am having some strange problems, so would like to start everything from beginning. Thanks so much. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Igor K. <igo...@gm...> - 2015-09-18 16:34:49
|
Would be really nice!!! Thanks so much!!!! From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-09-18 16:25:06
|
I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-18 15:15:23
|
HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-09-18 13:55:06
|
Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-17 17:08:38
|
Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Igor K. <igo...@gm...> - 2015-09-16 16:54:17
|
Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Igor K. <igo...@gm...> - 2015-09-11 13:23:35
|
Hi Leon,
Hmm, my filter.bpf is very simple:
Port 502
I would like only packets from and to port 502 to be captured.
Will try to run today another set of tests.
Have a nice weekend!
-Igor
From: Leon Ward [mailto:leo...@gm...]
Sent: Friday, September 11, 2015 5:46 AM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: BPF filter
Everything there looks like it should be working. What is the contents of /var/filter.bpf?
$ cat /var/filter.bpf
-L
On 10 Sep 2015, at 20:23, Igor Kaplan <igo...@gm...> wrote:
Hi Leon,
Here is the output which you asked:
- openfpc-daemonlogger is /usr/bin/daemonlogger
BPF file: /var/filter.bpf
Using BPF
DEBUG: Found BPF file /var/filter.bpf - openfpc-daemonlogger is /usr/bin/daemonlogger
BPF file: /var/filter.bpf
Using BPF
DEBUG: Found BPF file /var/filter.bpfStopping Daemonlogger... Done
- openfpc-daemonlogger is /usr/bin/daemonlogger
BPF file: /var/filter.bpf
Using BPF
DEBUG: Found BPF file /var/filter.bpfDEBUG: Command used to start daemonlogger is
/usr/bin/daemonlogger -d -f /var/filter.bpf -i eth1 -l /var/tmp/openfpc/pcap -M 50 -s 1G -p openfpc-daemonlogger.pid -P /var/run/openfpc-Default_Node -n openfpc-Default_Node.pcap -u openfpc -g openfpc 2>&1 | logger -t OFPC-DL-Default_Node
Starting Daemonlogger (Default_Node)...
DEBUG: Touching a canary file to check if DL needs to delete something at startup:/var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.0
Done
DEBUG: canary file still here (good)# BPF_FILE: An optional BPF filter to use while capturing traffic.
grep BPF_FILE /etc/openfpc/*.conf
BPF_FILE=/var/filter.bpf
ps aux |grep daemonlogger
openfpc 31867 0.0 0.1 19252 4304 ? Ss 15:06 0:00 /usr/bin/daemonlogger -d -f /var/filter.bpf -i eth1 -l /var/tmp/openfpc/pcap -M 50 -s 1G -p openfpc-daemonlogger.pid -P /var/run/openfpc-Default_Node -n openfpc-Default_Node.pcap -u openfpc -g openfpc
root 31894 0.0 0.0 10468 2136 pts/0 S+ 15:08 0:00
And here is the syslog
Sep 10 15:06:22 ikaplan-DH-2 daemonlogger[1724]: Quitting!
Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: sniffing on interface eth1
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Checking partition stats for log directory "/var/tmp/openfpc/pcap/."
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: 50% max disk utilization = 1885056 blocks free (out of 3770112)
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Blocksize = 4096
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Rollsize = 262144 blocks
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node:
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Daemon mode set
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Reading BPF filter in from file /var/filter.bpf
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Setting group ID to openfpc
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Interface set to eth1
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Logpath set to /var/tmp/openfpc/pcap
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Log filename set to "openfpc-Default_Node.pcap"
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pidfile configured to "openfpc-daemonlogger.pid"
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pidpath configured to "/var/run/openfpc-Default_Node"
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Rollover configured for 1 gigabytes
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Rollover configured for 0 none
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Setting user ID to openfpc
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pruning behavior set to oldest IN DIRECTORY
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node:
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: -*> DaemonLogger <*-
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Version 1.2.1
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: By Martin Roesch
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: (C) Copyright 2006-2007 Sourcefire Inc., All rights reserved
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node:
Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: start_sniffing() device eth1 network lookup: #011eth1: no IPv4 address assigned
Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: Logging packets to /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1441911983
Please let me know if there any other tests I should make.
Many thanks!!!
Igor.
From: leo...@gm... [mailto:leo...@gm...] On Behalf Of Leon Ward
Sent: Thursday, September 10, 2015 2:41 PM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: BPF filter
Okay, so just tested it and it looked like it works for me...
Can you please send the output of....
$ sudo openfpc -v -t openfpc-daemonlogger -a restart
$ grep BPF_FILE /etc/openfpc/*.conf
$ ps aux |grep daemonlogger
There should also be a chunk of data about daemonlogger in your syslog...
On Tue, Sep 8, 2015 at 4:14 PM, Igor Kaplan <igo...@gm...> wrote:
Leon,
Thanks so much, really appreciate!
From: leo...@gm... [mailto:leo...@gm...] On Behalf Of Leon Ward
Sent: Tuesday, September 08, 2015 9:44 AM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: BPF filter
Will take a look, can't from where I am right now.
Its likely that I've broken something years back, It used to work as I once used it a lot myself, but no longer.
On Tue, Sep 8, 2015 at 2:42 PM, Igor Kaplan <igo...@gm...> wrote:
Hi Leon,
I am still having problems with the input bpf filter. I wonder, if you
could please answer my question below.
Not sure, if I am doing everything correctly, however as my best
understanding, I do.
Many thanks.
-Igor.
-----Original Message-----
From: Igor Kaplan [mailto:igo...@gm...]
Sent: Friday, August 28, 2015 4:32 PM
To: 'Leon Ward'
Cc: ope...@li...
Subject: BPF filter
Hi Leon,
Not sure, if I am doing everything correctly, however my BPF filter in
openfpc-default.conf does not look to be working.
I have a line in my openfpc-default.conf:
BPF_File=/var/filter.bpf
While starting openfpc -a start --verbose I am able to see, the proper bpf
file is found and loaded.
My filter.bpf is very simple:
Port 502
So it should capture packets from port 502 only
However after fetching the data I still see packets from other ports.
I wonder, do I understand correctly, the BPF_FILE in the config file will
restrict, which packets are captured? So using the bpf file as above I
should not see any packets from other ports beside 502? Or it is something
different?
Many thanks and have a nice weekend!
-Igor
-----Original Message-----
From: Leon Ward [mailto:leo...@gm...] On Behalf Of Leon Ward
Sent: Tuesday, August 25, 2015 4:59 PM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: [Openfpc-users] Openfpc-users Digest, Vol 2, Issue 4
That's where the pcaps should live, and they will grow to the max percentage
that you allow in the node config.
What does an openfpc-client -a status show?
Also what's a df -h look like?
The pcaps will auto-prune unless something has gone wrong along the way....
Thinking out loud, what's an ls of your pcaps directory look like? Have you
got multiple nodes running on one box?
-L
Sent from a mobile device. Apologies for any typos but they happen.
> On 25 Aug 2015, at 16:25, Igor Kaplan <igo...@gm...> wrote:
>
> Hi Leon,
>
> Could you please help me with following.
> I am running openfpc for several days already and now I am out of
> space on my Ubuntu box which runs openfpc Under /var/tmp/openfpc I see
> directories, some of which contain number of large files:
> api-pcaps extracted pcap session
>
> I wonder, can I safely delete data under any of those directories above?
> Could you please let me know, which directory I can empty without
> breaking openfpc functionality?
>
> Is there any way to clean all captured data and start fresh?
>
> Many thanks.
>
> -Igor.
>
>
> -----Original Message-----
> From: ope...@li...
> [mailto:ope...@li...]
> Sent: Thursday, August 20, 2015 8:01 PM
> To: ope...@li...
> Subject: Openfpc-users Digest, Vol 2, Issue 4
>
> Send Openfpc-users mailing list submissions to
> ope...@li...
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
> or, via email, send a message with subject or body 'help' to
> ope...@li...
>
> You can reach the person managing the list at
> ope...@li...
>
> When replying, please edit your Subject line so it is more specific
> than
> "Re: Contents of Openfpc-users digest..."
>
>
> Today's Topics:
>
> 1. Re: Openfpc usage (Igor Kaplan)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 20 Aug 2015 20:01:08 -0400
> From: "Igor Kaplan" <igo...@gm...>
> Subject: Re: [Openfpc-users] Openfpc usage
> To: <ope...@li...>
> Message-ID: <000c01d0dba4$7c3c9910$74b5cb30$@gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Also sending my reply to the list, sorry, forgot to include it.
>
>
>
>
>
> From: Igor Kaplan [mailto:igo...@gm...]
> Sent: Thursday, August 20, 2015 6:44 PM
> To: 'Leon Ward'
> Subject: RE: [Openfpc-users] Openfpc usage
>
>
>
> Version of mergecap:
>
> Mergecap 1.10.6 (v1.10.6 from master-1.10)
>
>
>
> Linux is Ubuntu 14.04.2 LTS
>
>
>
> File list.pcap
>
> list.pcap: pcap-ng capture file - version 1.0
>
>
>
> Thanks.
>
>
>
> From: leo...@gm... [mailto:leo...@gm...] On Behalf
> Of Leon Ward
> Sent: Thursday, August 20, 2015 6:23 PM
> To: Igor Kaplan
> Cc: ope...@li...
> Subject: Re: [Openfpc-users] Openfpc usage
>
>
>
> What's your platform, version of mergecap etc.
>
> Also, if you '$ file list.pcap' what does it say?
>
>
>
> -L
>
>
>
>
>
> On Thu, Aug 20, 2015 at 10:15 PM, Igor Kaplan <igo...@gm...>
wrote:
>
> Hi Leon,
>
>
>
> Need your help please once again.
>
> Got the following problem and spent several hours trying to solve it.
>
>
>
> When making the API call to fetch the pcap data I am getting the data
> in pcapng format.
>
> The OpenFPC is using the mergecap to merge pcap files and by default
> mergecap creates the output in pcapng format instead of pcap.
>
>
>
> I have changed the following line in openfpc-default.conf file
>
> MERGECAP=/usr/bin/mergecap -F pcap
>
>
>
> This helped when I use the openfpc-client command to create pcap
> files, however when I use curl to fetch the data I still receive the
> output in pcapng format.
>
>
>
> curl -k
> 192.168.145.20:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C
> E8A48\
> <http://192.168.145.20:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F
> -C061B
> 4CE8A48%5C&stime=20150818%2010:00%5C&etime=20150818%2010:30%5C&dpt=22>
> &stime=20150818%2010:00\&etime=20150818%2010:30\&dpt=22 > list.pcap
>
>
>
> cat list.pcap|tshark -i-
>
> Capturing on 'Standard input'
>
> tshark: Unrecognized libpcap format
>
>
>
> Looks like in case of API call the mergecap utility is not used at all.
> And I was not able to find in the code how merging is done in this case.
>
>
>
> Could you please help me. Is it possible to make the fetch API call
> to return the data in pcap format?
>
>
>
> Thanks so much!
>
>
>
> Igor
>
>
>
>
>
> From: leo...@gm... [mailto:leo...@gm...] On Behalf
> Of Leon Ward
> Sent: Tuesday, August 18, 2015 1:29 PM
>
>
> To: Igor Kaplan
> Cc: ope...@li...
> Subject: Re: [Openfpc-users] Openfpc usage
>
>
>
> Actually it wont. It will only remove the oldest PCAP file. It's best
> to keep those PCAP files on their own partition.
>
> The old flow records in mysql actually get removed automatically based
> on the oldest packet in the store. So you won't have records that are
> older than the pcaps.
>
>
>
> -L
>
>
>
>
>
> On Tue, Aug 18, 2015 at 6:21 PM, Igor Kaplan <igo...@gm...>
wrote:
>
> Hi Leon, all,
>
>
>
> I have one more question please.
>
>
>
> Based on the documentation the following line in the openfpc config
> file restricts the space usage of captured data to 50 percent:
>
> PCAP_SPACE=50
>
>
>
> So, if the data size exceeds 50 percent old files will be deleted
> automatically?
>
> Will openfpc also delete the old MySQL session tables?
>
>
>
> Many thanks and all the best!
>
>
>
> -Igor.
>
>
>
>
>
> From: leo...@gm... [mailto:leo...@gm...] On Behalf
> Of Leon Ward
> Sent: Monday, August 17, 2015 11:51 AM
> To: Igor Kaplan
> Cc: ope...@li...
> Subject: Re: [Openfpc-users] Openfpc usage
>
>
>
> Hi,
>
>
>
> Documentation is really one of the places that really needs some extra
> focus.
>
>
>
> The best docs I can point you to are in that folder, plus there is
> some out-of date info on my blog http://www.leonward.com.
>
> I actually delivered a presentation at Defcon last weekend all about
> OpenFPC. I have forwarded the slides separately. Hopefully that will
> help as well.
>
>
>
> As for your specific question about OpenFPC GUI. That's actually now
> been deprecated as it's no longer relevant for how it functions in a
> distributed manner. The OpenFPC-Chrome Extension will be the next best
> thing for interacting with the QueueDaemon remotely in a GUI-like way.
>
>
>
> Cheers,
>
>
>
> -L
>
>
>
>
>
>
>
> On Mon, Aug 17, 2015 at 4:25 PM, Igor Kaplan <igo...@gm...>
wrote:
>
> Hi All,
>
>
>
> My name is Igor. I just found the OpenFPC and evaluating it. Looks
> like it is very good tool.
>
> I successfully installed on Ubuntu 14.4 with Perl 5.18
>
> I have installed the OpenFPC-master, so it is the latest code.
>
>
>
> Now I would like to find out if there is more documentation beside
> files which I could find under docs directory.
>
> For example the INSTALL.md refers to the USAGE document, however I
> was not able to find it anywhere
>
>
>
> I am looking for the usage other then basic, just to find out, what
> are my advanced options.
>
>
>
> For example the openfpc-dbmaint.sh script is also able to create the
> gui database, I wonder, what it is for?
>
>
>
> The OpenFPC looks to be very powerful, just would like to understand
> it as best as I can.
>
>
>
> Would so much appreciate any reply?s.
>
>
>
> Many thanks.
>
>
>
> Igor.
>
>
>
>
> ----------------------------------------------------------------------
> ------
> --
>
> _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
>
>
>
>
> ----------------------------------------------------------------------
> ------
> --
>
> _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
>
>
>
>
>
> -------------- next part -------------- An HTML attachment was
> scrubbed...
>
> ------------------------------
>
> ----------------------------------------------------------------------
> ------
> --
>
>
> ------------------------------
>
> _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
>
>
> End of Openfpc-users Digest, Vol 2, Issue 4
> *******************************************
>
>
> ----------------------------------------------------------------------
> -------- _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
|
|
From: Leon W. <leo...@gm...> - 2015-09-11 09:46:42
|
Everything there looks like it should be working. What is the contents of /var/filter.bpf? $ cat /var/filter.bpf -L > On 10 Sep 2015, at 20:23, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > Here is the output which you asked: > > - openfpc-daemonlogger is /usr/bin/daemonlogger > BPF file: /var/filter.bpf > Using BPF > DEBUG: Found BPF file /var/filter.bpf - openfpc-daemonlogger is /usr/bin/daemonlogger > BPF file: /var/filter.bpf > Using BPF > DEBUG: Found BPF file /var/filter.bpfStopping Daemonlogger... Done > - openfpc-daemonlogger is /usr/bin/daemonlogger > BPF file: /var/filter.bpf > Using BPF > DEBUG: Found BPF file /var/filter.bpfDEBUG: Command used to start daemonlogger is > /usr/bin/daemonlogger -d -f /var/filter.bpf -i eth1 -l /var/tmp/openfpc/pcap -M 50 -s 1G -p openfpc-daemonlogger.pid -P /var/run/openfpc-Default_Node -n openfpc-Default_Node.pcap -u openfpc -g openfpc 2>&1 | logger -t OFPC-DL-Default_Node > Starting Daemonlogger (Default_Node)... > DEBUG: Touching a canary file to check if DL needs to delete something at startup:/var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.0 > Done > DEBUG: canary file still here (good)# BPF_FILE: An optional BPF filter to use while capturing traffic. > > grep BPF_FILE /etc/openfpc/*.conf > BPF_FILE=/var/filter.bpf > > ps aux |grep daemonlogger > openfpc 31867 0.0 0.1 19252 4304 ? Ss 15:06 0:00 /usr/bin/daemonlogger -d -f /var/filter.bpf -i eth1 -l /var/tmp/openfpc/pcap -M 50 -s 1G -p openfpc-daemonlogger.pid -P /var/run/openfpc-Default_Node -n openfpc-Default_Node.pcap -u openfpc -g openfpc > root 31894 0.0 0.0 10468 2136 pts/0 S+ 15:08 0:00 > > And here is the syslog > > Sep 10 15:06:22 ikaplan-DH-2 daemonlogger[1724]: Quitting! > Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: sniffing on interface eth1 > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Checking partition stats for log directory "/var/tmp/openfpc/pcap/." > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: 50% max disk utilization = 1885056 blocks free (out of 3770112) > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Blocksize = 4096 > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Rollsize = 262144 blocks > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Daemon mode set > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Reading BPF filter in from file /var/filter.bpf > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Setting group ID to openfpc > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Interface set to eth1 > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Logpath set to /var/tmp/openfpc/pcap > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Log filename set to "openfpc-Default_Node.pcap" > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pidfile configured to "openfpc-daemonlogger.pid" > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pidpath configured to "/var/run/openfpc-Default_Node" > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Rollover configured for 1 gigabytes > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Rollover configured for 0 none > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Setting user ID to openfpc > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pruning behavior set to oldest IN DIRECTORY > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: -*> DaemonLogger <*- > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Version 1.2.1 > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: By Martin Roesch > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: (C) Copyright 2006-2007 Sourcefire Inc., All rights reserved > Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: > Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: start_sniffing() device eth1 network lookup: #011eth1: no IPv4 address assigned > Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: Logging packets to /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1441911983 > > > Please let me know if there any other tests I should make. > > Many thanks!!! > > Igor. > > From: leo...@gm... [mailto:leo...@gm...] On Behalf Of Leon Ward > Sent: Thursday, September 10, 2015 2:41 PM > To: Igor Kaplan > Cc: <ope...@li...> > Subject: Re: BPF filter > > Okay, so just tested it and it looked like it works for me... > > Can you please send the output of.... > > $ sudo openfpc -v -t openfpc-daemonlogger -a restart > $ grep BPF_FILE /etc/openfpc/*.conf > $ ps aux |grep daemonlogger > > There should also be a chunk of data about daemonlogger in your syslog... > > > > > > On Tue, Sep 8, 2015 at 4:14 PM, Igor Kaplan <igo...@gm...> wrote: > Leon, > > Thanks so much, really appreciate! > > From: leo...@gm... [mailto:leo...@gm...] On Behalf Of Leon Ward > Sent: Tuesday, September 08, 2015 9:44 AM > To: Igor Kaplan > Cc: <ope...@li...> > Subject: Re: BPF filter > > Will take a look, can't from where I am right now. > Its likely that I've broken something years back, It used to work as I once used it a lot myself, but no longer. > > On Tue, Sep 8, 2015 at 2:42 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > I am still having problems with the input bpf filter. I wonder, if you > could please answer my question below. > Not sure, if I am doing everything correctly, however as my best > understanding, I do. > > Many thanks. > > -Igor. > > -----Original Message----- > From: Igor Kaplan [mailto:igo...@gm...] > Sent: Friday, August 28, 2015 4:32 PM > To: 'Leon Ward' > Cc: ope...@li... > Subject: BPF filter > > Hi Leon, > > Not sure, if I am doing everything correctly, however my BPF filter in > openfpc-default.conf does not look to be working. > I have a line in my openfpc-default.conf: > BPF_File=/var/filter.bpf > > While starting openfpc -a start --verbose I am able to see, the proper bpf > file is found and loaded. > > My filter.bpf is very simple: > Port 502 > > So it should capture packets from port 502 only > > However after fetching the data I still see packets from other ports. > > I wonder, do I understand correctly, the BPF_FILE in the config file will > restrict, which packets are captured? So using the bpf file as above I > should not see any packets from other ports beside 502? Or it is something > different? > > Many thanks and have a nice weekend! > > -Igor > > -----Original Message----- > From: Leon Ward [mailto:leo...@gm...] On Behalf Of Leon Ward > Sent: Tuesday, August 25, 2015 4:59 PM > To: Igor Kaplan > Cc: <ope...@li...> > Subject: Re: [Openfpc-users] Openfpc-users Digest, Vol 2, Issue 4 > > That's where the pcaps should live, and they will grow to the max percentage > that you allow in the node config. > > What does an openfpc-client -a status show? > > Also what's a df -h look like? > > The pcaps will auto-prune unless something has gone wrong along the way.... > > Thinking out loud, what's an ls of your pcaps directory look like? Have you > got multiple nodes running on one box? > > -L > > Sent from a mobile device. Apologies for any typos but they happen. > > > On 25 Aug 2015, at 16:25, Igor Kaplan <igo...@gm...> wrote: > > > > Hi Leon, > > > > Could you please help me with following. > > I am running openfpc for several days already and now I am out of > > space on my Ubuntu box which runs openfpc Under /var/tmp/openfpc I see > > directories, some of which contain number of large files: > > api-pcaps extracted pcap session > > > > I wonder, can I safely delete data under any of those directories above? > > Could you please let me know, which directory I can empty without > > breaking openfpc functionality? > > > > Is there any way to clean all captured data and start fresh? > > > > Many thanks. > > > > -Igor. > > > > > > -----Original Message----- > > From: ope...@li... > > [mailto:ope...@li...] > > Sent: Thursday, August 20, 2015 8:01 PM > > To: ope...@li... > > Subject: Openfpc-users Digest, Vol 2, Issue 4 > > > > Send Openfpc-users mailing list submissions to > > ope...@li... > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > or, via email, send a message with subject or body 'help' to > > ope...@li... > > > > You can reach the person managing the list at > > ope...@li... > > > > When replying, please edit your Subject line so it is more specific > > than > > "Re: Contents of Openfpc-users digest..." > > > > > > Today's Topics: > > > > 1. Re: Openfpc usage (Igor Kaplan) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Thu, 20 Aug 2015 20:01:08 -0400 > > From: "Igor Kaplan" <igo...@gm...> > > Subject: Re: [Openfpc-users] Openfpc usage > > To: <ope...@li...> > > Message-ID: <000c01d0dba4$7c3c9910$74b5cb30$@gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Also sending my reply to the list, sorry, forgot to include it. > > > > > > > > > > > > From: Igor Kaplan [mailto:igo...@gm...] > > Sent: Thursday, August 20, 2015 6:44 PM > > To: 'Leon Ward' > > Subject: RE: [Openfpc-users] Openfpc usage > > > > > > > > Version of mergecap: > > > > Mergecap 1.10.6 (v1.10.6 from master-1.10) > > > > > > > > Linux is Ubuntu 14.04.2 LTS > > > > > > > > File list.pcap > > > > list.pcap: pcap-ng capture file - version 1.0 > > > > > > > > Thanks. > > > > > > > > From: leo...@gm... [mailto:leo...@gm...] On Behalf > > Of Leon Ward > > Sent: Thursday, August 20, 2015 6:23 PM > > To: Igor Kaplan > > Cc: ope...@li... > > Subject: Re: [Openfpc-users] Openfpc usage > > > > > > > > What's your platform, version of mergecap etc. > > > > Also, if you '$ file list.pcap' what does it say? > > > > > > > > -L > > > > > > > > > > > > On Thu, Aug 20, 2015 at 10:15 PM, Igor Kaplan <igo...@gm...> > wrote: > > > > Hi Leon, > > > > > > > > Need your help please once again. > > > > Got the following problem and spent several hours trying to solve it. > > > > > > > > When making the API call to fetch the pcap data I am getting the data > > in pcapng format. > > > > The OpenFPC is using the mergecap to merge pcap files and by default > > mergecap creates the output in pcapng format instead of pcap. > > > > > > > > I have changed the following line in openfpc-default.conf file > > > > MERGECAP=/usr/bin/mergecap -F pcap > > > > > > > > This helped when I use the openfpc-client command to create pcap > > files, however when I use curl to fetch the data I still receive the > > output in pcapng format. > > > > > > > > curl -k > > 192.168.145.20:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C > > E8A48\ > > <http://192.168.145.20:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F > > -C061B > > 4CE8A48%5C&stime=20150818%2010:00%5C&etime=20150818%2010:30%5C&dpt=22> > > &stime=20150818%2010:00\&etime=20150818%2010:30\&dpt=22 > list.pcap > > > > > > > > cat list.pcap|tshark -i- > > > > Capturing on 'Standard input' > > > > tshark: Unrecognized libpcap format > > > > > > > > Looks like in case of API call the mergecap utility is not used at all. > > And I was not able to find in the code how merging is done in this case. > > > > > > > > Could you please help me. Is it possible to make the fetch API call > > to return the data in pcap format? > > > > > > > > Thanks so much! > > > > > > > > Igor > > > > > > > > > > > > From: leo...@gm... [mailto:leo...@gm...] On Behalf > > Of Leon Ward > > Sent: Tuesday, August 18, 2015 1:29 PM > > > > > > To: Igor Kaplan > > Cc: ope...@li... > > Subject: Re: [Openfpc-users] Openfpc usage > > > > > > > > Actually it wont. It will only remove the oldest PCAP file. It's best > > to keep those PCAP files on their own partition. > > > > The old flow records in mysql actually get removed automatically based > > on the oldest packet in the store. So you won't have records that are > > older than the pcaps. > > > > > > > > -L > > > > > > > > > > > > On Tue, Aug 18, 2015 at 6:21 PM, Igor Kaplan <igo...@gm...> > wrote: > > > > Hi Leon, all, > > > > > > > > I have one more question please. > > > > > > > > Based on the documentation the following line in the openfpc config > > file restricts the space usage of captured data to 50 percent: > > > > PCAP_SPACE=50 > > > > > > > > So, if the data size exceeds 50 percent old files will be deleted > > automatically? > > > > Will openfpc also delete the old MySQL session tables? > > > > > > > > Many thanks and all the best! > > > > > > > > -Igor. > > > > > > > > > > > > From: leo...@gm... [mailto:leo...@gm...] On Behalf > > Of Leon Ward > > Sent: Monday, August 17, 2015 11:51 AM > > To: Igor Kaplan > > Cc: ope...@li... > > Subject: Re: [Openfpc-users] Openfpc usage > > > > > > > > Hi, > > > > > > > > Documentation is really one of the places that really needs some extra > > focus. > > > > > > > > The best docs I can point you to are in that folder, plus there is > > some out-of date info on my blog http://www.leonward.com. > > > > I actually delivered a presentation at Defcon last weekend all about > > OpenFPC. I have forwarded the slides separately. Hopefully that will > > help as well. > > > > > > > > As for your specific question about OpenFPC GUI. That's actually now > > been deprecated as it's no longer relevant for how it functions in a > > distributed manner. The OpenFPC-Chrome Extension will be the next best > > thing for interacting with the QueueDaemon remotely in a GUI-like way. > > > > > > > > Cheers, > > > > > > > > -L > > > > > > > > > > > > > > > > On Mon, Aug 17, 2015 at 4:25 PM, Igor Kaplan <igo...@gm...> > wrote: > > > > Hi All, > > > > > > > > My name is Igor. I just found the OpenFPC and evaluating it. Looks > > like it is very good tool. > > > > I successfully installed on Ubuntu 14.4 with Perl 5.18 > > > > I have installed the OpenFPC-master, so it is the latest code. > > > > > > > > Now I would like to find out if there is more documentation beside > > files which I could find under docs directory. > > > > For example the INSTALL.md refers to the USAGE document, however I > > was not able to find it anywhere > > > > > > > > I am looking for the usage other then basic, just to find out, what > > are my advanced options. > > > > > > > > For example the openfpc-dbmaint.sh script is also able to create the > > gui database, I wonder, what it is for? > > > > > > > > The OpenFPC looks to be very powerful, just would like to understand > > it as best as I can. > > > > > > > > Would so much appreciate any reply?s. > > > > > > > > Many thanks. > > > > > > > > Igor. > > > > > > > > > > ---------------------------------------------------------------------- > > ------ > > -- > > > > _______________________________________________ > > Openfpc-users mailing list > > Ope...@li... > > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > > > > > > > > > ---------------------------------------------------------------------- > > ------ > > -- > > > > _______________________________________________ > > Openfpc-users mailing list > > Ope...@li... > > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was > > scrubbed... > > > > ------------------------------ > > > > ---------------------------------------------------------------------- > > ------ > > -- > > > > > > ------------------------------ > > > > _______________________________________________ > > Openfpc-users mailing list > > Ope...@li... > > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > > > > > End of Openfpc-users Digest, Vol 2, Issue 4 > > ******************************************* > > > > > > ---------------------------------------------------------------------- > > -------- _______________________________________________ > > Openfpc-users mailing list > > Ope...@li... > > https://lists.sourceforge.net/lists/listinfo/openfpc-users > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-10 19:23:45
|
Hi Leon,
Here is the output which you asked:
- openfpc-daemonlogger is /usr/bin/daemonlogger
BPF file: /var/filter.bpf
Using BPF
DEBUG: Found BPF file /var/filter.bpf - openfpc-daemonlogger is /usr/bin/daemonlogger
BPF file: /var/filter.bpf
Using BPF
DEBUG: Found BPF file /var/filter.bpfStopping Daemonlogger... Done
- openfpc-daemonlogger is /usr/bin/daemonlogger
BPF file: /var/filter.bpf
Using BPF
DEBUG: Found BPF file /var/filter.bpfDEBUG: Command used to start daemonlogger is
/usr/bin/daemonlogger -d -f /var/filter.bpf -i eth1 -l /var/tmp/openfpc/pcap -M 50 -s 1G -p openfpc-daemonlogger.pid -P /var/run/openfpc-Default_Node -n openfpc-Default_Node.pcap -u openfpc -g openfpc 2>&1 | logger -t OFPC-DL-Default_Node
Starting Daemonlogger (Default_Node)...
DEBUG: Touching a canary file to check if DL needs to delete something at startup:/var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.0
Done
DEBUG: canary file still here (good)# BPF_FILE: An optional BPF filter to use while capturing traffic.
grep BPF_FILE /etc/openfpc/*.conf
BPF_FILE=/var/filter.bpf
ps aux |grep daemonlogger
openfpc 31867 0.0 0.1 19252 4304 ? Ss 15:06 0:00 /usr/bin/daemonlogger -d -f /var/filter.bpf -i eth1 -l /var/tmp/openfpc/pcap -M 50 -s 1G -p openfpc-daemonlogger.pid -P /var/run/openfpc-Default_Node -n openfpc-Default_Node.pcap -u openfpc -g openfpc
root 31894 0.0 0.0 10468 2136 pts/0 S+ 15:08 0:00
And here is the syslog
Sep 10 15:06:22 ikaplan-DH-2 daemonlogger[1724]: Quitting!
Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: sniffing on interface eth1
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Checking partition stats for log directory "/var/tmp/openfpc/pcap/."
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: 50% max disk utilization = 1885056 blocks free (out of 3770112)
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Blocksize = 4096
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Rollsize = 262144 blocks
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node:
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Daemon mode set
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Reading BPF filter in from file /var/filter.bpf
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Setting group ID to openfpc
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Interface set to eth1
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Logpath set to /var/tmp/openfpc/pcap
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Log filename set to "openfpc-Default_Node.pcap"
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pidfile configured to "openfpc-daemonlogger.pid"
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pidpath configured to "/var/run/openfpc-Default_Node"
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Rollover configured for 1 gigabytes
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Rollover configured for 0 none
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Setting user ID to openfpc
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: [-] Pruning behavior set to oldest IN DIRECTORY
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node:
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: -*> DaemonLogger <*-
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: Version 1.2.1
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: By Martin Roesch
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node: (C) Copyright 2006-2007 Sourcefire Inc., All rights reserved
Sep 10 15:06:23 ikaplan-DH-2 OFPC-DL-Default_Node:
Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: start_sniffing() device eth1 network lookup: #011eth1: no IPv4 address assigned
Sep 10 15:06:23 ikaplan-DH-2 daemonlogger[31867]: Logging packets to /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1441911983
Please let me know if there any other tests I should make.
Many thanks!!!
Igor.
From: leo...@gm... [mailto:leo...@gm...] On Behalf Of Leon Ward
Sent: Thursday, September 10, 2015 2:41 PM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: BPF filter
Okay, so just tested it and it looked like it works for me...
Can you please send the output of....
$ sudo openfpc -v -t openfpc-daemonlogger -a restart
$ grep BPF_FILE /etc/openfpc/*.conf
$ ps aux |grep daemonlogger
There should also be a chunk of data about daemonlogger in your syslog...
On Tue, Sep 8, 2015 at 4:14 PM, Igor Kaplan <igo...@gm...> wrote:
Leon,
Thanks so much, really appreciate!
From: leo...@gm... [mailto:leo...@gm...] On Behalf Of Leon Ward
Sent: Tuesday, September 08, 2015 9:44 AM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: BPF filter
Will take a look, can't from where I am right now.
Its likely that I've broken something years back, It used to work as I once used it a lot myself, but no longer.
On Tue, Sep 8, 2015 at 2:42 PM, Igor Kaplan <igo...@gm...> wrote:
Hi Leon,
I am still having problems with the input bpf filter. I wonder, if you
could please answer my question below.
Not sure, if I am doing everything correctly, however as my best
understanding, I do.
Many thanks.
-Igor.
-----Original Message-----
From: Igor Kaplan [mailto:igo...@gm...]
Sent: Friday, August 28, 2015 4:32 PM
To: 'Leon Ward'
Cc: ope...@li...
Subject: BPF filter
Hi Leon,
Not sure, if I am doing everything correctly, however my BPF filter in
openfpc-default.conf does not look to be working.
I have a line in my openfpc-default.conf:
BPF_File=/var/filter.bpf
While starting openfpc -a start --verbose I am able to see, the proper bpf
file is found and loaded.
My filter.bpf is very simple:
Port 502
So it should capture packets from port 502 only
However after fetching the data I still see packets from other ports.
I wonder, do I understand correctly, the BPF_FILE in the config file will
restrict, which packets are captured? So using the bpf file as above I
should not see any packets from other ports beside 502? Or it is something
different?
Many thanks and have a nice weekend!
-Igor
-----Original Message-----
From: Leon Ward [mailto:leo...@gm...] On Behalf Of Leon Ward
Sent: Tuesday, August 25, 2015 4:59 PM
To: Igor Kaplan
Cc: <ope...@li...>
Subject: Re: [Openfpc-users] Openfpc-users Digest, Vol 2, Issue 4
That's where the pcaps should live, and they will grow to the max percentage
that you allow in the node config.
What does an openfpc-client -a status show?
Also what's a df -h look like?
The pcaps will auto-prune unless something has gone wrong along the way....
Thinking out loud, what's an ls of your pcaps directory look like? Have you
got multiple nodes running on one box?
-L
Sent from a mobile device. Apologies for any typos but they happen.
> On 25 Aug 2015, at 16:25, Igor Kaplan <igo...@gm...> wrote:
>
> Hi Leon,
>
> Could you please help me with following.
> I am running openfpc for several days already and now I am out of
> space on my Ubuntu box which runs openfpc Under /var/tmp/openfpc I see
> directories, some of which contain number of large files:
> api-pcaps extracted pcap session
>
> I wonder, can I safely delete data under any of those directories above?
> Could you please let me know, which directory I can empty without
> breaking openfpc functionality?
>
> Is there any way to clean all captured data and start fresh?
>
> Many thanks.
>
> -Igor.
>
>
> -----Original Message-----
> From: ope...@li...
> [mailto:ope...@li...]
> Sent: Thursday, August 20, 2015 8:01 PM
> To: ope...@li...
> Subject: Openfpc-users Digest, Vol 2, Issue 4
>
> Send Openfpc-users mailing list submissions to
> ope...@li...
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
> or, via email, send a message with subject or body 'help' to
> ope...@li...
>
> You can reach the person managing the list at
> ope...@li...
>
> When replying, please edit your Subject line so it is more specific
> than
> "Re: Contents of Openfpc-users digest..."
>
>
> Today's Topics:
>
> 1. Re: Openfpc usage (Igor Kaplan)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 20 Aug 2015 20:01:08 -0400
> From: "Igor Kaplan" <igo...@gm...>
> Subject: Re: [Openfpc-users] Openfpc usage
> To: <ope...@li...>
> Message-ID: <000c01d0dba4$7c3c9910$74b5cb30$@gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Also sending my reply to the list, sorry, forgot to include it.
>
>
>
>
>
> From: Igor Kaplan [mailto:igo...@gm...]
> Sent: Thursday, August 20, 2015 6:44 PM
> To: 'Leon Ward'
> Subject: RE: [Openfpc-users] Openfpc usage
>
>
>
> Version of mergecap:
>
> Mergecap 1.10.6 (v1.10.6 from master-1.10)
>
>
>
> Linux is Ubuntu 14.04.2 LTS
>
>
>
> File list.pcap
>
> list.pcap: pcap-ng capture file - version 1.0
>
>
>
> Thanks.
>
>
>
> From: leo...@gm... [mailto:leo...@gm...] On Behalf
> Of Leon Ward
> Sent: Thursday, August 20, 2015 6:23 PM
> To: Igor Kaplan
> Cc: ope...@li...
> Subject: Re: [Openfpc-users] Openfpc usage
>
>
>
> What's your platform, version of mergecap etc.
>
> Also, if you '$ file list.pcap' what does it say?
>
>
>
> -L
>
>
>
>
>
> On Thu, Aug 20, 2015 at 10:15 PM, Igor Kaplan <igo...@gm...>
wrote:
>
> Hi Leon,
>
>
>
> Need your help please once again.
>
> Got the following problem and spent several hours trying to solve it.
>
>
>
> When making the API call to fetch the pcap data I am getting the data
> in pcapng format.
>
> The OpenFPC is using the mergecap to merge pcap files and by default
> mergecap creates the output in pcapng format instead of pcap.
>
>
>
> I have changed the following line in openfpc-default.conf file
>
> MERGECAP=/usr/bin/mergecap -F pcap
>
>
>
> This helped when I use the openfpc-client command to create pcap
> files, however when I use curl to fetch the data I still receive the
> output in pcapng format.
>
>
>
> curl -k
> 192.168.145.20:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C
> E8A48\
> <http://192.168.145.20:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F
> -C061B
> 4CE8A48%5C&stime=20150818%2010:00%5C&etime=20150818%2010:30%5C&dpt=22>
> &stime=20150818%2010:00\&etime=20150818%2010:30\&dpt=22 > list.pcap
>
>
>
> cat list.pcap|tshark -i-
>
> Capturing on 'Standard input'
>
> tshark: Unrecognized libpcap format
>
>
>
> Looks like in case of API call the mergecap utility is not used at all.
> And I was not able to find in the code how merging is done in this case.
>
>
>
> Could you please help me. Is it possible to make the fetch API call
> to return the data in pcap format?
>
>
>
> Thanks so much!
>
>
>
> Igor
>
>
>
>
>
> From: leo...@gm... [mailto:leo...@gm...] On Behalf
> Of Leon Ward
> Sent: Tuesday, August 18, 2015 1:29 PM
>
>
> To: Igor Kaplan
> Cc: ope...@li...
> Subject: Re: [Openfpc-users] Openfpc usage
>
>
>
> Actually it wont. It will only remove the oldest PCAP file. It's best
> to keep those PCAP files on their own partition.
>
> The old flow records in mysql actually get removed automatically based
> on the oldest packet in the store. So you won't have records that are
> older than the pcaps.
>
>
>
> -L
>
>
>
>
>
> On Tue, Aug 18, 2015 at 6:21 PM, Igor Kaplan <igo...@gm...>
wrote:
>
> Hi Leon, all,
>
>
>
> I have one more question please.
>
>
>
> Based on the documentation the following line in the openfpc config
> file restricts the space usage of captured data to 50 percent:
>
> PCAP_SPACE=50
>
>
>
> So, if the data size exceeds 50 percent old files will be deleted
> automatically?
>
> Will openfpc also delete the old MySQL session tables?
>
>
>
> Many thanks and all the best!
>
>
>
> -Igor.
>
>
>
>
>
> From: leo...@gm... [mailto:leo...@gm...] On Behalf
> Of Leon Ward
> Sent: Monday, August 17, 2015 11:51 AM
> To: Igor Kaplan
> Cc: ope...@li...
> Subject: Re: [Openfpc-users] Openfpc usage
>
>
>
> Hi,
>
>
>
> Documentation is really one of the places that really needs some extra
> focus.
>
>
>
> The best docs I can point you to are in that folder, plus there is
> some out-of date info on my blog http://www.leonward.com.
>
> I actually delivered a presentation at Defcon last weekend all about
> OpenFPC. I have forwarded the slides separately. Hopefully that will
> help as well.
>
>
>
> As for your specific question about OpenFPC GUI. That's actually now
> been deprecated as it's no longer relevant for how it functions in a
> distributed manner. The OpenFPC-Chrome Extension will be the next best
> thing for interacting with the QueueDaemon remotely in a GUI-like way.
>
>
>
> Cheers,
>
>
>
> -L
>
>
>
>
>
>
>
> On Mon, Aug 17, 2015 at 4:25 PM, Igor Kaplan <igo...@gm...>
wrote:
>
> Hi All,
>
>
>
> My name is Igor. I just found the OpenFPC and evaluating it. Looks
> like it is very good tool.
>
> I successfully installed on Ubuntu 14.4 with Perl 5.18
>
> I have installed the OpenFPC-master, so it is the latest code.
>
>
>
> Now I would like to find out if there is more documentation beside
> files which I could find under docs directory.
>
> For example the INSTALL.md refers to the USAGE document, however I
> was not able to find it anywhere
>
>
>
> I am looking for the usage other then basic, just to find out, what
> are my advanced options.
>
>
>
> For example the openfpc-dbmaint.sh script is also able to create the
> gui database, I wonder, what it is for?
>
>
>
> The OpenFPC looks to be very powerful, just would like to understand
> it as best as I can.
>
>
>
> Would so much appreciate any reply?s.
>
>
>
> Many thanks.
>
>
>
> Igor.
>
>
>
>
> ----------------------------------------------------------------------
> ------
> --
>
> _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
>
>
>
>
> ----------------------------------------------------------------------
> ------
> --
>
> _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
>
>
>
>
>
> -------------- next part -------------- An HTML attachment was
> scrubbed...
>
> ------------------------------
>
> ----------------------------------------------------------------------
> ------
> --
>
>
> ------------------------------
>
> _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
>
>
> End of Openfpc-users Digest, Vol 2, Issue 4
> *******************************************
>
>
> ----------------------------------------------------------------------
> -------- _______________________________________________
> Openfpc-users mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openfpc-users
|
