Re: [Openfpc-users] fetch query time format.
Open Source Full Packet Capture
Brought to you by:
leonward
Menu
▾
▴
Re: [Openfpc-users] fetch query time format.
|
From: Leon W. <leo...@gm...> - 2015-09-18 13:55:06
|
Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > |
