Re: [Openfpc-users] fetch query time format.
Open Source Full Packet Capture
Brought to you by:
leonward
Menu
▾
▴
Re: [Openfpc-users] fetch query time format.
|
From: Igor K. <igo...@gm...> - 2015-09-18 16:34:49
|
Would be really nice!!! Thanks so much!!!! From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
