Re: [Openfpc-users] fetch query time format.
Open Source Full Packet Capture
Brought to you by:
leonward
Menu
▾
▴
Re: [Openfpc-users] fetch query time format.
|
From: Igor K. <igo...@gm...> - 2015-09-17 17:08:38
|
Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
