Thread: [Openfpc-users] fetch query time format.
Open Source Full Packet Capture
Brought to you by:
leonward
Menu
▾
▴
openfpc-users
|
From: Igor K. <igo...@gm...> - 2015-09-16 16:54:17
|
Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Igor K. <igo...@gm...> - 2015-09-17 17:08:38
|
Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-09-18 13:55:06
|
Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-18 15:15:23
|
HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-09-18 16:25:06
|
I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-18 16:34:49
|
Would be really nice!!! Thanks so much!!!! From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Igor K. <igo...@gm...> - 2015-09-29 19:33:11
|
Hi Leon, Is there any easy way to clean up mysql database tables and start everything fresh? I am having some strange problems, so would like to start everything from beginning. Thanks so much. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-09-29 20:08:59
|
Sure, Use openfpc-dbmaint. Something /like/ the below should do the trick. 1) Stop OpenFPC sudo openfpc -a stop 2) Drop the session database openfpc-dbmaint -a drop -t session -c /etc/openfpc/openfpc-<nodename>.conf 3) recreate the session db openfpc-dbmaint -a create -t session -c /etc/openfpc/openfpc-<nodename>.conf 4) Start again sudo openfpc -a start If you wanted to remove all of the pcap files, you could purge your install and re-install from scratch instead of option #3 sudo ./openfpc-install.sh purge sudo ./openfpc-install.sh install -Leon On Tue, Sep 29, 2015 at 12:32 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Is there any easy way to clean up mysql database tables and start > everything fresh? > > I am having some strange problems, so would like to start everything > from beginning. > > > > Thanks so much. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-09-29 20:13:38
|
Leon, Many many thanks! From: Leon Ward [mailto:leo...@gm...] Sent: Tuesday, September 29, 2015 4:09 PM To: Igor Kaplan Cc: <ope...@li...> Subject: Re: fetch query time format. Sure, Use openfpc-dbmaint. Something /like/ the below should do the trick. 1) Stop OpenFPC sudo openfpc -a stop 2) Drop the session database openfpc-dbmaint -a drop -t session -c /etc/openfpc/openfpc-<nodename>.conf 3) recreate the session db openfpc-dbmaint -a create -t session -c /etc/openfpc/openfpc-<nodename>.conf 4) Start again sudo openfpc -a start If you wanted to remove all of the pcap files, you could purge your install and re-install from scratch instead of option #3 sudo ./openfpc-install.sh purge sudo ./openfpc-install.sh install -Leon On Tue, Sep 29, 2015 at 12:32 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Is there any easy way to clean up mysql database tables and start everything fresh? I am having some strange problems, so would like to start everything from beginning. Thanks so much. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Igor K. <igo...@gm...> - 2015-11-11 16:30:26
|
Hi Leon, Just would like to ask you, if you had a chance to take a look at the fix, which we talked below, regarding retrieving only data for the time frame? Now I am facing the issue when the retrieved pcap file is so large,, so the tshark is having problems to process it. Is there any way to reduce the size of pcap file which is fetched? Many thanks. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-11-23 16:12:23
|
Hi, Yeah, I did find time to poke at this over the weekend while recovering from jetlag. It's in a branch called hard_time_window, and I've decided to enforce the behaviour by default because it works pretty well. Before I merge it for everyone into the main branch, I'd appreciate it if you could test it out. Any chance you could test the branch, and if it functions I'll merge it. It is also configurable (but to exposed by default) in the node config file, so if you need to go back to the old behaviour you can set STRICT_TIME=0. The log will reflect if it is enforcing it or not. It's working for me here. Cheers -L On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Just would like to ask you, if you had a chance to take a look at the > fix, which we talked below, regarding retrieving only data for the time > frame? > > > > Now I am facing the issue when the retrieved pcap file is so large,, so > the tshark is having problems to process it. > > > > Is there any way to reduce the size of pcap file which is fetched? > > > > Many thanks. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 12:25 PM > *To:* Igor Kaplan > *Cc:* ope...@li... > > *Subject:* Re: fetch query time format. > > > > I don't think it's hard to do - I'll take a look sometime... > > > > -L > > > > > > On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-11-23 16:57:14
|
Leon, Thanks so much. I’ll take a look at this as soon as I can. Sounds really good! So now, if I specify the exact time window it will return only packets from that time frame?. Many thanks. -Igor From: Leon Ward [mailto:leo...@gm...] Sent: Monday, November 23, 2015 11:12 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, Yeah, I did find time to poke at this over the weekend while recovering from jetlag. It's in a branch called hard_time_window, and I've decided to enforce the behaviour by default because it works pretty well. Before I merge it for everyone into the main branch, I'd appreciate it if you could test it out. Any chance you could test the branch, and if it functions I'll merge it. It is also configurable (but to exposed by default) in the node config file, so if you need to go back to the old behaviour you can set STRICT_TIME=0. The log will reflect if it is enforcing it or not. It's working for me here. Cheers -L On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Just would like to ask you, if you had a chance to take a look at the fix, which we talked below, regarding retrieving only data for the time frame? Now I am facing the issue when the retrieved pcap file is so large,, so the tshark is having problems to process it. Is there any way to reduce the size of pcap file which is fetched? Many thanks. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-11-23 17:03:22
|
Yup, On Mon, Nov 23, 2015 at 4:57 PM, Igor Kaplan <igo...@gm...> wrote: > Leon, > > > > Thanks so much. I’ll take a look at this as soon as I can. Sounds really > good! > > > > So now, if I specify the exact time window it will return only packets > from that time frame?. > Yup, that's the plan. All of my testing was using "--last", so please test other options. For example --last 3 would only return 3 seconds of traffic. -L > > > Many thanks. > > > > -Igor > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Monday, November 23, 2015 11:12 AM > > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > Yeah, I did find time to poke at this over the weekend while recovering > from jetlag. > > > > It's in a branch called hard_time_window, and I've decided to enforce the > behaviour by default because it works pretty well. > > > > Before I merge it for everyone into the main branch, I'd appreciate it if > you could test it out. Any chance you could test the branch, and if it > functions I'll merge it. It is also configurable (but to exposed by > default) in the node config file, so if you need to go back to the old > behaviour you can set STRICT_TIME=0. The log will reflect if it is > enforcing it or not. > > > > It's working for me here. > > > > Cheers > > > > -L > > > > > > > > On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Just would like to ask you, if you had a chance to take a look at the > fix, which we talked below, regarding retrieving only data for the time > frame? > > > > Now I am facing the issue when the retrieved pcap file is so large,, so > the tshark is having problems to process it. > > > > Is there any way to reduce the size of pcap file which is fetched? > > > > Many thanks. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 12:25 PM > *To:* Igor Kaplan > *Cc:* ope...@li... > > > *Subject:* Re: fetch query time format. > > > > I don't think it's hard to do - I'll take a look sometime... > > > > -L > > > > > > On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > > > > > |
|
From: Igor K. <igo...@gm...> - 2015-12-03 17:26:43
|
Hi Leon, Sorry for delaying the reply. I have tested the hard_time_window version and found it works fine. I have tested it with –last and also using curl with starttime and endtime options. It returns the correct time window. Thanks so much for this fix!!! -Igor From: Leon Ward [mailto:leo...@gm...] Sent: Monday, November 23, 2015 12:03 PM To: Igor Kaplan Cc: <ope...@li...> Subject: Re: fetch query time format. Yup, On Mon, Nov 23, 2015 at 4:57 PM, Igor Kaplan <igo...@gm...> wrote: Leon, Thanks so much. I’ll take a look at this as soon as I can. Sounds really good! So now, if I specify the exact time window it will return only packets from that time frame?. Yup, that's the plan. All of my testing was using "--last", so please test other options. For example --last 3 would only return 3 seconds of traffic. -L Many thanks. -Igor From: Leon Ward [mailto:leo...@gm...] Sent: Monday, November 23, 2015 11:12 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, Yeah, I did find time to poke at this over the weekend while recovering from jetlag. It's in a branch called hard_time_window, and I've decided to enforce the behaviour by default because it works pretty well. Before I merge it for everyone into the main branch, I'd appreciate it if you could test it out. Any chance you could test the branch, and if it functions I'll merge it. It is also configurable (but to exposed by default) in the node config file, so if you need to go back to the old behaviour you can set STRICT_TIME=0. The log will reflect if it is enforcing it or not. It's working for me here. Cheers -L On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Just would like to ask you, if you had a chance to take a look at the fix, which we talked below, regarding retrieving only data for the time frame? Now I am facing the issue when the retrieved pcap file is so large,, so the tshark is having problems to process it. Is there any way to reduce the size of pcap file which is fetched? Many thanks. -Igor. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 12:25 PM To: Igor Kaplan Cc: ope...@li... Subject: Re: fetch query time format. I don't think it's hard to do - I'll take a look sometime... -L On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: HI Leon, Thanks so much for your message. Well, in my application I thought to specify the short time frame and fetch all packets between start and end time only. It would be really nice to be able to fetch the data exactly during that time frame, however I understand, it might be difficult to program. From: Leon Ward [mailto:leo...@gm...] Sent: Friday, September 18, 2015 9:55 AM To: Igor Kaplan; <ope...@li...> Subject: Re: fetch query time format. Hi, This is actually by design (believe it or not). The time anchors are only used to define the start -- end pcap buffer files to extract the packets from that daemonlogger has created. For example (using simplistic epoch offsets) The pcap buffer files are... nodename.0000000.pcap nodename.0000060.pcap nodename.0000120.pcap nodename.0000180.pcap nodename.0000240.pcap stime: 000061 etime:00000181 will extract all packets from 000060, 0000120, and 0000180. You'll have packets with timestamps between 0000182 - 0000239 (the rest of the packets in the 0000180 file). So you'll get the same output pcap as you would have with stime:00059 etime: 0000239 I had toyed with the idea of performing a post-processing task to constrain to *only* the packets within the time range, but having a bit more data than I need isn't a problem for my use cases. It wouldn't be much effort to add this as I had always thought about it, but it would create an extra processing step. Is this causing you a problem for the way you use ofpc? -L On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: Hi Leon, Sorry for bugging you, still having the problem to retrieve packets for the specific time. I actually made today several more tests to fetch the data for the specified time window. Here are my results. The following curl command was used: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> &stime=1442499800\&etime=1442499801\&dpt=22 > igor.pcap The test time is Thu Sep 17 10:23:21 2015 So the difference between stime and etime is 1 second. However I receive a very large file which contains packets for the time before stime and ends with the current time. -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap Here is the tshark –t ad –r igor.pcap output: 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 Encrypted request packet len=68 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 Encrypted response packet len=36 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 TSecr=51990631 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 Encrypted response packet len=112 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 … 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 TSecr=1729547 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 TSecr=1729599 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 Encrypted request packet len=48 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 Encrypted response packet len=48 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 TSecr=1729654 As you could see, there are packets for more then an hour time period. I tried different test cases, specified timestamp instead of stime and etime, specified all 3 parameters, still the same. Could you please suggest anything, is there anything I am doing incorrectly? Thanks so much for any help! Igor. P. S. Here is the syslog debug output from the fetch request. Everything looks fine, time is converted correctly, however data is still retrieved for much longer time frame. Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted new connection from 127.0.0.1 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding user "admin" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: GOT version, sending OFPC-v2 OK Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: GOT USER admin Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Sending challenge: 76481436316616557168860816866580268462 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Waiting for response to challenge Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Got RESPONSE Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Expected resp: '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: Actual resp : '9afe89910819c2142e997fbfaa60c72c' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: Pass Okay Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding request Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Received action fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Normalizing timestamps Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: timestamp not set in request. Nothing to normalize Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: stime in request was: 1442499800 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499800 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: etime in request was: 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for timestamp 1442499801 not set. Defaulting to local tz (America/New_York) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local TZ:America/New_York, Target TZ:America/New_York, offset 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No BPF or logline detected in request, using session identifiers Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Timestamp is Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. Comment: 0 Filetype : PCAP Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No value for timestamp has been passed from the user requets Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final stime and etime are set in request as 1442499800 / 1442499801 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Node action Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and 1442499801 (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer Range mode Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499800) (Thu Sep 17 10:23:20 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499799.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last file in buffer range Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING vdebug not enabled to inspect pcap filename selection Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request is to look in 0 files each side of target timestamp (1442499801) (Thu Sep 17 10:23:21 2015) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got TARGET match of 1442499800.5 in array location 2 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount value (number of files before target is :0 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499800) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 (1442499801) Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final PCAP roster (1 files in total) for extract is: /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: doExtract: Extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge command is "/usr/bin/mergecap -F pcap -w /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building bpf from: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , DIP: Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , DPT: 22 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf "port 22" Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Validating list of pcap files /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: BPF looks valid Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded 3219 x 1KB chunks Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. From: Igor Kaplan [mailto:igo...@gm...] Sent: Wednesday, September 16, 2015 12:54 PM To: 'Leon Ward' Cc: ope...@li... Subject: fetch query time format. Hi Leon, Could you please tell me, what time format the openfpc API will accept? I just found, if I specify number of seconds from 1970 for both stime and etime the fetch retrieves the same big pcap regardless what time I specify. For example my curl fetch command: curl -k 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\ <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> &stime=1442350627\&etime=1442350827\&dpt=502 Does not metter, how I manipulate stime and etime, I always receive one big pcap file with exactly the same size. If openfpc does not like the time in number of seconds, what is the time format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, this format is not convenient for me. Many thanks for all your help. -Igor. |
|
From: Leon W. <leo...@gm...> - 2015-12-03 17:37:21
|
Nice, thanks. I'll merge it in. -L On Thu, Dec 3, 2015 at 5:26 PM, Igor Kaplan <igo...@gm...> wrote: > Hi Leon, > > > > Sorry for delaying the reply. > > > > I have tested the hard_time_window version and found it works fine. > > > > I have tested it with –last and also using curl with starttime and > endtime options. > > It returns the correct time window. > > > > Thanks so much for this fix!!! > > > > -Igor > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Monday, November 23, 2015 12:03 PM > > *To:* Igor Kaplan > *Cc:* <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Yup, > > > > On Mon, Nov 23, 2015 at 4:57 PM, Igor Kaplan <igo...@gm...> wrote: > > Leon, > > > > Thanks so much. I’ll take a look at this as soon as I can. Sounds really > good! > > > > So now, if I specify the exact time window it will return only packets > from that time frame?. > > > > Yup, that's the plan. All of my testing was using "--last", so please test > other options. For example --last 3 would only return 3 seconds of traffic. > > > > -L > > > > > > Many thanks. > > > > -Igor > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Monday, November 23, 2015 11:12 AM > > > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > Yeah, I did find time to poke at this over the weekend while recovering > from jetlag. > > > > It's in a branch called hard_time_window, and I've decided to enforce the > behaviour by default because it works pretty well. > > > > Before I merge it for everyone into the main branch, I'd appreciate it if > you could test it out. Any chance you could test the branch, and if it > functions I'll merge it. It is also configurable (but to exposed by > default) in the node config file, so if you need to go back to the old > behaviour you can set STRICT_TIME=0. The log will reflect if it is > enforcing it or not. > > > > It's working for me here. > > > > Cheers > > > > -L > > > > > > > > On Wed, Nov 11, 2015 at 4:30 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Just would like to ask you, if you had a chance to take a look at the > fix, which we talked below, regarding retrieving only data for the time > frame? > > > > Now I am facing the issue when the retrieved pcap file is so large,, so > the tshark is having problems to process it. > > > > Is there any way to reduce the size of pcap file which is fetched? > > > > Many thanks. > > > > -Igor. > > > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 12:25 PM > *To:* Igor Kaplan > *Cc:* ope...@li... > > > *Subject:* Re: fetch query time format. > > > > I don't think it's hard to do - I'll take a look sometime... > > > > -L > > > > > > On Fri, Sep 18, 2015 at 4:15 PM, Igor Kaplan <igo...@gm...> wrote: > > HI Leon, > > > > Thanks so much for your message. > > Well, in my application I thought to specify the short time frame and > fetch all packets between start and end time only. > > It would be really nice to be able to fetch the data exactly during > that time frame, however I understand, it might be difficult to program. > > > > *From:* Leon Ward [mailto:leo...@gm...] > *Sent:* Friday, September 18, 2015 9:55 AM > *To:* Igor Kaplan; <ope...@li...> > *Subject:* Re: fetch query time format. > > > > Hi, > > > > This is actually by design (believe it or not). The time anchors are only > used to define the start -- end pcap buffer files to extract the packets > from that daemonlogger has created. > > For example (using simplistic epoch offsets) > > > > The pcap buffer files are... > > > > nodename.0000000.pcap > > nodename.0000060.pcap > > nodename.0000120.pcap > > nodename.0000180.pcap > > nodename.0000240.pcap > > > > stime: 000061 etime:00000181 will extract all packets from 000060, > 0000120, and 0000180. You'll have packets with timestamps between 0000182 - > 0000239 (the rest of the packets in the 0000180 file). > > So you'll get the same output pcap as you would have with stime:00059 > etime: 0000239 > > > > I had toyed with the idea of performing a post-processing task to > constrain to *only* the packets within the time range, but having a bit > more data than I need isn't a problem for my use cases. It wouldn't be much > effort to add this as I had always thought about it, but it would create an > extra processing step. > > > > Is this causing you a problem for the way you use ofpc? > > > > -L > > > > > > On Thu, Sep 17, 2015 at 6:08 PM, Igor Kaplan <igo...@gm...> wrote: > > Hi Leon, > > > > Sorry for bugging you, still having the problem to retrieve packets for > the specific time. > > I actually made today several more tests to fetch the data for the > specified time window. > > Here are my results. > > The following curl command was used: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48\&stime=1442499800\&etime=1442499801\&dpt=22 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4CE8A48%5C&stime=1442499800%5C&etime=1442499801%5C&dpt=22> > > igor.pcap > > > > The test time is Thu Sep 17 10:23:21 2015 > > > > So the difference between stime and etime is 1 second. > > > > However I receive a very large file which contains packets for the time > before stime and ends with the current time. > > > > -rw-r--r-- 1 root root 3295302 Sep 17 12:34 igor.pcap > > > > Here is the tshark –t ad –r igor.pcap output: > > 1 2015-09-17 10:00:23.917990 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=984480799 TSecr=265459 > > 2 2015-09-17 10:00:23.922293 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=113 Win=501 Len=0 TSval=984480800 TSecr=265460 > > 3 2015-09-17 10:00:24.321684 10.8.28.11 -> 192.168.141.244 SSH 134 > Encrypted request packet len=68 > > 4 2015-09-17 10:00:24.322073 192.168.141.244 -> 10.8.28.11 SSH 102 > Encrypted response packet len=36 > > 5 2015-09-17 10:00:24.351296 10.8.28.11 -> 192.168.141.244 TCP 66 > 55454 > ssh [ACK] Seq=69 Ack=37 Win=1262 Len=0 TSval=984480843 > TSecr=51990631 > > 6 2015-09-17 10:00:24.553313 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 7 2015-09-17 10:00:24.558903 192.168.141.248 -> 10.8.28.11 SSH 178 > Encrypted response packet len=112 > > 8 2015-09-17 10:00:24.582672 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=161 Win=501 Len=0 TSval=984480866 TSecr=265625 > > 9 2015-09-17 10:00:24.588222 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=1 Ack=273 Win=501 Len=0 TSval=984480866 TSecr=265626 > > 10 2015-09-17 10:00:24.601384 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 11 2015-09-17 10:00:24.601475 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > … > > 16593 2015-09-17 11:38:00.240420 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16594 2015-09-17 11:38:00.240655 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16595 2015-09-17 11:38:00.270065 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140657 Ack=1588305 Win=5191 Len=0 TSval=985066445 > TSecr=1729547 > > 16596 2015-09-17 11:38:00.450431 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16597 2015-09-17 11:38:00.450684 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16598 2015-09-17 11:38:00.480036 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140705 Ack=1588353 Win=5191 Len=0 TSval=985066466 > TSecr=1729599 > > 16599 2015-09-17 11:38:00.670482 10.8.28.11 -> 192.168.141.248 SSH 114 > Encrypted request packet len=48 > > 16600 2015-09-17 11:38:00.671233 192.168.141.248 -> 10.8.28.11 SSH 114 > Encrypted response packet len=48 > > 16601 2015-09-17 11:38:00.700582 10.8.28.11 -> 192.168.141.248 TCP 66 > 42060 > ssh [ACK] Seq=140753 Ack=1588401 Win=5191 Len=0 TSval=985066488 > TSecr=1729654 > > > > As you could see, there are packets for more then an hour time period. > > I tried different test cases, specified timestamp instead of stime and > etime, specified all 3 parameters, still the same. > > > > Could you please suggest anything, is there anything I am doing > incorrectly? > > > > Thanks so much for any help! > > > > Igor. > > > > P. S. > > > > Here is the syslog debug output from the fetch request. Everything looks > fine, time is converted correctly, however data is still retrieved for > much longer time frame. > > > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Accepted > new connection from 127.0.0.1 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Adding > user "admin" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG 127.0.0.1: > GOT version, sending OFPC-v2 OK > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > GOT USER admin > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Sending challenge: 76481436316616557168860816866580268462 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Waiting for response to challenge > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Got RESPONSE > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Expected resp: '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: 127.0.0.1: > Actual resp : '9afe89910819c2142e997fbfaa60c72c' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node AUTH : 127.0.0.1: > Pass Okay > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Decoding > request > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Received action fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Normalizing timestamps > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > timestamp not set in request. Nothing to normalize > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > stime in request was: 1442499800 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499800 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > stime normalized to 1442499800 (Thu Sep 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > etime in request was: 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: Timezone for > timestamp 1442499801 not set. Defaulting to local tz (America/New_York) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Unknown: TIME : Local > TZ:America/New_York, Target TZ:America/New_York, offset 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: DEBUG: > etime normalized to 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > BPF or logline detected in request, using session identifiers > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Timestamp is > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: > Session IDs sip: '' dip: '' spt: '' dpt: '22' proto: '0' > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: User > admin assigned RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA for action fetch. > Comment: 0 Filetype : PCAP > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: DEBUG: No > value for timestamp has been passed from the user requets > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DECOD: Final > stime and etime are set in request as 1442499800 / 1442499801 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node REQ: Action Fetch > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1: > RID: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA Fetch Request OK, sending RID > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Node action > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Action: fetch BPF: port 22 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Getting a > bunch of pcap files between 1442499800 (Thu Sep 17 10:23:20 2015) and > 1442499801 (Thu Sep 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Buffer > Range mode > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting First > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499800) (Thu Sep > 17 10:23:20 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499799.5 Thu Sep 17 10:23:19 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499799.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node Getting Last > file in buffer range > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: WARNING > vdebug not enabled to inspect pcap filename selection > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Request > is to look in 0 files each side of target timestamp (1442499801) (Thu Sep > 17 10:23:21 2015) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Requested > timestamp is 1442499800.5 Thu Sep 17 10:23:20 2015 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Got > TARGET match of 1442499800.5 in array location 2 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Precount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Postcount > value (number of files before target is :0 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Starting > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499800) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Ending > search in file /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > (1442499801) > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Final > PCAP roster (1 files in total) for extract is: > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Doing > Extraction with BPF port 22 into tempdir /tmp/ij2rMusQgg > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:33 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DBEUG: > doExtract: Extracting from > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node EXTR : Merge > command is "/usr/bin/mergecap -F pcap -w > /var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > /tmp/ij2rMusQgg/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap-1442498423.pcap" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node NODE : Request: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA User: admin Result: > 1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap, 3.2M, > 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Building > bpf from: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SIP: , > DIP: > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: SPT: , > DPT: 22 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Proto 0 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node MKBPF: Built bpf > "port 22" > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: > Validating list of pcap files > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442504295 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: Now > extracting from /var/tmp/openfpc/pcap/openfpc-Default_Node.pcap.1442498423 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node DEBUG: MKBPF: > BPF looks valid > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node JSON : Created > log JSON log for session FA11C270-5D59-11E5-BEA1-CFC9B98D42CA > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: > FA11C270-5D59-11E5-BEA1-CFC9B98D42CA 127.0.0.1 Sending > File:/var/tmp/openfpc/extracted/1442507673-FA11C270-5D59-11E5-BEA1-CFC9B98D42CA.pcap > MD5: 58e66f017a7eaae143b00e2fa8b9f9b2 > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: Uploaded > 3219 x 1KB chunks > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Transfer complete > > Sep 17 12:34:34 ikaplan-DH-2 OpenfpcQ[3831]: Default_Node COMMS: 127.0.0.1 > Request: FA11C270-5D59-11E5-BEA1-CFC9B98D42CA : Cleaning up. > > > > > > > > > > *From:* Igor Kaplan [mailto:igo...@gm...] > *Sent:* Wednesday, September 16, 2015 12:54 PM > *To:* 'Leon Ward' > *Cc:* ope...@li... > *Subject:* fetch query time format. > > > > Hi Leon, > > > > Could you please tell me, what time format the openfpc API will accept? > > I just found, if I specify number of seconds from 1970 for both stime > and etime the fetch retrieves the same big pcap regardless what time I > specify. > > > > For example my curl fetch command: > > curl -k > 192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C\&stime=1442350627\&etime=1442350827\&dpt=502 > <http://192.168.141.248:4222/api/1/fetch?apikey=38A69684-41F6-11E5-B47F-C061B4C%5C&stime=1442350627%5C&etime=1442350827%5C&dpt=502> > > > > Does not metter, how I manipulate stime and etime, I always receive one > big pcap file with exactly the same size. > > > > If openfpc does not like the time in number of seconds, what is the time > format? Only examples in the documentation I see Tue sep 15 2015 16:57:00, > this format is not convenient for me. > > > > Many thanks for all your help. > > > > -Igor. > > > > > > > > > > > |
Thanks for helping keep SourceForge clean.
X