Re: [sleuthkit-users] Slow Ingest
Brought to you by:
carrier
Menu
▾
▴
Re: [sleuthkit-users] Slow Ingest
|
From: MATT P. <mat...@ad...> - 2016-10-06 17:26:18
|
I would like to circle back on this. I made the suggested modifications to reflect the physical cores of my system versus the hyper thread count. It honestly didn't make much a difference. When I reran the ingest it took 96 hours to complete the Analyzing Files stage. Periodic Keyword Search was at 3 Percent. One problem might be that my case was hosted on an OCZ350 Revo drive and it was waring of Overheat. I circled back and deleted the case, then reran it with the case on 15k SAS raid0 and the source file on 10k SAS. [cid:image007.jpg@01D21FCA.6B9D4710] [cid:image008.jpg@01D21FCA.6B9D4710] [cid:image009.jpg@01D21FCA.6B9D4710] As I read the perfmon data the system is just ticking over, not really pushing hard. Are my expectations off? The source drive image is 488 Gig in eo1 format. How long should an ingest take for such a drive. What are the system performance specs I should be working to improve to bring this down. My goal with migrating to the new system was to get a case ingest down to under 24 hours. From: MATT PIERCE Sent: Wednesday, September 14, 2016 6:25 PM To: Richard Cordovano <rco...@ba...> Subject: Re: [sleuthkit-users] Slow Ingest Thank you. I'm restarting the run tomorrow. I don't have any keywords defined so I might skip that module. Sent Mobily ________________________________ From: Richard Cordovano <rco...@ba...<mailto:rco...@ba...>> Sent: Sep 14, 2016 6:03 PM To: MATT PIERCE Cc: sle...@li...<mailto:sle...@li...> Subject: Re: [sleuthkit-users] Slow Ingest It looks like you are overtaxing your system with too many file ingest threads.The error notifications you are getting should not be considered to be typical and look to be symptomatic of a system struggling with out of memory errors. Based on your ingest progress snapshot, all file level analysis has been completed except keyword search, which is the most memory intensive aspect of Autopsy data source ingest. >> The system has dual Xeon x5550 2.66 quad core processors...I've set number of threads to 12 as suggested by the Options dialog. We generally recommend at most one file ingest thread per core, and the code for the Autopsy options dialog is even more conservative. For the system you describe, I would expect the Java library we use to detect the available processors (java.lang.Runtime.getRuntime().availableProcessors()) would return eight (http://ark.intel.com/products/37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed give you a recommendation of only six file ingest threads! However, it appears that the Java lib is reporting the "hyperthreads" (sixteen) rather than the cores, which would indeed result in a recommendation of twelve threads. We will need to look into this further! In the mean time, the best advice I can give you is back way off on file ingest threads. Sincerely, Richard Cordovano Basis Technology On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...<mailto:mat...@ad...>> wrote: I'm working a case and again have issues with performance using Autopsy. I have setup a dedicated server for running Autopsy. In two days of ingest I'm at 45%. In 8 hours it has only progressed 7%. I was hoping someone can spot where my bottle neck is? The system has dual Xeon x5550 2.66 quad core processors. 24 GB RAM. Windows 2012 R2 x64. The case drive is an OCZ Revo 350 PCIe SSD. Autopsy is loaded on a Raid0 15k SAS volume. Autopsy Load. Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on amd64; Cp1252; en_US (autopsy) The image is from a Windows 7 workstation. FTKimager took the disk image in E01 format. I have the NSRL known good hash database loaded. I've set number of threads to 12 as suggested by the Options dialog. I'm running the default ingest process with no 3rd party modules. Performance Diagnostics [cid:image001.png@01D21FC8.72F0CE30] Ingest Progress Snapshot 1 IDLE Wed Sep 14 01:10:44 CDT 2016 15:49:08.024 0 2 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 2 3 IDLE Wed Sep 14 16:59:26 CDT 2016 0:00:25.758 0 4 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.798 2 5 IDLE Wed Sep 14 16:59:26 CDT 2016 0:00:25.764 0 6 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 2 7 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 2 8 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.132 2 9 IDLE Wed Sep 14 16:59:26 CDT 2016 0:00:25.742 0 10 IDLE Wed Sep 14 16:59:26 CDT 2016 0:00:25.812 0 11 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.120 2 12 IDLE Wed Sep 14 16:59:26 CDT 2016 0:00:25.811 0 13 Keyword Search 2016-08-31-1-1.E01 image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 2 2 2016-08-31-1-1.E01 15:23:00 193034 2.0938940654524942 84 2 36 34 0 Keyword Search 155:52:59.626 (36%) File Type Identification 139:55:17.562 (32%) Hash Lookup 48:49:44.445 (11%) Embedded File Extractor 46:01:20.323 (10%) Email Parser 28:37:58.461 (6%) Extension Mismatch Detector 4:51:51.763 (1%) Exif Parser 1:41:39.597 (0%) PhotoRec Carver 0:00:04.762 (0%) Interesting Files Identifier 0:00:00.105 (0%) I get a bunch of errors but that is pretty typical. [cid:image002.png@01D21FC8.72F0CE30] ------------------------------------------------------------------------------ _______________________________________________ sleuthkit-users mailing list https://lists.sourceforge.net/lists/listinfo/sleuthkit-users http://www.sleuthkit.org |
