Re: [sleuthkit-users] Slow Ingest
Brought to you by:
carrier
Menu
▾
▴
Re: [sleuthkit-users] Slow Ingest
|
From: Richard C. <rco...@ba...> - 2016-09-14 23:32:13
|
It looks like you are overtaxing your system with too many file ingest threads.The error notifications you are getting should not be considered to be typical and look to be symptomatic of a system struggling with out of memory errors. Based on your ingest progress snapshot, all file level analysis has been completed except keyword search, which is the most memory intensive aspect of Autopsy data source ingest. >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set number of threads to 12 as suggested by the Options dialog. We generally recommend at most one file ingest thread per core, and the code for the Autopsy options dialog is even more conservative. For the system you describe, I would expect the Java library we use to detect the available processors (java.lang.Runtime.getRuntime().availableProcessors()) would return eight ( http://ark.intel.com/products/37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed give you a recommendation of only six file ingest threads! However, it appears that the Java lib is reporting the "hyperthreads" (sixteen) rather than the cores, which would indeed result in a recommendation of twelve threads. We will need to look into this further! In the mean time, the best advice I can give you is back way off on file ingest threads. Sincerely, Richard Cordovano Basis Technology On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...> wrote: > I’m working a case and again have issues with performance using Autopsy. > I have setup a dedicated server for running Autopsy. In two days of ingest > I’m at 45%. In 8 hours it has only progressed 7%. I was hoping someone can > spot where my bottle neck is? > > > > The system has dual Xeon x5550 2.66 quad core processors. 24 GB RAM. > Windows 2012 R2 x64. The case drive is an OCZ Revo 350 PCIe SSD. Autopsy > is loaded on a Raid0 15k SAS volume. > > > > Autopsy Load. > > Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 > Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit > Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on > amd64; Cp1252; en_US (autopsy) > > > > The image is from a Windows 7 workstation. FTKimager took the disk image > in E01 format. I have the NSRL known good hash database loaded. I’ve set > number of threads to 12 as suggested by the Options dialog. I’m running > the default ingest process with no 3rd party modules. > > > > Performance Diagnostics > > > > Ingest Progress Snapshot > > > > 1 IDLE Wed Sep 14 > 01:10:44 CDT 2016 15:49:08.024 0 > > 2 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 > 2 > > 3 IDLE Wed Sep 14 > 16:59:26 CDT 2016 0:00:25.758 0 > > 4 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.798 > 2 > > 5 IDLE Wed Sep 14 > 16:59:26 CDT 2016 0:00:25.764 0 > > 6 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 > 2 > > 7 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 > 2 > > 8 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.132 > 2 > > 9 IDLE Wed Sep 14 > 16:59:26 CDT 2016 0:00:25.742 0 > > 10 IDLE Wed Sep 14 > 16:59:26 CDT 2016 0:00:25.812 0 > > 11 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.120 > 2 > > 12 IDLE Wed Sep 14 > 16:59:26 CDT 2016 0:00:25.811 0 > > 13 Keyword Search 2016-08-31-1-1.E01 > image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 > 2 > > > > 2 2016-08-31-1-1.E01 15:23:00 193034 > 2.0938940654524942 84 2 36 > 34 0 > > > > Keyword Search 155:52:59.626 (36%) > > File Type Identification 139:55:17.562 (32%) > > Hash Lookup 48:49:44.445 (11%) > > Embedded File Extractor 46:01:20.323 (10%) > > Email Parser 28:37:58.461 (6%) > > Extension Mismatch Detector 4:51:51.763 (1%) > > Exif Parser 1:41:39.597 (0%) > > PhotoRec Carver 0:00:04.762 (0%) > > Interesting Files Identifier 0:00:00.105 (0%) > > > > I get a bunch of errors but that is pretty typical. > > > > > ------------------------------------------------------------ > ------------------ > > _______________________________________________ > sleuthkit-users mailing list > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > http://www.sleuthkit.org > > |
