Re: [sleuthkit-users] Slow Ingest
Brought to you by:
carrier
Menu
▾
▴
Re: [sleuthkit-users] Slow Ingest
|
From: Richard C. <rco...@ba...> - 2016-10-13 22:19:35
|
Ketil's hypothesis is very interesting, as is the remark about overheating. Does the image have a lot of unallocated space? Another possible test would be to reduce the number of file ingest threads to two and compare processing times. On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote: > If you have the time, opportunity and disk space, could you convert the > e01 to raw/dd and retry? I suspect that in some cases, libewf may be > spending a lot of time seeking in compressed e01 files. If you're > traversing a complex file system that requires a lot of seeking, my hunch > is that libewf, which is also single threaded as far as I understand, may > be your bottleneck. It'd be interesting to hear if simply changing to a raw > image makes much of a difference. > > https://github.com/libyal/libewf > > Regards, Ketil > > On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote: > >> I would like to circle back on this. I made the suggested modifications >> to reflect the physical cores of my system versus the hyper thread count. >> It honestly didn’t make much a difference. When I reran the ingest it took >> 96 hours to complete the Analyzing Files stage. Periodic Keyword Search >> was at 3 Percent. >> >> >> >> One problem might be that my case was hosted on an OCZ350 Revo drive and >> it was waring of Overheat. I circled back and deleted the case, then >> reran it with the case on 15k SAS raid0 and the source file on 10k SAS. >> >> >> >> >> >> >> >> >> >> As I read the perfmon data the system is just ticking over, not really >> pushing hard. Are my expectations off? The source drive image is 488 Gig >> in eo1 format. How long should an ingest take for such a drive. What are >> the system performance specs I should be working to improve to bring this >> down. My goal with migrating to the new system was to get a case ingest >> down to under 24 hours. >> >> >> >> >> >> *From:* MATT PIERCE >> *Sent:* Wednesday, September 14, 2016 6:25 PM >> *To:* Richard Cordovano <rco...@ba...> >> *Subject:* Re: [sleuthkit-users] Slow Ingest >> >> >> >> Thank you. I'm restarting the run tomorrow. I don't have any keywords >> defined so I might skip that module. >> >> >> >> Sent Mobily >> ------------------------------ >> >> *From:* Richard Cordovano <rco...@ba...> >> *Sent:* Sep 14, 2016 6:03 PM >> *To:* MATT PIERCE >> *Cc:* sle...@li... >> *Subject:* Re: [sleuthkit-users] Slow Ingest >> >> >> >> It looks like you are overtaxing your system with too many file ingest >> threads.The error notifications you are getting should not be considered to >> be typical and look to be symptomatic of a system struggling with out of >> memory errors. Based on your ingest progress snapshot, all file level >> analysis has been completed except keyword search, which is the most memory >> intensive aspect of Autopsy data source ingest. >> >> >> >> >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set >> number of threads to 12 as suggested by the Options dialog. >> >> >> >> We generally recommend at most one file ingest thread per core, and the >> code for the Autopsy options dialog is even more conservative. For the >> system you describe, I would expect the Java library we use to detect the >> available processors (java.lang.Runtime.getRuntime().availableProcessors()) >> would return eight (http://ark.intel.com/products >> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), >> which would indeed give you a recommendation of only six file ingest >> threads! However, it appears that the Java lib is reporting the >> "hyperthreads" (sixteen) rather than the cores, which would indeed result >> in a recommendation of twelve threads. We will need to look into this >> further! In the mean time, the best advice I can give you is back way off >> on file ingest threads. >> >> >> >> Sincerely, >> >> Richard Cordovano >> >> Basis Technology >> >> >> >> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...> >> wrote: >> >> I’m working a case and again have issues with performance using Autopsy. >> I have setup a dedicated server for running Autopsy. In two days of ingest >> I’m at 45%. In 8 hours it has only progressed 7%. I was hoping someone can >> spot where my bottle neck is? >> >> >> >> The system has dual Xeon x5550 2.66 quad core processors. 24 GB RAM. >> Windows 2012 R2 x64. The case drive is an OCZ Revo 350 PCIe SSD. Autopsy >> is loaded on a Raid0 15k SAS volume. >> >> >> >> Autopsy Load. >> >> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 >> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit >> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on >> amd64; Cp1252; en_US (autopsy) >> >> >> >> The image is from a Windows 7 workstation. FTKimager took the disk image >> in E01 format. I have the NSRL known good hash database loaded. I’ve set >> number of threads to 12 as suggested by the Options dialog. I’m running >> the default ingest process with no 3rd party modules. >> >> >> >> Performance Diagnostics >> >> >> >> Ingest Progress Snapshot >> >> >> >> 1 IDLE Wed Sep 14 >> 01:10:44 CDT 2016 15:49:08.024 0 >> >> 2 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 >> 2 >> >> 3 IDLE Wed Sep 14 >> 16:59:26 CDT 2016 0:00:25.758 0 >> >> 4 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.798 >> 2 >> >> 5 IDLE Wed Sep 14 >> 16:59:26 CDT 2016 0:00:25.764 0 >> >> 6 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 >> 2 >> >> 7 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 >> 2 >> >> 8 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.132 >> 2 >> >> 9 IDLE Wed Sep 14 >> 16:59:26 CDT 2016 0:00:25.742 0 >> >> 10 IDLE Wed Sep 14 >> 16:59:26 CDT 2016 0:00:25.812 0 >> >> 11 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.120 >> 2 >> >> 12 IDLE Wed Sep 14 >> 16:59:26 CDT 2016 0:00:25.811 0 >> >> 13 Keyword Search 2016-08-31-1-1.E01 >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 >> 2 >> >> >> >> 2 2016-08-31-1-1.E01 15:23:00 193034 >> 2.0938940654524942 84 2 36 >> 34 0 >> >> >> >> Keyword Search 155:52:59.626 (36%) >> >> File Type Identification 139:55:17.562 (32%) >> >> Hash Lookup 48:49:44.445 (11%) >> >> Embedded File Extractor 46:01:20.323 (10%) >> >> Email Parser 28:37:58.461 (6%) >> >> Extension Mismatch Detector 4:51:51.763 (1%) >> >> Exif Parser 1:41:39.597 (0%) >> >> PhotoRec Carver 0:00:04.762 (0%) >> >> Interesting Files Identifier 0:00:00.105 (0%) >> >> >> >> I get a bunch of errors but that is pretty typical. >> >> >> >> >> ------------------------------------------------------------ >> ------------------ >> >> _______________________________________________ >> sleuthkit-users mailing list >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users >> http://www.sleuthkit.org >> >> >> >> ------------------------------------------------------------ >> ------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot >> _______________________________________________ >> sleuthkit-users mailing list >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users >> http://www.sleuthkit.org >> >> > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > _______________________________________________ > sleuthkit-users mailing list > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > http://www.sleuthkit.org > > |
