Re: [sleuthkit-users] Slow Ingest

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

If you have the time, opportunity and disk space, could you convert the e01
to raw/dd and retry? I suspect that in some cases, libewf may be spending a
lot of time seeking in compressed e01 files. If you're traversing a complex
file system that requires a lot of seeking, my hunch is that libewf, which
is also single threaded as far as I understand, may be your bottleneck.
It'd be interesting to hear if simply changing to a raw image makes much of
a difference.

https://github.com/libyal/libewf

Regards, Ketil

On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:

> I would like to circle back on this.  I made the suggested modifications
> to reflect the physical cores of my system versus the hyper thread count.
> It honestly didn’t make much a difference.  When I reran the ingest it took
> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
> was at 3 Percent.
>
>
>
> One problem might be that my case was hosted on an OCZ350 Revo drive and
> it was waring of Overheat.   I circled back and deleted the case, then
> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
>
>
>
>
>
>
>
>
>
> As I read the perfmon data the system is just ticking over, not really
> pushing hard.  Are  my expectations off?  The source drive image is 488 Gig
> in eo1 format.  How long should an ingest take for such a drive.  What are
> the system performance specs I should be working to improve to bring this
> down.  My goal with migrating to the new system was to get a case ingest
> down to under 24 hours.
>
>
>
>
>
> *From:* MATT PIERCE
> *Sent:* Wednesday, September 14, 2016 6:25 PM
> *To:* Richard Cordovano <rco...@ba...>
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
> defined so I might skip that module.
>
>
>
> Sent Mobily
> ------------------------------
>
> *From:* Richard Cordovano <rco...@ba...>
> *Sent:* Sep 14, 2016 6:03 PM
> *To:* MATT PIERCE
> *Cc:* sle...@li...
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> It looks like you are overtaxing your system with too many file ingest
> threads.The error notifications you are getting should not be considered to
> be typical and look to be symptomatic of a system struggling with out of
> memory errors. Based on your ingest progress snapshot, all file level
> analysis has been completed except keyword search, which is the most memory
> intensive aspect of Autopsy data source ingest.
>
>
>
> >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set
> number of threads to 12 as suggested by the Options dialog.
>
>
>
> We generally recommend at most one file ingest thread per core, and the
> code for the Autopsy options dialog is even more conservative. For the
> system you describe, I would expect the Java library we use to detect the
> available processors (java.lang.Runtime.getRuntime().availableProcessors())
> would return eight (http://ark.intel.com/products/37106/Intel-Xeon-
> Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed
> give you a recommendation of only six file ingest threads! However, it
> appears that the Java lib is reporting the "hyperthreads" (sixteen) rather
> than the cores, which would indeed result in a recommendation of twelve
> threads. We will need to look into this further! In the mean time, the best
> advice I can give you is back way off on file ingest threads.
>
>
>
> Sincerely,
>
> Richard Cordovano
>
> Basis Technology
>
>
>
> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> wrote:
>
> I’m working a case and again have issues with performance using Autopsy.
> I have setup a dedicated server for running Autopsy.  In two days of ingest
> I’m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can
> spot where my bottle neck is?
>
>
>
> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy
> is loaded on a Raid0 15k SAS volume.
>
>
>
> Autopsy Load.
>
> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on
> amd64; Cp1252; en_US (autopsy)
>
>
>
> The image is from a Windows 7 workstation.  FTKimager took the disk image
> in E01 format.  I have the NSRL known good hash database loaded.  I’ve set
> number of threads to 12 as suggested by the Options dialog.  I’m running
> the default ingest process with no 3rd party modules.
>
>
>
> Performance Diagnostics
>
>
>
> Ingest Progress Snapshot
>
>
>
> 1              IDLE                                       Wed Sep 14
> 01:10:44 CDT 2016   15:49:08.024       0
>
> 2              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 3              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.758         0
>
> 4              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> 2
>
> 5              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.764         0
>
> 6              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 7              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
> 8              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> 2
>
> 9              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.742         0
>
> 10           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.812         0
>
> 11           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> 2
>
> 12           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.811         0
>
> 13           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
>
>
> 2              2016-08-31-1-1.E01          15:23:00               193034
> 2.0938940654524942       84           2              36
> 34           0
>
>
>
> Keyword Search               155:52:59.626 (36%)
>
> File Type Identification  139:55:17.562 (32%)
>
> Hash Lookup      48:49:44.445 (11%)
>
> Embedded File Extractor              46:01:20.323 (10%)
>
> Email Parser       28:37:58.461 (6%)
>
> Extension Mismatch Detector    4:51:51.763 (1%)
>
> Exif Parser           1:41:39.597 (0%)
>
> PhotoRec Carver              0:00:04.762 (0%)
>
> Interesting Files Identifier            0:00:00.105 (0%)
>
>
>
> I get a bunch of errors but that is pretty typical.
>
>
>
>
> ------------------------------------------------------------
> ------------------
>
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>