Re: [sleuthkit-users] Slow Ingest

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

If you are not trying to look at intermediate results, you should get some
speed boost by going to Tools->Options->Keyword Search->General and setting
"Results update frequency during ingest" to "No periodic searches" instead
of the default frequency of every five minutes (if you have not already
done so).

On Fri, Oct 14, 2016 at 2:30 PM, MATT PIERCE <mat...@ad...> wrote:

> OK.  So I have the requested results.  These were both generated from the
> same case image as my first post on the same platform.  The source image
> was read from a pci SSD.  The case directory is a raid 0 SAS 15k.  There
> does appear to be a significant performance issue with e01 image
> processing.  I still would have expected better than 3 days to process a
> 500 gig disk image.  So what factors should I identify for to get that down
> to 24 hours?
>
>
>
> E01 Image File 7672 minutes
>
> Oct 04 16:56
>
> Oct 09 12:48:11
>
>
>
> Raw Image File 4688 minutes
>
> Oct 11 17:04
>
> Oct 14 11:12:21
>
>
>
>
>
> *From:* Richard Cordovano [mailto:rco...@ba...]
> *Sent:* Thursday, October 13, 2016 5:19 PM
> *To:* Ketil Froyn <ke...@fr...>
> *Cc:* MATT PIERCE <mat...@ad...>; sleuthkit-users <
> sle...@li...>
>
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> Ketil's hypothesis is very interesting, as is the remark about
> overheating.
>
>
>
> Does the image have a lot of unallocated space?
>
>
>
> Another possible test would be to reduce the number of file ingest threads
> to two and compare processing times.
>
>
>
> On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote:
>
> If you have the time, opportunity and disk space, could you convert the
> e01 to raw/dd and retry? I suspect that in some cases, libewf may be
> spending a lot of time seeking in compressed e01 files. If you're
> traversing a complex file system that requires a lot of seeking, my hunch
> is that libewf, which is also single threaded as far as I understand, may
> be your bottleneck. It'd be interesting to hear if simply changing to a raw
> image makes much of a difference.
>
> https://github.com/libyal/libewf
>
> Regards, Ketil
>
>
>
> On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:
>
> I would like to circle back on this.  I made the suggested modifications
> to reflect the physical cores of my system versus the hyper thread count.
> It honestly didn’t make much a difference.  When I reran the ingest it took
> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
> was at 3 Percent.
>
>
>
> One problem might be that my case was hosted on an OCZ350 Revo drive and
> it was waring of Overheat.   I circled back and deleted the case, then
> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
>
>
>
>
>
>
>
>
>
> As I read the perfmon data the system is just ticking over, not really
> pushing hard.  Are  my expectations off?  The source drive image is 488 Gig
> in eo1 format.  How long should an ingest take for such a drive.  What are
> the system performance specs I should be working to improve to bring this
> down.  My goal with migrating to the new system was to get a case ingest
> down to under 24 hours.
>
>
>
>
>
> *From:* MATT PIERCE
> *Sent:* Wednesday, September 14, 2016 6:25 PM
> *To:* Richard Cordovano <rco...@ba...>
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
> defined so I might skip that module.
>
>
>
> Sent Mobily
> ------------------------------
>
> *From:* Richard Cordovano <rco...@ba...>
> *Sent:* Sep 14, 2016 6:03 PM
> *To:* MATT PIERCE
> *Cc:* sle...@li...
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> It looks like you are overtaxing your system with too many file ingest
> threads.The error notifications you are getting should not be considered to
> be typical and look to be symptomatic of a system struggling with out of
> memory errors. Based on your ingest progress snapshot, all file level
> analysis has been completed except keyword search, which is the most memory
> intensive aspect of Autopsy data source ingest.
>
>
>
> >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set
> number of threads to 12 as suggested by the Options dialog.
>
>
>
> We generally recommend at most one file ingest thread per core, and the
> code for the Autopsy options dialog is even more conservative. For the
> system you describe, I would expect the Java library we use to detect the
> available processors (java.lang.Runtime.getRuntime().availableProcessors())
> would return eight (http://ark.intel.com/products/37106/Intel-Xeon-
> Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed
> give you a recommendation of only six file ingest threads! However, it
> appears that the Java lib is reporting the "hyperthreads" (sixteen) rather
> than the cores, which would indeed result in a recommendation of twelve
> threads. We will need to look into this further! In the mean time, the best
> advice I can give you is back way off on file ingest threads.
>
>
>
> Sincerely,
>
> Richard Cordovano
>
> Basis Technology
>
>
>
> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> wrote:
>
> I’m working a case and again have issues with performance using Autopsy.
> I have setup a dedicated server for running Autopsy.  In two days of ingest
> I’m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can
> spot where my bottle neck is?
>
>
>
> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy
> is loaded on a Raid0 15k SAS volume.
>
>
>
> Autopsy Load.
>
> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on
> amd64; Cp1252; en_US (autopsy)
>
>
>
> The image is from a Windows 7 workstation.  FTKimager took the disk image
> in E01 format.  I have the NSRL known good hash database loaded.  I’ve set
> number of threads to 12 as suggested by the Options dialog.  I’m running
> the default ingest process with no 3rd party modules.
>
>
>
> Performance Diagnostics
>
>
>
> Ingest Progress Snapshot
>
>
>
> 1              IDLE                                       Wed Sep 14
> 01:10:44 CDT 2016   15:49:08.024       0
>
> 2              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 3              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.758         0
>
> 4              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> 2
>
> 5              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.764         0
>
> 6              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 7              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
> 8              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> 2
>
> 9              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.742         0
>
> 10           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.812         0
>
> 11           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> 2
>
> 12           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.811         0
>
> 13           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
>
>
> 2              2016-08-31-1-1.E01          15:23:00               193034
> 2.0938940654524942       84           2              36
> 34           0
>
>
>
> Keyword Search               155:52:59.626 (36%)
>
> File Type Identification  139:55:17.562 (32%)
>
> Hash Lookup      48:49:44.445 (11%)
>
> Embedded File Extractor              46:01:20.323 (10%)
>
> Email Parser       28:37:58.461 (6%)
>
> Extension Mismatch Detector    4:51:51.763 (1%)
>
> Exif Parser           1:41:39.597 (0%)
>
> PhotoRec Carver              0:00:04.762 (0%)
>
> Interesting Files Identifier            0:00:00.105 (0%)
>
>
>
> I get a bunch of errors but that is pretty typical.
>
>
>
>
> ------------------------------------------------------------
> ------------------
>
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>