sleuthkit-users — List to discuss Autopsy and The Sleuth Kit.

Re: [sleuthkit-users] Slow Ingest

[Richard C. <rco...@ba...>]

If you are not trying to look at intermediate results, you should get some
speed boost by going to Tools->Options->Keyword Search->General and setting
"Results update frequency during ingest" to "No periodic searches" instead
of the default frequency of every five minutes (if you have not already
done so).

On Fri, Oct 14, 2016 at 2:30 PM, MATT PIERCE <mat...@ad...> wrote:

> OK.  So I have the requested results.  These were both generated from the
> same case image as my first post on the same platform.  The source image
> was read from a pci SSD.  The case directory is a raid 0 SAS 15k.  There
> does appear to be a significant performance issue with e01 image
> processing.  I still would have expected better than 3 days to process a
> 500 gig disk image.  So what factors should I identify for to get that down
> to 24 hours?
>
>
>
> E01 Image File 7672 minutes
>
> Oct 04 16:56
>
> Oct 09 12:48:11
>
>
>
> Raw Image File 4688 minutes
>
> Oct 11 17:04
>
> Oct 14 11:12:21
>
>
>
>
>
> *From:* Richard Cordovano [mailto:rco...@ba...]
> *Sent:* Thursday, October 13, 2016 5:19 PM
> *To:* Ketil Froyn <ke...@fr...>
> *Cc:* MATT PIERCE <mat...@ad...>; sleuthkit-users <
> sle...@li...>
>
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> Ketil's hypothesis is very interesting, as is the remark about
> overheating.
>
>
>
> Does the image have a lot of unallocated space?
>
>
>
> Another possible test would be to reduce the number of file ingest threads
> to two and compare processing times.
>
>
>
> On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote:
>
> If you have the time, opportunity and disk space, could you convert the
> e01 to raw/dd and retry? I suspect that in some cases, libewf may be
> spending a lot of time seeking in compressed e01 files. If you're
> traversing a complex file system that requires a lot of seeking, my hunch
> is that libewf, which is also single threaded as far as I understand, may
> be your bottleneck. It'd be interesting to hear if simply changing to a raw
> image makes much of a difference.
>
> https://github.com/libyal/libewf
>
> Regards, Ketil
>
>
>
> On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:
>
> I would like to circle back on this.  I made the suggested modifications
> to reflect the physical cores of my system versus the hyper thread count.
> It honestly didn’t make much a difference.  When I reran the ingest it took
> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
> was at 3 Percent.
>
>
>
> One problem might be that my case was hosted on an OCZ350 Revo drive and
> it was waring of Overheat.   I circled back and deleted the case, then
> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
>
>
>
>
>
>
>
>
>
> As I read the perfmon data the system is just ticking over, not really
> pushing hard.  Are  my expectations off?  The source drive image is 488 Gig
> in eo1 format.  How long should an ingest take for such a drive.  What are
> the system performance specs I should be working to improve to bring this
> down.  My goal with migrating to the new system was to get a case ingest
> down to under 24 hours.
>
>
>
>
>
> *From:* MATT PIERCE
> *Sent:* Wednesday, September 14, 2016 6:25 PM
> *To:* Richard Cordovano <rco...@ba...>
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
> defined so I might skip that module.
>
>
>
> Sent Mobily
> ------------------------------
>
> *From:* Richard Cordovano <rco...@ba...>
> *Sent:* Sep 14, 2016 6:03 PM
> *To:* MATT PIERCE
> *Cc:* sle...@li...
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> It looks like you are overtaxing your system with too many file ingest
> threads.The error notifications you are getting should not be considered to
> be typical and look to be symptomatic of a system struggling with out of
> memory errors. Based on your ingest progress snapshot, all file level
> analysis has been completed except keyword search, which is the most memory
> intensive aspect of Autopsy data source ingest.
>
>
>
> >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set
> number of threads to 12 as suggested by the Options dialog.
>
>
>
> We generally recommend at most one file ingest thread per core, and the
> code for the Autopsy options dialog is even more conservative. For the
> system you describe, I would expect the Java library we use to detect the
> available processors (java.lang.Runtime.getRuntime().availableProcessors())
> would return eight (http://ark.intel.com/products/37106/Intel-Xeon-
> Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed
> give you a recommendation of only six file ingest threads! However, it
> appears that the Java lib is reporting the "hyperthreads" (sixteen) rather
> than the cores, which would indeed result in a recommendation of twelve
> threads. We will need to look into this further! In the mean time, the best
> advice I can give you is back way off on file ingest threads.
>
>
>
> Sincerely,
>
> Richard Cordovano
>
> Basis Technology
>
>
>
> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> wrote:
>
> I’m working a case and again have issues with performance using Autopsy.
> I have setup a dedicated server for running Autopsy.  In two days of ingest
> I’m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can
> spot where my bottle neck is?
>
>
>
> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy
> is loaded on a Raid0 15k SAS volume.
>
>
>
> Autopsy Load.
>
> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on
> amd64; Cp1252; en_US (autopsy)
>
>
>
> The image is from a Windows 7 workstation.  FTKimager took the disk image
> in E01 format.  I have the NSRL known good hash database loaded.  I’ve set
> number of threads to 12 as suggested by the Options dialog.  I’m running
> the default ingest process with no 3rd party modules.
>
>
>
> Performance Diagnostics
>
>
>
> Ingest Progress Snapshot
>
>
>
> 1              IDLE                                       Wed Sep 14
> 01:10:44 CDT 2016   15:49:08.024       0
>
> 2              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 3              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.758         0
>
> 4              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> 2
>
> 5              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.764         0
>
> 6              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 7              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
> 8              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> 2
>
> 9              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.742         0
>
> 10           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.812         0
>
> 11           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> 2
>
> 12           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.811         0
>
> 13           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
>
>
> 2              2016-08-31-1-1.E01          15:23:00               193034
> 2.0938940654524942       84           2              36
> 34           0
>
>
>
> Keyword Search               155:52:59.626 (36%)
>
> File Type Identification  139:55:17.562 (32%)
>
> Hash Lookup      48:49:44.445 (11%)
>
> Embedded File Extractor              46:01:20.323 (10%)
>
> Email Parser       28:37:58.461 (6%)
>
> Extension Mismatch Detector    4:51:51.763 (1%)
>
> Exif Parser           1:41:39.597 (0%)
>
> PhotoRec Carver              0:00:04.762 (0%)
>
> Interesting Files Identifier            0:00:00.105 (0%)
>
>
>
> I get a bunch of errors but that is pretty typical.
>
>
>
>
> ------------------------------------------------------------
> ------------------
>
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Slow Ingest

[MATT P. <mat...@ad...>]

OK.  So I have the requested results.  These were both generated from the same case image as my first post on the same platform.  The source image was read from a pci SSD.  The case directory is a raid 0 SAS 15k.  There does appear to be a significant performance issue with e01 image processing.  I still would have expected better than 3 days to process a 500 gig disk image.  So what factors should I identify for to get that down to 24 hours?

E01 Image File 7672 minutes
Oct 04 16:56
Oct 09 12:48:11

Raw Image File 4688 minutes
Oct 11 17:04
Oct 14 11:12:21


From: Richard Cordovano [mailto:rco...@ba...]
Sent: Thursday, October 13, 2016 5:19 PM
To: Ketil Froyn <ke...@fr...>
Cc: MATT PIERCE <mat...@ad...>; sleuthkit-users <sle...@li...>
Subject: Re: [sleuthkit-users] Slow Ingest

Ketil's hypothesis is very interesting, as is the remark about overheating.

Does the image have a lot of unallocated space?

Another possible test would be to reduce the number of file ingest threads to two and compare processing times.

On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...<mailto:ke...@fr...>> wrote:

If you have the time, opportunity and disk space, could you convert the e01 to raw/dd and retry? I suspect that in some cases, libewf may be spending a lot of time seeking in compressed e01 files. If you're traversing a complex file system that requires a lot of seeking, my hunch is that libewf, which is also single threaded as far as I understand, may be your bottleneck. It'd be interesting to hear if simply changing to a raw image makes much of a difference.

https://github.com/libyal/libewf

Regards, Ketil

On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...<mailto:mat...@ad...>> wrote:
I would like to circle back on this.  I made the suggested modifications to reflect the physical cores of my system versus the hyper thread count.  It honestly didn’t make much a difference.  When I reran the ingest it took 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search was at 3 Percent.

One problem might be that my case was hosted on an OCZ350 Revo drive and it was waring of Overheat.   I circled back and deleted the case, then reran it with the case on 15k SAS raid0 and the source file on 10k SAS.

[cid:image003.jpg@01D2261D.34376B80]

[cid:image004.jpg@01D2261D.34376B80]

[cid:image005.jpg@01D2261D.34376B80]

As I read the perfmon data the system is just ticking over, not really pushing hard.  Are  my expectations off?  The source drive image is 488 Gig in eo1 format.  How long should an ingest take for such a drive.  What are the system performance specs I should be working to improve to bring this down.  My goal with migrating to the new system was to get a case ingest down to under 24 hours.


From: MATT PIERCE
Sent: Wednesday, September 14, 2016 6:25 PM
To: Richard Cordovano <rco...@ba...<mailto:rco...@ba...>>
Subject: Re: [sleuthkit-users] Slow Ingest

Thank you.  I'm restarting the run tomorrow.  I don't have any keywords defined so I might skip that module.

Sent Mobily
________________________________
From: Richard Cordovano <rco...@ba...<mailto:rco...@ba...>>
Sent: Sep 14, 2016 6:03 PM
To: MATT PIERCE
Cc: sle...@li...<mailto:sle...@li...>
Subject: Re: [sleuthkit-users] Slow Ingest

It looks like you are overtaxing your system with too many file ingest threads.The error notifications you are getting should not be considered to be typical and look to be symptomatic of a system struggling with out of memory errors. Based on your ingest progress snapshot, all file level analysis has been completed except keyword search, which is the most memory intensive aspect of Autopsy data source ingest.

>> The system has dual Xeon x5550 2.66 quad core processors...I’ve set number of threads to 12 as suggested by the Options dialog.

We generally recommend at most one file ingest thread per core, and the code for the Autopsy options dialog is even more conservative. For the system you describe, I would expect the Java library we use to detect the available processors (java.lang.Runtime.getRuntime().availableProcessors()) would return eight (http://ark.intel.com/products/37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed give you a recommendation of only six file ingest threads! However, it appears that the Java lib is reporting the "hyperthreads" (sixteen) rather than the cores, which would indeed result in a recommendation of twelve threads. We will need to look into this further! In the mean time, the best advice I can give you is back way off on file ingest threads.

Sincerely,
Richard Cordovano
Basis Technology

On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...<mailto:mat...@ad...>> wrote:
I’m working a case and again have issues with performance using Autopsy.  I have setup a dedicated server for running Autopsy.  In two days of ingest I’m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can spot where my bottle neck is?

The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.  Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy is loaded on a Raid0 15k SAS volume.

Autopsy Load.
Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on amd64; Cp1252; en_US (autopsy)

The image is from a Windows 7 workstation.  FTKimager took the disk image in E01 format.  I have the NSRL known good hash database loaded.  I’ve set number of threads to 12 as suggested by the Options dialog.  I’m running the default ingest process with no 3rd party modules.

Performance Diagnostics
[cid:image006.png@01D2261D.34376B80]

Ingest Progress Snapshot

1              IDLE                                       Wed Sep 14 01:10:44 CDT 2016   15:49:08.024       0
2              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759                2
3              IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.758         0
4              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798                2
5              IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.764         0
6              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759                2
7              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155                2
8              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132                2
9              IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.742         0
10           IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.812         0
11           Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120                2
12           IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.811         0
13           Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155                2

2              2016-08-31-1-1.E01          15:23:00               193034  2.0938940654524942       84           2              36           34           0

Keyword Search               155:52:59.626 (36%)
File Type Identification  139:55:17.562 (32%)
Hash Lookup      48:49:44.445 (11%)
Embedded File Extractor              46:01:20.323 (10%)
Email Parser       28:37:58.461 (6%)
Extension Mismatch Detector    4:51:51.763 (1%)
Exif Parser           1:41:39.597 (0%)
PhotoRec Carver              0:00:04.762 (0%)
Interesting Files Identifier            0:00:00.105 (0%)

I get a bunch of errors but that is pretty typical.

[cid:image007.png@01D2261D.34376B80]

------------------------------------------------------------------------------

_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org

Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6

[Joachim M. <joa...@gm...>]

They are LGPL v3 not GPL, they can be linked in statically, there is no
issue if the rest of the code is LGPL compatible

https://en.wikipedia.org/wiki/License_compatibility

However there more licenses to consider such as that of the VS C runtime.

On Fri, Oct 14, 2016 at 3:38 PM, Brian Carrier <ca...@sl...>
wrote:

> Excellent point.  Given that libewf et al. are GPL, they cannot be
> statically linked in.
>
>
> > On Oct 14, 2016, at 1:01 AM, Joachim Metz <joa...@gm...>
> wrote:
> >
> > > It would be better to build static (or mostly static binaries) even
> through they end up being very large.
> >
> > Redistributing these might not be possible with certain licenses.
> >
> >
> > On Fri, Oct 14, 2016 at 12:19 AM, <sleuthkit-users-request@
> lists.sourceforge.net> wrote:
> > Send sleuthkit-users mailing list submissions to
> >         sle...@li...
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > or, via email, send a message with subject or body 'help' to
> >         sle...@li...
> >
> > You can reach the person managing the list at
> >         sle...@li...
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of sleuthkit-users digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: Windows Precompiled EXEs (Mark W. Jeanmougin)
> >    2. Re: Windows Precompiled EXEs (Michael Cohen)
> >    3. Re: Windows Precompiled EXEs (Brian Carrier)
> >    4. Re: Slow Ingest (Richard Cordovano)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Tue, 11 Oct 2016 21:21:42 -0400
> > From: "Mark W. Jeanmougin" <ma...@gm...>
> > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> > To: lfc...@gm...
> > Cc: "sle...@li... users"
> >         <sle...@li...>
> > Message-ID:
> >         <CAMMgxRWnCpP7HB7WTq0uo=WwnnPBCTzs3QUCoHbX-9xkNM_eDg@mail.
> gmail.com>
> > Content-Type: text/plain; charset=UTF-8
> >
> > I vote #1 for the same reasons as Luis.
> >
> > MJ
> >
> >
> > On Tue, Oct 11, 2016 at 9:09 PM, Lu?s Filipe Nassif <lfc...@gm...>
> wrote:
> > > I prefer option 1, it is much more portable, you can run tsk tools
> from a
> > > thumb drive in the Field for example.
> > >
> > > Luis Nassif
> > >
> > >
> > > Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...>
> > > escreveu:
> > >>
> > >> As was mentioned in another thread, we?re looking at having The
> Sleuth Kit
> > >> use a more modern Visual Studio compiler.  I need some feedback on how
> > >> people are using the precompiled EXEs that we ship.
> > >>
> > >> One of the big changes is that executables now depend on a lot more
> > >> official microsoft dlls (C runtime dlls).
> > >>
> > >> Historically, I?ve always tried to make it be so that you can open a
> ZIP
> > >> file with the EXEs and run them with no prerequisites. We did this by
> > >> shipping the 2 or 3 run time dlls in the same folder as the exes.
> > >>
> > >> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there
> are
> > >> two options:
> > >>
> > >> 1) We copy all 40+ dlls into the folder like we did before and it is
> now
> > >> just a bit more work to find the exe tools in there among all of the
> dlls
> > >> (they are all similarly named, which makes it somewhat easy).
> > >>
> > >> 2) We make the user run a Microsoft Redistributable Installer to
> install
> > >> the needed dlls into the c:\windows folder if they are not already
> there. We
> > >> can then ship a ZIP file and the user has to know to install the MS
> redist
> > >> or we start shipping a TSK installer that also installs the MS redist.
> > >>
> > >>
> > >> Opinions?    Do people care more about prerequisites or big folders?
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------
> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> sleuthkit-users mailing list
> > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > >> http://www.sleuthkit.org
> > >
> > >
> > > ------------------------------------------------------------
> ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> > >
> >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Tue, 11 Oct 2016 22:44:37 -0700
> > From: Michael Cohen <scu...@gm...>
> > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> > To: Brian Carrier <ca...@sl...>
> > Cc: "sle...@li... users"
> >         <sle...@li...>
> > Message-ID:
> >         <CAJ...@ma...ail.
> com>
> > Content-Type: text/plain; charset=UTF-8
> >
> > It would be better to build static (or mostly static binaries) even
> > through they end up being very large. Depending on the system you have
> > to run on, SxS configurations can cause huge problems, even when the
> > right dlls are in the same folder as the binary (the system might
> > insist on using some other version which it decides is better).
> >
> > For pytsk we use the python build system to build the TSK library into
> > one big python module with no external dependencies. Having to ship
> > extra dlls that link at runtime to a python program is a pain and it
> > is error prone. It might make more sense to have a shared library
> > between all the tsk tools (libtsk) but I would opt to keep that as
> > dependency free as possible even if it means building it with static
> > linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> > (Because this is still the standard for python 2.7) so it would be
> > nice if the code remains portable.
> >
> > Thanks
> > Michael.
> >
> > On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> > > As was mentioned in another thread, we?re looking at having The Sleuth
> Kit
> > > use a more modern Visual Studio compiler.  I need some feedback on how
> > > people are using the precompiled EXEs that we ship.
> > >
> > > One of the big changes is that executables now depend on a lot more
> official
> > > microsoft dlls (C runtime dlls).
> > >
> > > Historically, I?ve always tried to make it be so that you can open a
> ZIP
> > > file with the EXEs and run them with no prerequisites. We did this by
> > > shipping the 2 or 3 run time dlls in the same folder as the exes.
> > >
> > > With Visual Studio 2015, there are over 40 dlls, not 2!   So, there
> are two
> > > options:
> > >
> > > 1) We copy all 40+ dlls into the folder like we did before and it is
> now
> > > just a bit more work to find the exe tools in there among all of the
> dlls
> > > (they are all similarly named, which makes it somewhat easy).
> > >
> > > 2) We make the user run a Microsoft Redistributable Installer to
> install the
> > > needed dlls into the c:\windows folder if they are not already there.
> We can
> > > then ship a ZIP file and the user has to know to install the MS redist
> or we
> > > start shipping a TSK installer that also installs the MS redist.
> > >
> > >
> > > Opinions?    Do people care more about prerequisites or big folders?
> > >
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------
> ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> > >
> >
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Wed, 12 Oct 2016 09:15:22 -0400
> > From: Brian Carrier <ca...@sl...>
> > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> > To: Michael Cohen <scu...@gm...>
> > Cc: "sle...@li... users"
> >         <sle...@li...>
> > Message-ID: <31B...@sl...>
> > Content-Type: text/plain; charset=utf-8
> >
> > Hi Michael,
> >
> > We thought about the entirely static approach, but we?d also then need
> libewf, libvhdi, zlib, etc. to be static and they don?t ship with a static
> library option.  We could obviously change their visual studio projects to
> make them, but we were first trying to do this as simply as possible.
> >
> > FWIW: There is now a Release_NoLibs build target for The Sleuth Kit that
> is static (i.e. no runtime dlls), but it doesn?t support E01 files.   We
> use that for our incident response Cyber Triage collection tool (?agent?).
> >
> > We?ll continue to keep it portable.  There are still a lot of gcc
> compilers out there that don?t like new fancy things.
> >
> > brian
> >
> >
> >
> > > On Oct 12, 2016, at 1:44 AM, Michael Cohen <scu...@gm...> wrote:
> > >
> > > It would be better to build static (or mostly static binaries) even
> > > through they end up being very large. Depending on the system you have
> > > to run on, SxS configurations can cause huge problems, even when the
> > > right dlls are in the same folder as the binary (the system might
> > > insist on using some other version which it decides is better).
> > >
> > > For pytsk we use the python build system to build the TSK library into
> > > one big python module with no external dependencies. Having to ship
> > > extra dlls that link at runtime to a python program is a pain and it
> > > is error prone. It might make more sense to have a shared library
> > > between all the tsk tools (libtsk) but I would opt to keep that as
> > > dependency free as possible even if it means building it with static
> > > linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> > > (Because this is still the standard for python 2.7) so it would be
> > > nice if the code remains portable.
> > >
> > > Thanks
> > > Michael.
> > >
> > > On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> > >> As was mentioned in another thread, we?re looking at having The
> Sleuth Kit
> > >> use a more modern Visual Studio compiler.  I need some feedback on how
> > >> people are using the precompiled EXEs that we ship.
> > >>
> > >> One of the big changes is that executables now depend on a lot more
> official
> > >> microsoft dlls (C runtime dlls).
> > >>
> > >> Historically, I?ve always tried to make it be so that you can open a
> ZIP
> > >> file with the EXEs and run them with no prerequisites. We did this by
> > >> shipping the 2 or 3 run time dlls in the same folder as the exes.
> > >>
> > >> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there
> are two
> > >> options:
> > >>
> > >> 1) We copy all 40+ dlls into the folder like we did before and it is
> now
> > >> just a bit more work to find the exe tools in there among all of the
> dlls
> > >> (they are all similarly named, which makes it somewhat easy).
> > >>
> > >> 2) We make the user run a Microsoft Redistributable Installer to
> install the
> > >> needed dlls into the c:\windows folder if they are not already there.
> We can
> > >> then ship a ZIP file and the user has to know to install the MS
> redist or we
> > >> start shipping a TSK installer that also installs the MS redist.
> > >>
> > >>
> > >> Opinions?    Do people care more about prerequisites or big folders?
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------
> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> sleuthkit-users mailing list
> > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > >> http://www.sleuthkit.org
> > >>
> > >
> > > ------------------------------------------------------------
> ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> >
> >
> >
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Thu, 13 Oct 2016 18:19:26 -0400
> > From: Richard Cordovano <rco...@ba...>
> > Subject: Re: [sleuthkit-users] Slow Ingest
> > To: Ketil Froyn <ke...@fr...>
> > Cc: sleuthkit-users <sle...@li...>
> > Message-ID:
> >         <CAPuuEWQ9FU6zo+0t4iVdQ4SmFa8S_XjeARcSCPa2c3ttXP+VQA@mail.
> gmail.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Ketil's hypothesis is very interesting, as is the remark about
> overheating.
> >
> > Does the image have a lot of unallocated space?
> >
> > Another possible test would be to reduce the number of file ingest
> threads
> > to two and compare processing times.
> >
> > On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote:
> >
> > > If you have the time, opportunity and disk space, could you convert the
> > > e01 to raw/dd and retry? I suspect that in some cases, libewf may be
> > > spending a lot of time seeking in compressed e01 files. If you're
> > > traversing a complex file system that requires a lot of seeking, my
> hunch
> > > is that libewf, which is also single threaded as far as I understand,
> may
> > > be your bottleneck. It'd be interesting to hear if simply changing to
> a raw
> > > image makes much of a difference.
> > >
> > > https://github.com/libyal/libewf
> > >
> > > Regards, Ketil
> > >
> > > On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:
> > >
> > >> I would like to circle back on this.  I made the suggested
> modifications
> > >> to reflect the physical cores of my system versus the hyper thread
> count.
> > >> It honestly didn?t make much a difference.  When I reran the ingest
> it took
> > >> 96 hours to complete the Analyzing Files stage.  Periodic Keyword
> Search
> > >> was at 3 Percent.
> > >>
> > >>
> > >>
> > >> One problem might be that my case was hosted on an OCZ350 Revo drive
> and
> > >> it was waring of Overheat.   I circled back and deleted the case, then
> > >> reran it with the case on 15k SAS raid0 and the source file on 10k
> SAS.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> As I read the perfmon data the system is just ticking over, not really
> > >> pushing hard.  Are  my expectations off?  The source drive image is
> 488 Gig
> > >> in eo1 format.  How long should an ingest take for such a drive.
> What are
> > >> the system performance specs I should be working to improve to bring
> this
> > >> down.  My goal with migrating to the new system was to get a case
> ingest
> > >> down to under 24 hours.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> *From:* MATT PIERCE
> > >> *Sent:* Wednesday, September 14, 2016 6:25 PM
> > >> *To:* Richard Cordovano <rco...@ba...>
> > >> *Subject:* Re: [sleuthkit-users] Slow Ingest
> > >>
> > >>
> > >>
> > >> Thank you.  I'm restarting the run tomorrow.  I don't have any
> keywords
> > >> defined so I might skip that module.
> > >>
> > >>
> > >>
> > >> Sent Mobily
> > >> ------------------------------
> > >>
> > >> *From:* Richard Cordovano <rco...@ba...>
> > >> *Sent:* Sep 14, 2016 6:03 PM
> > >> *To:* MATT PIERCE
> > >> *Cc:* sle...@li...
> > >> *Subject:* Re: [sleuthkit-users] Slow Ingest
> > >>
> > >>
> > >>
> > >> It looks like you are overtaxing your system with too many file ingest
> > >> threads.The error notifications you are getting should not be
> considered to
> > >> be typical and look to be symptomatic of a system struggling with out
> of
> > >> memory errors. Based on your ingest progress snapshot, all file level
> > >> analysis has been completed except keyword search, which is the most
> memory
> > >> intensive aspect of Autopsy data source ingest.
> > >>
> > >>
> > >>
> > >> >> The system has dual Xeon x5550 2.66 quad core processors...I?ve set
> > >> number of threads to 12 as suggested by the Options dialog.
> > >>
> > >>
> > >>
> > >> We generally recommend at most one file ingest thread per core, and
> the
> > >> code for the Autopsy options dialog is even more conservative. For the
> > >> system you describe, I would expect the Java library we use to detect
> the
> > >> available processors (java.lang.Runtime.getRuntime(
> ).availableProcessors())
> > >> would return eight (http://ark.intel.com/products
> > >> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-
> GTs-Intel-QPI),
> > >> which would indeed give you a recommendation of only six file ingest
> > >> threads! However, it appears that the Java lib is reporting the
> > >> "hyperthreads" (sixteen) rather than the cores, which would indeed
> result
> > >> in a recommendation of twelve threads. We will need to look into this
> > >> further! In the mean time, the best advice I can give you is back way
> off
> > >> on file ingest threads.
> > >>
> > >>
> > >>
> > >> Sincerely,
> > >>
> > >> Richard Cordovano
> > >>
> > >> Basis Technology
> > >>
> > >>
> > >>
> > >> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> > >> wrote:
> > >>
> > >> I?m working a case and again have issues with performance using
> Autopsy.
> > >> I have setup a dedicated server for running Autopsy.  In two days of
> ingest
> > >> I?m at 45%.  In 8 hours it has only progressed 7%. I was hoping
> someone can
> > >> spot where my bottle neck is?
> > >>
> > >>
> > >>
> > >> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> > >> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.
> Autopsy
> > >> is loaded on a Raid0 15k SAS volume.
> > >>
> > >>
> > >>
> > >> Autopsy Load.
> > >>
> > >> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> > >> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM)
> 64-Bit
> > >> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3
> running on
> > >> amd64; Cp1252; en_US (autopsy)
> > >>
> > >>
> > >>
> > >> The image is from a Windows 7 workstation.  FTKimager took the disk
> image
> > >> in E01 format.  I have the NSRL known good hash database loaded.
> I?ve set
> > >> number of threads to 12 as suggested by the Options dialog.  I?m
> running
> > >> the default ingest process with no 3rd party modules.
> > >>
> > >>
> > >>
> > >> Performance Diagnostics
> > >>
> > >>
> > >>
> > >> Ingest Progress Snapshot
> > >>
> > >>
> > >>
> > >> 1              IDLE                                       Wed Sep 14
> > >> 01:10:44 CDT 2016   15:49:08.024       0
> > >>
> > >> 2              Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> > >> 2
> > >>
> > >> 3              IDLE                                       Wed Sep 14
> > >> 16:59:26 CDT 2016   0:00:25.758         0
> > >>
> > >> 4              Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> > >> 2
> > >>
> > >> 5              IDLE                                       Wed Sep 14
> > >> 16:59:26 CDT 2016   0:00:25.764         0
> > >>
> > >> 6              Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> > >> 2
> > >>
> > >> 7              Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> > >> 2
> > >>
> > >> 8              Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> > >> 2
> > >>
> > >> 9              IDLE                                       Wed Sep 14
> > >> 16:59:26 CDT 2016   0:00:25.742         0
> > >>
> > >> 10           IDLE                                       Wed Sep 14
> > >> 16:59:26 CDT 2016   0:00:25.812         0
> > >>
> > >> 11           Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> > >> 2
> > >>
> > >> 12           IDLE                                       Wed Sep 14
> > >> 16:59:26 CDT 2016   0:00:25.811         0
> > >>
> > >> 13           Keyword Search               2016-08-31-1-1.E01
> > >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> > >> 2
> > >>
> > >>
> > >>
> > >> 2              2016-08-31-1-1.E01          15:23:00
>  193034
> > >> 2.0938940654524942       84           2              36
> > >> 34           0
> > >>
> > >>
> > >>
> > >> Keyword Search               155:52:59.626 (36%)
> > >>
> > >> File Type Identification  139:55:17.562 (32%)
> > >>
> > >> Hash Lookup      48:49:44.445 (11%)
> > >>
> > >> Embedded File Extractor              46:01:20.323 (10%)
> > >>
> > >> Email Parser       28:37:58.461 (6%)
> > >>
> > >> Extension Mismatch Detector    4:51:51.763 (1%)
> > >>
> > >> Exif Parser           1:41:39.597 (0%)
> > >>
> > >> PhotoRec Carver              0:00:04.762 (0%)
> > >>
> > >> Interesting Files Identifier            0:00:00.105 (0%)
> > >>
> > >>
> > >>
> > >> I get a bunch of errors but that is pretty typical.
> > >>
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------
> > >> ------------------
> > >>
> > >> _______________________________________________
> > >> sleuthkit-users mailing list
> > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > >> http://www.sleuthkit.org
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------
> > >> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> sleuthkit-users mailing list
> > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > >> http://www.sleuthkit.org
> > >>
> > >>
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> > >
> > >
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: image008.jpg
> > Type: image/jpeg
> > Size: 38407 bytes
> > Desc: not available
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: image007.jpg
> > Type: image/jpeg
> > Size: 12346 bytes
> > Desc: not available
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: image002.png
> > Type: image/png
> > Size: 19142 bytes
> > Desc: not available
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: image009.jpg
> > Type: image/jpeg
> > Size: 93667 bytes
> > Desc: not available
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: image001.png
> > Type: image/png
> > Size: 13311 bytes
> > Desc: not available
> >
> > ------------------------------
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >
> > ------------------------------
> >
> > _______________________________________________
> > sleuthkit-users mailing list
> > sle...@li...
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >
> >
> > End of sleuthkit-users Digest, Vol 124, Issue 6
> > ***********************************************
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot______
> _________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6

[Brian C. <ca...@sl...>]

Excellent point.  Given that libewf et al. are GPL, they cannot be statically linked in. 


> On Oct 14, 2016, at 1:01 AM, Joachim Metz <joa...@gm...> wrote:
> 
> > It would be better to build static (or mostly static binaries) even through they end up being very large. 
> 
> Redistributing these might not be possible with certain licenses.
> 
> 
> On Fri, Oct 14, 2016 at 12:19 AM, <sle...@li...> wrote:
> Send sleuthkit-users mailing list submissions to
>         sle...@li...
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> or, via email, send a message with subject or body 'help' to
>         sle...@li...
> 
> You can reach the person managing the list at
>         sle...@li...
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of sleuthkit-users digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: Windows Precompiled EXEs (Mark W. Jeanmougin)
>    2. Re: Windows Precompiled EXEs (Michael Cohen)
>    3. Re: Windows Precompiled EXEs (Brian Carrier)
>    4. Re: Slow Ingest (Richard Cordovano)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 11 Oct 2016 21:21:42 -0400
> From: "Mark W. Jeanmougin" <ma...@gm...>
> Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> To: lfc...@gm...
> Cc: "sle...@li... users"
>         <sle...@li...>
> Message-ID:
>         <CAMMgxRWnCpP7HB7WTq0uo=Wwn...@ma...>
> Content-Type: text/plain; charset=UTF-8
> 
> I vote #1 for the same reasons as Luis.
> 
> MJ
> 
> 
> On Tue, Oct 11, 2016 at 9:09 PM, Lu?s Filipe Nassif <lfc...@gm...> wrote:
> > I prefer option 1, it is much more portable, you can run tsk tools from a
> > thumb drive in the Field for example.
> >
> > Luis Nassif
> >
> >
> > Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...>
> > escreveu:
> >>
> >> As was mentioned in another thread, we?re looking at having The Sleuth Kit
> >> use a more modern Visual Studio compiler.  I need some feedback on how
> >> people are using the precompiled EXEs that we ship.
> >>
> >> One of the big changes is that executables now depend on a lot more
> >> official microsoft dlls (C runtime dlls).
> >>
> >> Historically, I?ve always tried to make it be so that you can open a ZIP
> >> file with the EXEs and run them with no prerequisites. We did this by
> >> shipping the 2 or 3 run time dlls in the same folder as the exes.
> >>
> >> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> >> two options:
> >>
> >> 1) We copy all 40+ dlls into the folder like we did before and it is now
> >> just a bit more work to find the exe tools in there among all of the dlls
> >> (they are all similarly named, which makes it somewhat easy).
> >>
> >> 2) We make the user run a Microsoft Redistributable Installer to install
> >> the needed dlls into the c:\windows folder if they are not already there. We
> >> can then ship a ZIP file and the user has to know to install the MS redist
> >> or we start shipping a TSK installer that also installs the MS redist.
> >>
> >>
> >> Opinions?    Do people care more about prerequisites or big folders?
> >>
> >>
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >
> >
> > ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> >
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Tue, 11 Oct 2016 22:44:37 -0700
> From: Michael Cohen <scu...@gm...>
> Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> To: Brian Carrier <ca...@sl...>
> Cc: "sle...@li... users"
>         <sle...@li...>
> Message-ID:
>         <CAJ...@ma...>
> Content-Type: text/plain; charset=UTF-8
> 
> It would be better to build static (or mostly static binaries) even
> through they end up being very large. Depending on the system you have
> to run on, SxS configurations can cause huge problems, even when the
> right dlls are in the same folder as the binary (the system might
> insist on using some other version which it decides is better).
> 
> For pytsk we use the python build system to build the TSK library into
> one big python module with no external dependencies. Having to ship
> extra dlls that link at runtime to a python program is a pain and it
> is error prone. It might make more sense to have a shared library
> between all the tsk tools (libtsk) but I would opt to keep that as
> dependency free as possible even if it means building it with static
> linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> (Because this is still the standard for python 2.7) so it would be
> nice if the code remains portable.
> 
> Thanks
> Michael.
> 
> On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> > As was mentioned in another thread, we?re looking at having The Sleuth Kit
> > use a more modern Visual Studio compiler.  I need some feedback on how
> > people are using the precompiled EXEs that we ship.
> >
> > One of the big changes is that executables now depend on a lot more official
> > microsoft dlls (C runtime dlls).
> >
> > Historically, I?ve always tried to make it be so that you can open a ZIP
> > file with the EXEs and run them with no prerequisites. We did this by
> > shipping the 2 or 3 run time dlls in the same folder as the exes.
> >
> > With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are two
> > options:
> >
> > 1) We copy all 40+ dlls into the folder like we did before and it is now
> > just a bit more work to find the exe tools in there among all of the dlls
> > (they are all similarly named, which makes it somewhat easy).
> >
> > 2) We make the user run a Microsoft Redistributable Installer to install the
> > needed dlls into the c:\windows folder if they are not already there. We can
> > then ship a ZIP file and the user has to know to install the MS redist or we
> > start shipping a TSK installer that also installs the MS redist.
> >
> >
> > Opinions?    Do people care more about prerequisites or big folders?
> >
> >
> >
> >
> >
> > ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> >
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Wed, 12 Oct 2016 09:15:22 -0400
> From: Brian Carrier <ca...@sl...>
> Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> To: Michael Cohen <scu...@gm...>
> Cc: "sle...@li... users"
>         <sle...@li...>
> Message-ID: <31B...@sl...>
> Content-Type: text/plain; charset=utf-8
> 
> Hi Michael,
> 
> We thought about the entirely static approach, but we?d also then need libewf, libvhdi, zlib, etc. to be static and they don?t ship with a static library option.  We could obviously change their visual studio projects to make them, but we were first trying to do this as simply as possible.
> 
> FWIW: There is now a Release_NoLibs build target for The Sleuth Kit that is static (i.e. no runtime dlls), but it doesn?t support E01 files.   We use that for our incident response Cyber Triage collection tool (?agent?).
> 
> We?ll continue to keep it portable.  There are still a lot of gcc compilers out there that don?t like new fancy things.
> 
> brian
> 
> 
> 
> > On Oct 12, 2016, at 1:44 AM, Michael Cohen <scu...@gm...> wrote:
> >
> > It would be better to build static (or mostly static binaries) even
> > through they end up being very large. Depending on the system you have
> > to run on, SxS configurations can cause huge problems, even when the
> > right dlls are in the same folder as the binary (the system might
> > insist on using some other version which it decides is better).
> >
> > For pytsk we use the python build system to build the TSK library into
> > one big python module with no external dependencies. Having to ship
> > extra dlls that link at runtime to a python program is a pain and it
> > is error prone. It might make more sense to have a shared library
> > between all the tsk tools (libtsk) but I would opt to keep that as
> > dependency free as possible even if it means building it with static
> > linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> > (Because this is still the standard for python 2.7) so it would be
> > nice if the code remains portable.
> >
> > Thanks
> > Michael.
> >
> > On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> >> As was mentioned in another thread, we?re looking at having The Sleuth Kit
> >> use a more modern Visual Studio compiler.  I need some feedback on how
> >> people are using the precompiled EXEs that we ship.
> >>
> >> One of the big changes is that executables now depend on a lot more official
> >> microsoft dlls (C runtime dlls).
> >>
> >> Historically, I?ve always tried to make it be so that you can open a ZIP
> >> file with the EXEs and run them with no prerequisites. We did this by
> >> shipping the 2 or 3 run time dlls in the same folder as the exes.
> >>
> >> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are two
> >> options:
> >>
> >> 1) We copy all 40+ dlls into the folder like we did before and it is now
> >> just a bit more work to find the exe tools in there among all of the dlls
> >> (they are all similarly named, which makes it somewhat easy).
> >>
> >> 2) We make the user run a Microsoft Redistributable Installer to install the
> >> needed dlls into the c:\windows folder if they are not already there. We can
> >> then ship a ZIP file and the user has to know to install the MS redist or we
> >> start shipping a TSK installer that also installs the MS redist.
> >>
> >>
> >> Opinions?    Do people care more about prerequisites or big folders?
> >>
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >>
> >
> > ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> 
> 
> 
> 
> ------------------------------
> 
> Message: 4
> Date: Thu, 13 Oct 2016 18:19:26 -0400
> From: Richard Cordovano <rco...@ba...>
> Subject: Re: [sleuthkit-users] Slow Ingest
> To: Ketil Froyn <ke...@fr...>
> Cc: sleuthkit-users <sle...@li...>
> Message-ID:
>         <CAP...@ma...>
> Content-Type: text/plain; charset="utf-8"
> 
> Ketil's hypothesis is very interesting, as is the remark about overheating.
> 
> Does the image have a lot of unallocated space?
> 
> Another possible test would be to reduce the number of file ingest threads
> to two and compare processing times.
> 
> On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote:
> 
> > If you have the time, opportunity and disk space, could you convert the
> > e01 to raw/dd and retry? I suspect that in some cases, libewf may be
> > spending a lot of time seeking in compressed e01 files. If you're
> > traversing a complex file system that requires a lot of seeking, my hunch
> > is that libewf, which is also single threaded as far as I understand, may
> > be your bottleneck. It'd be interesting to hear if simply changing to a raw
> > image makes much of a difference.
> >
> > https://github.com/libyal/libewf
> >
> > Regards, Ketil
> >
> > On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:
> >
> >> I would like to circle back on this.  I made the suggested modifications
> >> to reflect the physical cores of my system versus the hyper thread count.
> >> It honestly didn?t make much a difference.  When I reran the ingest it took
> >> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
> >> was at 3 Percent.
> >>
> >>
> >>
> >> One problem might be that my case was hosted on an OCZ350 Revo drive and
> >> it was waring of Overheat.   I circled back and deleted the case, then
> >> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> As I read the perfmon data the system is just ticking over, not really
> >> pushing hard.  Are  my expectations off?  The source drive image is 488 Gig
> >> in eo1 format.  How long should an ingest take for such a drive.  What are
> >> the system performance specs I should be working to improve to bring this
> >> down.  My goal with migrating to the new system was to get a case ingest
> >> down to under 24 hours.
> >>
> >>
> >>
> >>
> >>
> >> *From:* MATT PIERCE
> >> *Sent:* Wednesday, September 14, 2016 6:25 PM
> >> *To:* Richard Cordovano <rco...@ba...>
> >> *Subject:* Re: [sleuthkit-users] Slow Ingest
> >>
> >>
> >>
> >> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
> >> defined so I might skip that module.
> >>
> >>
> >>
> >> Sent Mobily
> >> ------------------------------
> >>
> >> *From:* Richard Cordovano <rco...@ba...>
> >> *Sent:* Sep 14, 2016 6:03 PM
> >> *To:* MATT PIERCE
> >> *Cc:* sle...@li...
> >> *Subject:* Re: [sleuthkit-users] Slow Ingest
> >>
> >>
> >>
> >> It looks like you are overtaxing your system with too many file ingest
> >> threads.The error notifications you are getting should not be considered to
> >> be typical and look to be symptomatic of a system struggling with out of
> >> memory errors. Based on your ingest progress snapshot, all file level
> >> analysis has been completed except keyword search, which is the most memory
> >> intensive aspect of Autopsy data source ingest.
> >>
> >>
> >>
> >> >> The system has dual Xeon x5550 2.66 quad core processors...I?ve set
> >> number of threads to 12 as suggested by the Options dialog.
> >>
> >>
> >>
> >> We generally recommend at most one file ingest thread per core, and the
> >> code for the Autopsy options dialog is even more conservative. For the
> >> system you describe, I would expect the Java library we use to detect the
> >> available processors (java.lang.Runtime.getRuntime().availableProcessors())
> >> would return eight (http://ark.intel.com/products
> >> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI),
> >> which would indeed give you a recommendation of only six file ingest
> >> threads! However, it appears that the Java lib is reporting the
> >> "hyperthreads" (sixteen) rather than the cores, which would indeed result
> >> in a recommendation of twelve threads. We will need to look into this
> >> further! In the mean time, the best advice I can give you is back way off
> >> on file ingest threads.
> >>
> >>
> >>
> >> Sincerely,
> >>
> >> Richard Cordovano
> >>
> >> Basis Technology
> >>
> >>
> >>
> >> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> >> wrote:
> >>
> >> I?m working a case and again have issues with performance using Autopsy.
> >> I have setup a dedicated server for running Autopsy.  In two days of ingest
> >> I?m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can
> >> spot where my bottle neck is?
> >>
> >>
> >>
> >> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> >> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy
> >> is loaded on a Raid0 15k SAS volume.
> >>
> >>
> >>
> >> Autopsy Load.
> >>
> >> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> >> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
> >> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on
> >> amd64; Cp1252; en_US (autopsy)
> >>
> >>
> >>
> >> The image is from a Windows 7 workstation.  FTKimager took the disk image
> >> in E01 format.  I have the NSRL known good hash database loaded.  I?ve set
> >> number of threads to 12 as suggested by the Options dialog.  I?m running
> >> the default ingest process with no 3rd party modules.
> >>
> >>
> >>
> >> Performance Diagnostics
> >>
> >>
> >>
> >> Ingest Progress Snapshot
> >>
> >>
> >>
> >> 1              IDLE                                       Wed Sep 14
> >> 01:10:44 CDT 2016   15:49:08.024       0
> >>
> >> 2              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> >> 2
> >>
> >> 3              IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.758         0
> >>
> >> 4              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> >> 2
> >>
> >> 5              IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.764         0
> >>
> >> 6              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> >> 2
> >>
> >> 7              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> >> 2
> >>
> >> 8              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> >> 2
> >>
> >> 9              IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.742         0
> >>
> >> 10           IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.812         0
> >>
> >> 11           Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> >> 2
> >>
> >> 12           IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.811         0
> >>
> >> 13           Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> >> 2
> >>
> >>
> >>
> >> 2              2016-08-31-1-1.E01          15:23:00               193034
> >> 2.0938940654524942       84           2              36
> >> 34           0
> >>
> >>
> >>
> >> Keyword Search               155:52:59.626 (36%)
> >>
> >> File Type Identification  139:55:17.562 (32%)
> >>
> >> Hash Lookup      48:49:44.445 (11%)
> >>
> >> Embedded File Extractor              46:01:20.323 (10%)
> >>
> >> Email Parser       28:37:58.461 (6%)
> >>
> >> Extension Mismatch Detector    4:51:51.763 (1%)
> >>
> >> Exif Parser           1:41:39.597 (0%)
> >>
> >> PhotoRec Carver              0:00:04.762 (0%)
> >>
> >> Interesting Files Identifier            0:00:00.105 (0%)
> >>
> >>
> >>
> >> I get a bunch of errors but that is pretty typical.
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------
> >> ------------------
> >>
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >>
> >>
> >>
> >> ------------------------------------------------------------
> >> ------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >>
> >>
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image008.jpg
> Type: image/jpeg
> Size: 38407 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image007.jpg
> Type: image/jpeg
> Size: 12346 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image002.png
> Type: image/png
> Size: 19142 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image009.jpg
> Type: image/jpeg
> Size: 93667 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image001.png
> Type: image/png
> Size: 13311 bytes
> Desc: not available
> 
> ------------------------------
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> 
> ------------------------------
> 
> _______________________________________________
> sleuthkit-users mailing list
> sle...@li...
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> 
> 
> End of sleuthkit-users Digest, Vol 124, Issue 6
> ***********************************************
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most 
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6

[Joachim M. <joa...@gm...>]

> It would be better to build static (or mostly static binaries) even through
they end up being very large.

Redistributing these might not be possible with certain licenses.


On Fri, Oct 14, 2016 at 12:19 AM, <
sle...@li...> wrote:

> Send sleuthkit-users mailing list submissions to
>         sle...@li...
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> or, via email, send a message with subject or body 'help' to
>         sle...@li...
>
> You can reach the person managing the list at
>         sle...@li...
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of sleuthkit-users digest..."
>
>
> Today's Topics:
>
>    1. Re: Windows Precompiled EXEs (Mark W. Jeanmougin)
>    2. Re: Windows Precompiled EXEs (Michael Cohen)
>    3. Re: Windows Precompiled EXEs (Brian Carrier)
>    4. Re: Slow Ingest (Richard Cordovano)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 11 Oct 2016 21:21:42 -0400
> From: "Mark W. Jeanmougin" <ma...@gm...>
> Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> To: lfc...@gm...
> Cc: "sle...@li... users"
>         <sle...@li...>
> Message-ID:
>         <CAMMgxRWnCpP7HB7WTq0uo=WwnnPBCTzs3QUCoHbX-9xkNM_eDg@mail.
> gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> I vote #1 for the same reasons as Luis.
>
> MJ
>
>
> On Tue, Oct 11, 2016 at 9:09 PM, Lu?s Filipe Nassif <lfc...@gm...>
> wrote:
> > I prefer option 1, it is much more portable, you can run tsk tools from a
> > thumb drive in the Field for example.
> >
> > Luis Nassif
> >
> >
> > Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...>
> > escreveu:
> >>
> >> As was mentioned in another thread, we?re looking at having The Sleuth
> Kit
> >> use a more modern Visual Studio compiler.  I need some feedback on how
> >> people are using the precompiled EXEs that we ship.
> >>
> >> One of the big changes is that executables now depend on a lot more
> >> official microsoft dlls (C runtime dlls).
> >>
> >> Historically, I?ve always tried to make it be so that you can open a ZIP
> >> file with the EXEs and run them with no prerequisites. We did this by
> >> shipping the 2 or 3 run time dlls in the same folder as the exes.
> >>
> >> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> >> two options:
> >>
> >> 1) We copy all 40+ dlls into the folder like we did before and it is now
> >> just a bit more work to find the exe tools in there among all of the
> dlls
> >> (they are all similarly named, which makes it somewhat easy).
> >>
> >> 2) We make the user run a Microsoft Redistributable Installer to install
> >> the needed dlls into the c:\windows folder if they are not already
> there. We
> >> can then ship a ZIP file and the user has to know to install the MS
> redist
> >> or we start shipping a TSK installer that also installs the MS redist.
> >>
> >>
> >> Opinions?    Do people care more about prerequisites or big folders?
> >>
> >>
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------
> ------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> >
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 11 Oct 2016 22:44:37 -0700
> From: Michael Cohen <scu...@gm...>
> Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> To: Brian Carrier <ca...@sl...>
> Cc: "sle...@li... users"
>         <sle...@li...>
> Message-ID:
>         <CAJ...@ma...ail.
> com>
> Content-Type: text/plain; charset=UTF-8
>
> It would be better to build static (or mostly static binaries) even
> through they end up being very large. Depending on the system you have
> to run on, SxS configurations can cause huge problems, even when the
> right dlls are in the same folder as the binary (the system might
> insist on using some other version which it decides is better).
>
> For pytsk we use the python build system to build the TSK library into
> one big python module with no external dependencies. Having to ship
> extra dlls that link at runtime to a python program is a pain and it
> is error prone. It might make more sense to have a shared library
> between all the tsk tools (libtsk) but I would opt to keep that as
> dependency free as possible even if it means building it with static
> linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> (Because this is still the standard for python 2.7) so it would be
> nice if the code remains portable.
>
> Thanks
> Michael.
>
> On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> > As was mentioned in another thread, we?re looking at having The Sleuth
> Kit
> > use a more modern Visual Studio compiler.  I need some feedback on how
> > people are using the precompiled EXEs that we ship.
> >
> > One of the big changes is that executables now depend on a lot more
> official
> > microsoft dlls (C runtime dlls).
> >
> > Historically, I?ve always tried to make it be so that you can open a ZIP
> > file with the EXEs and run them with no prerequisites. We did this by
> > shipping the 2 or 3 run time dlls in the same folder as the exes.
> >
> > With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> two
> > options:
> >
> > 1) We copy all 40+ dlls into the folder like we did before and it is now
> > just a bit more work to find the exe tools in there among all of the dlls
> > (they are all similarly named, which makes it somewhat easy).
> >
> > 2) We make the user run a Microsoft Redistributable Installer to install
> the
> > needed dlls into the c:\windows folder if they are not already there. We
> can
> > then ship a ZIP file and the user has to know to install the MS redist
> or we
> > start shipping a TSK installer that also installs the MS redist.
> >
> >
> > Opinions?    Do people care more about prerequisites or big folders?
> >
> >
> >
> >
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> >
>
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 12 Oct 2016 09:15:22 -0400
> From: Brian Carrier <ca...@sl...>
> Subject: Re: [sleuthkit-users] Windows Precompiled EXEs
> To: Michael Cohen <scu...@gm...>
> Cc: "sle...@li... users"
>         <sle...@li...>
> Message-ID: <31B...@sl...>
> Content-Type: text/plain; charset=utf-8
>
> Hi Michael,
>
> We thought about the entirely static approach, but we?d also then need
> libewf, libvhdi, zlib, etc. to be static and they don?t ship with a static
> library option.  We could obviously change their visual studio projects to
> make them, but we were first trying to do this as simply as possible.
>
> FWIW: There is now a Release_NoLibs build target for The Sleuth Kit that
> is static (i.e. no runtime dlls), but it doesn?t support E01 files.   We
> use that for our incident response Cyber Triage collection tool (?agent?).
>
> We?ll continue to keep it portable.  There are still a lot of gcc
> compilers out there that don?t like new fancy things.
>
> brian
>
>
>
> > On Oct 12, 2016, at 1:44 AM, Michael Cohen <scu...@gm...> wrote:
> >
> > It would be better to build static (or mostly static binaries) even
> > through they end up being very large. Depending on the system you have
> > to run on, SxS configurations can cause huge problems, even when the
> > right dlls are in the same folder as the binary (the system might
> > insist on using some other version which it decides is better).
> >
> > For pytsk we use the python build system to build the TSK library into
> > one big python module with no external dependencies. Having to ship
> > extra dlls that link at runtime to a python program is a pain and it
> > is error prone. It might make more sense to have a shared library
> > between all the tsk tools (libtsk) but I would opt to keep that as
> > dependency free as possible even if it means building it with static
> > linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> > (Because this is still the standard for python 2.7) so it would be
> > nice if the code remains portable.
> >
> > Thanks
> > Michael.
> >
> > On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> >> As was mentioned in another thread, we?re looking at having The Sleuth
> Kit
> >> use a more modern Visual Studio compiler.  I need some feedback on how
> >> people are using the precompiled EXEs that we ship.
> >>
> >> One of the big changes is that executables now depend on a lot more
> official
> >> microsoft dlls (C runtime dlls).
> >>
> >> Historically, I?ve always tried to make it be so that you can open a ZIP
> >> file with the EXEs and run them with no prerequisites. We did this by
> >> shipping the 2 or 3 run time dlls in the same folder as the exes.
> >>
> >> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> two
> >> options:
> >>
> >> 1) We copy all 40+ dlls into the folder like we did before and it is now
> >> just a bit more work to find the exe tools in there among all of the
> dlls
> >> (they are all similarly named, which makes it somewhat easy).
> >>
> >> 2) We make the user run a Microsoft Redistributable Installer to
> install the
> >> needed dlls into the c:\windows folder if they are not already there.
> We can
> >> then ship a ZIP file and the user has to know to install the MS redist
> or we
> >> start shipping a TSK installer that also installs the MS redist.
> >>
> >>
> >> Opinions?    Do people care more about prerequisites or big folders?
> >>
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------
> ------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >>
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 13 Oct 2016 18:19:26 -0400
> From: Richard Cordovano <rco...@ba...>
> Subject: Re: [sleuthkit-users] Slow Ingest
> To: Ketil Froyn <ke...@fr...>
> Cc: sleuthkit-users <sle...@li...>
> Message-ID:
>         <CAPuuEWQ9FU6zo+0t4iVdQ4SmFa8S_XjeARcSCPa2c3ttXP+VQA@mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Ketil's hypothesis is very interesting, as is the remark about overheating.
>
> Does the image have a lot of unallocated space?
>
> Another possible test would be to reduce the number of file ingest threads
> to two and compare processing times.
>
> On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote:
>
> > If you have the time, opportunity and disk space, could you convert the
> > e01 to raw/dd and retry? I suspect that in some cases, libewf may be
> > spending a lot of time seeking in compressed e01 files. If you're
> > traversing a complex file system that requires a lot of seeking, my hunch
> > is that libewf, which is also single threaded as far as I understand, may
> > be your bottleneck. It'd be interesting to hear if simply changing to a
> raw
> > image makes much of a difference.
> >
> > https://github.com/libyal/libewf
> >
> > Regards, Ketil
> >
> > On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:
> >
> >> I would like to circle back on this.  I made the suggested modifications
> >> to reflect the physical cores of my system versus the hyper thread
> count.
> >> It honestly didn?t make much a difference.  When I reran the ingest it
> took
> >> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
> >> was at 3 Percent.
> >>
> >>
> >>
> >> One problem might be that my case was hosted on an OCZ350 Revo drive and
> >> it was waring of Overheat.   I circled back and deleted the case, then
> >> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> As I read the perfmon data the system is just ticking over, not really
> >> pushing hard.  Are  my expectations off?  The source drive image is 488
> Gig
> >> in eo1 format.  How long should an ingest take for such a drive.  What
> are
> >> the system performance specs I should be working to improve to bring
> this
> >> down.  My goal with migrating to the new system was to get a case ingest
> >> down to under 24 hours.
> >>
> >>
> >>
> >>
> >>
> >> *From:* MATT PIERCE
> >> *Sent:* Wednesday, September 14, 2016 6:25 PM
> >> *To:* Richard Cordovano <rco...@ba...>
> >> *Subject:* Re: [sleuthkit-users] Slow Ingest
> >>
> >>
> >>
> >> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
> >> defined so I might skip that module.
> >>
> >>
> >>
> >> Sent Mobily
> >> ------------------------------
> >>
> >> *From:* Richard Cordovano <rco...@ba...>
> >> *Sent:* Sep 14, 2016 6:03 PM
> >> *To:* MATT PIERCE
> >> *Cc:* sle...@li...
> >> *Subject:* Re: [sleuthkit-users] Slow Ingest
> >>
> >>
> >>
> >> It looks like you are overtaxing your system with too many file ingest
> >> threads.The error notifications you are getting should not be
> considered to
> >> be typical and look to be symptomatic of a system struggling with out of
> >> memory errors. Based on your ingest progress snapshot, all file level
> >> analysis has been completed except keyword search, which is the most
> memory
> >> intensive aspect of Autopsy data source ingest.
> >>
> >>
> >>
> >> >> The system has dual Xeon x5550 2.66 quad core processors...I?ve set
> >> number of threads to 12 as suggested by the Options dialog.
> >>
> >>
> >>
> >> We generally recommend at most one file ingest thread per core, and the
> >> code for the Autopsy options dialog is even more conservative. For the
> >> system you describe, I would expect the Java library we use to detect
> the
> >> available processors (java.lang.Runtime.getRuntime(
> ).availableProcessors())
> >> would return eight (http://ark.intel.com/products
> >> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-
> GTs-Intel-QPI),
> >> which would indeed give you a recommendation of only six file ingest
> >> threads! However, it appears that the Java lib is reporting the
> >> "hyperthreads" (sixteen) rather than the cores, which would indeed
> result
> >> in a recommendation of twelve threads. We will need to look into this
> >> further! In the mean time, the best advice I can give you is back way
> off
> >> on file ingest threads.
> >>
> >>
> >>
> >> Sincerely,
> >>
> >> Richard Cordovano
> >>
> >> Basis Technology
> >>
> >>
> >>
> >> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> >> wrote:
> >>
> >> I?m working a case and again have issues with performance using Autopsy.
> >> I have setup a dedicated server for running Autopsy.  In two days of
> ingest
> >> I?m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone
> can
> >> spot where my bottle neck is?
> >>
> >>
> >>
> >> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> >> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.
> Autopsy
> >> is loaded on a Raid0 15k SAS volume.
> >>
> >>
> >>
> >> Autopsy Load.
> >>
> >> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> >> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
> >> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running
> on
> >> amd64; Cp1252; en_US (autopsy)
> >>
> >>
> >>
> >> The image is from a Windows 7 workstation.  FTKimager took the disk
> image
> >> in E01 format.  I have the NSRL known good hash database loaded.  I?ve
> set
> >> number of threads to 12 as suggested by the Options dialog.  I?m running
> >> the default ingest process with no 3rd party modules.
> >>
> >>
> >>
> >> Performance Diagnostics
> >>
> >>
> >>
> >> Ingest Progress Snapshot
> >>
> >>
> >>
> >> 1              IDLE                                       Wed Sep 14
> >> 01:10:44 CDT 2016   15:49:08.024       0
> >>
> >> 2              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> >> 2
> >>
> >> 3              IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.758         0
> >>
> >> 4              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> >> 2
> >>
> >> 5              IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.764         0
> >>
> >> 6              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> >> 2
> >>
> >> 7              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> >> 2
> >>
> >> 8              Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> >> 2
> >>
> >> 9              IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.742         0
> >>
> >> 10           IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.812         0
> >>
> >> 11           Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> >> 2
> >>
> >> 12           IDLE                                       Wed Sep 14
> >> 16:59:26 CDT 2016   0:00:25.811         0
> >>
> >> 13           Keyword Search               2016-08-31-1-1.E01
> >> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> >> 2
> >>
> >>
> >>
> >> 2              2016-08-31-1-1.E01          15:23:00               193034
> >> 2.0938940654524942       84           2              36
> >> 34           0
> >>
> >>
> >>
> >> Keyword Search               155:52:59.626 (36%)
> >>
> >> File Type Identification  139:55:17.562 (32%)
> >>
> >> Hash Lookup      48:49:44.445 (11%)
> >>
> >> Embedded File Extractor              46:01:20.323 (10%)
> >>
> >> Email Parser       28:37:58.461 (6%)
> >>
> >> Extension Mismatch Detector    4:51:51.763 (1%)
> >>
> >> Exif Parser           1:41:39.597 (0%)
> >>
> >> PhotoRec Carver              0:00:04.762 (0%)
> >>
> >> Interesting Files Identifier            0:00:00.105 (0%)
> >>
> >>
> >>
> >> I get a bunch of errors but that is pretty typical.
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------
> >> ------------------
> >>
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >>
> >>
> >>
> >> ------------------------------------------------------------
> >> ------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >>
> >>
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image008.jpg
> Type: image/jpeg
> Size: 38407 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image007.jpg
> Type: image/jpeg
> Size: 12346 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image002.png
> Type: image/png
> Size: 19142 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image009.jpg
> Type: image/jpeg
> Size: 93667 bytes
> Desc: not available
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image001.png
> Type: image/png
> Size: 13311 bytes
> Desc: not available
>
> ------------------------------
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>
> ------------------------------
>
> _______________________________________________
> sleuthkit-users mailing list
> sle...@li...
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>
>
> End of sleuthkit-users Digest, Vol 124, Issue 6
> ***********************************************
>

Re: [sleuthkit-users] Slow Ingest

[Richard C. <rco...@ba...>]

Ketil's hypothesis is very interesting, as is the remark about overheating.

Does the image have a lot of unallocated space?

Another possible test would be to reduce the number of file ingest threads
to two and compare processing times.

On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote:

> If you have the time, opportunity and disk space, could you convert the
> e01 to raw/dd and retry? I suspect that in some cases, libewf may be
> spending a lot of time seeking in compressed e01 files. If you're
> traversing a complex file system that requires a lot of seeking, my hunch
> is that libewf, which is also single threaded as far as I understand, may
> be your bottleneck. It'd be interesting to hear if simply changing to a raw
> image makes much of a difference.
>
> https://github.com/libyal/libewf
>
> Regards, Ketil
>
> On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:
>
>> I would like to circle back on this.  I made the suggested modifications
>> to reflect the physical cores of my system versus the hyper thread count.
>> It honestly didn’t make much a difference.  When I reran the ingest it took
>> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
>> was at 3 Percent.
>>
>>
>>
>> One problem might be that my case was hosted on an OCZ350 Revo drive and
>> it was waring of Overheat.   I circled back and deleted the case, then
>> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> As I read the perfmon data the system is just ticking over, not really
>> pushing hard.  Are  my expectations off?  The source drive image is 488 Gig
>> in eo1 format.  How long should an ingest take for such a drive.  What are
>> the system performance specs I should be working to improve to bring this
>> down.  My goal with migrating to the new system was to get a case ingest
>> down to under 24 hours.
>>
>>
>>
>>
>>
>> *From:* MATT PIERCE
>> *Sent:* Wednesday, September 14, 2016 6:25 PM
>> *To:* Richard Cordovano <rco...@ba...>
>> *Subject:* Re: [sleuthkit-users] Slow Ingest
>>
>>
>>
>> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
>> defined so I might skip that module.
>>
>>
>>
>> Sent Mobily
>> ------------------------------
>>
>> *From:* Richard Cordovano <rco...@ba...>
>> *Sent:* Sep 14, 2016 6:03 PM
>> *To:* MATT PIERCE
>> *Cc:* sle...@li...
>> *Subject:* Re: [sleuthkit-users] Slow Ingest
>>
>>
>>
>> It looks like you are overtaxing your system with too many file ingest
>> threads.The error notifications you are getting should not be considered to
>> be typical and look to be symptomatic of a system struggling with out of
>> memory errors. Based on your ingest progress snapshot, all file level
>> analysis has been completed except keyword search, which is the most memory
>> intensive aspect of Autopsy data source ingest.
>>
>>
>>
>> >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set
>> number of threads to 12 as suggested by the Options dialog.
>>
>>
>>
>> We generally recommend at most one file ingest thread per core, and the
>> code for the Autopsy options dialog is even more conservative. For the
>> system you describe, I would expect the Java library we use to detect the
>> available processors (java.lang.Runtime.getRuntime().availableProcessors())
>> would return eight (http://ark.intel.com/products
>> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI),
>> which would indeed give you a recommendation of only six file ingest
>> threads! However, it appears that the Java lib is reporting the
>> "hyperthreads" (sixteen) rather than the cores, which would indeed result
>> in a recommendation of twelve threads. We will need to look into this
>> further! In the mean time, the best advice I can give you is back way off
>> on file ingest threads.
>>
>>
>>
>> Sincerely,
>>
>> Richard Cordovano
>>
>> Basis Technology
>>
>>
>>
>> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
>> wrote:
>>
>> I’m working a case and again have issues with performance using Autopsy.
>> I have setup a dedicated server for running Autopsy.  In two days of ingest
>> I’m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can
>> spot where my bottle neck is?
>>
>>
>>
>> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
>> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy
>> is loaded on a Raid0 15k SAS volume.
>>
>>
>>
>> Autopsy Load.
>>
>> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
>> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
>> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on
>> amd64; Cp1252; en_US (autopsy)
>>
>>
>>
>> The image is from a Windows 7 workstation.  FTKimager took the disk image
>> in E01 format.  I have the NSRL known good hash database loaded.  I’ve set
>> number of threads to 12 as suggested by the Options dialog.  I’m running
>> the default ingest process with no 3rd party modules.
>>
>>
>>
>> Performance Diagnostics
>>
>>
>>
>> Ingest Progress Snapshot
>>
>>
>>
>> 1              IDLE                                       Wed Sep 14
>> 01:10:44 CDT 2016   15:49:08.024       0
>>
>> 2              Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
>> 2
>>
>> 3              IDLE                                       Wed Sep 14
>> 16:59:26 CDT 2016   0:00:25.758         0
>>
>> 4              Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
>> 2
>>
>> 5              IDLE                                       Wed Sep 14
>> 16:59:26 CDT 2016   0:00:25.764         0
>>
>> 6              Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
>> 2
>>
>> 7              Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
>> 2
>>
>> 8              Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
>> 2
>>
>> 9              IDLE                                       Wed Sep 14
>> 16:59:26 CDT 2016   0:00:25.742         0
>>
>> 10           IDLE                                       Wed Sep 14
>> 16:59:26 CDT 2016   0:00:25.812         0
>>
>> 11           Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
>> 2
>>
>> 12           IDLE                                       Wed Sep 14
>> 16:59:26 CDT 2016   0:00:25.811         0
>>
>> 13           Keyword Search               2016-08-31-1-1.E01
>> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
>> 2
>>
>>
>>
>> 2              2016-08-31-1-1.E01          15:23:00               193034
>> 2.0938940654524942       84           2              36
>> 34           0
>>
>>
>>
>> Keyword Search               155:52:59.626 (36%)
>>
>> File Type Identification  139:55:17.562 (32%)
>>
>> Hash Lookup      48:49:44.445 (11%)
>>
>> Embedded File Extractor              46:01:20.323 (10%)
>>
>> Email Parser       28:37:58.461 (6%)
>>
>> Extension Mismatch Detector    4:51:51.763 (1%)
>>
>> Exif Parser           1:41:39.597 (0%)
>>
>> PhotoRec Carver              0:00:04.762 (0%)
>>
>> Interesting Files Identifier            0:00:00.105 (0%)
>>
>>
>>
>> I get a bunch of errors but that is pretty typical.
>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>>
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>>
>>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Windows Precompiled EXEs

[Brian C. <ca...@sl...>]

Hi Michael,

We thought about the entirely static approach, but we’d also then need libewf, libvhdi, zlib, etc. to be static and they don’t ship with a static library option.  We could obviously change their visual studio projects to make them, but we were first trying to do this as simply as possible. 

FWIW: There is now a Release_NoLibs build target for The Sleuth Kit that is static (i.e. no runtime dlls), but it doesn’t support E01 files.   We use that for our incident response Cyber Triage collection tool (“agent”). 

We’ll continue to keep it portable.  There are still a lot of gcc compilers out there that don’t like new fancy things.

brian



> On Oct 12, 2016, at 1:44 AM, Michael Cohen <scu...@gm...> wrote:
> 
> It would be better to build static (or mostly static binaries) even
> through they end up being very large. Depending on the system you have
> to run on, SxS configurations can cause huge problems, even when the
> right dlls are in the same folder as the binary (the system might
> insist on using some other version which it decides is better).
> 
> For pytsk we use the python build system to build the TSK library into
> one big python module with no external dependencies. Having to ship
> extra dlls that link at runtime to a python program is a pain and it
> is error prone. It might make more sense to have a shared library
> between all the tsk tools (libtsk) but I would opt to keep that as
> dependency free as possible even if it means building it with static
> linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
> (Because this is still the standard for python 2.7) so it would be
> nice if the code remains portable.
> 
> Thanks
> Michael.
> 
> On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
>> As was mentioned in another thread, we’re looking at having The Sleuth Kit
>> use a more modern Visual Studio compiler.  I need some feedback on how
>> people are using the precompiled EXEs that we ship.
>> 
>> One of the big changes is that executables now depend on a lot more official
>> microsoft dlls (C runtime dlls).
>> 
>> Historically, I’ve always tried to make it be so that you can open a ZIP
>> file with the EXEs and run them with no prerequisites. We did this by
>> shipping the 2 or 3 run time dlls in the same folder as the exes.
>> 
>> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are two
>> options:
>> 
>> 1) We copy all 40+ dlls into the folder like we did before and it is now
>> just a bit more work to find the exe tools in there among all of the dlls
>> (they are all similarly named, which makes it somewhat easy).
>> 
>> 2) We make the user run a Microsoft Redistributable Installer to install the
>> needed dlls into the c:\windows folder if they are not already there. We can
>> then ship a ZIP file and the user has to know to install the MS redist or we
>> start shipping a TSK installer that also installs the MS redist.
>> 
>> 
>> Opinions?    Do people care more about prerequisites or big folders?
>> 
>> 
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most 
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Windows Precompiled EXEs

[Michael C. <scu...@gm...>]

It would be better to build static (or mostly static binaries) even
through they end up being very large. Depending on the system you have
to run on, SxS configurations can cause huge problems, even when the
right dlls are in the same folder as the binary (the system might
insist on using some other version which it decides is better).

For pytsk we use the python build system to build the TSK library into
one big python module with no external dependencies. Having to ship
extra dlls that link at runtime to a python program is a pain and it
is error prone. It might make more sense to have a shared library
between all the tsk tools (libtsk) but I would opt to keep that as
dependency free as possible even if it means building it with static
linked MSVCRT.  FWIW we have to continue using MSVC 2008 for pytsk
(Because this is still the standard for python 2.7) so it would be
nice if the code remains portable.

Thanks
Michael.

On 11/10/2016, Brian Carrier <ca...@sl...> wrote:
> As was mentioned in another thread, we’re looking at having The Sleuth Kit
> use a more modern Visual Studio compiler.  I need some feedback on how
> people are using the precompiled EXEs that we ship.
>
> One of the big changes is that executables now depend on a lot more official
> microsoft dlls (C runtime dlls).
>
> Historically, I’ve always tried to make it be so that you can open a ZIP
> file with the EXEs and run them with no prerequisites. We did this by
> shipping the 2 or 3 run time dlls in the same folder as the exes.
>
> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are two
> options:
>
> 1) We copy all 40+ dlls into the folder like we did before and it is now
> just a bit more work to find the exe tools in there among all of the dlls
> (they are all similarly named, which makes it somewhat easy).
>
> 2) We make the user run a Microsoft Redistributable Installer to install the
> needed dlls into the c:\windows folder if they are not already there. We can
> then ship a ZIP file and the user has to know to install the MS redist or we
> start shipping a TSK installer that also installs the MS redist.
>
>
> Opinions?    Do people care more about prerequisites or big folders?
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>

Re: [sleuthkit-users] Windows Precompiled EXEs

[Mark W. J. <ma...@gm...>]

I vote #1 for the same reasons as Luis.

MJ


On Tue, Oct 11, 2016 at 9:09 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> I prefer option 1, it is much more portable, you can run tsk tools from a
> thumb drive in the Field for example.
>
> Luis Nassif
>
>
> Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...>
> escreveu:
>>
>> As was mentioned in another thread, we’re looking at having The Sleuth Kit
>> use a more modern Visual Studio compiler.  I need some feedback on how
>> people are using the precompiled EXEs that we ship.
>>
>> One of the big changes is that executables now depend on a lot more
>> official microsoft dlls (C runtime dlls).
>>
>> Historically, I’ve always tried to make it be so that you can open a ZIP
>> file with the EXEs and run them with no prerequisites. We did this by
>> shipping the 2 or 3 run time dlls in the same folder as the exes.
>>
>> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
>> two options:
>>
>> 1) We copy all 40+ dlls into the folder like we did before and it is now
>> just a bit more work to find the exe tools in there among all of the dlls
>> (they are all similarly named, which makes it somewhat easy).
>>
>> 2) We make the user run a Microsoft Redistributable Installer to install
>> the needed dlls into the c:\windows folder if they are not already there. We
>> can then ship a ZIP file and the user has to know to install the MS redist
>> or we start shipping a TSK installer that also installs the MS redist.
>>
>>
>> Opinions?    Do people care more about prerequisites or big folders?
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>

Re: [sleuthkit-users] Windows Precompiled EXEs

[Luís F. N. <lfc...@gm...>]

I prefer option 1, it is much more portable, you can run tsk tools from a
thumb drive in the Field for example.

Luis Nassif

Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...>
escreveu:

> As was mentioned in another thread, we’re looking at having The Sleuth Kit
> use a more modern Visual Studio compiler.  I need some feedback on how
> people are using the precompiled EXEs that we ship.
>
> One of the big changes is that executables now depend on a lot more
> official microsoft dlls (C runtime dlls).
>
> Historically, I’ve always tried to make it be so that you can open a ZIP
> file with the EXEs and run them with no prerequisites. We did this by
> shipping the 2 or 3 run time dlls in the same folder as the exes.
>
> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> two options:
>
> 1) We copy all 40+ dlls into the folder like we did before and it is now
> just a bit more work to find the exe tools in there among all of the dlls
> (they are all similarly named, which makes it somewhat easy).
>
> 2) We make the user run a Microsoft Redistributable Installer to install
> the needed dlls into the c:\windows folder if they are not already there.
> We can then ship a ZIP file and the user has to know to install the MS
> redist or we start shipping a TSK installer that also installs the MS
> redist.
>
>
> Opinions?    Do people care more about prerequisites or big folders?
>
>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>

Re: [sleuthkit-users] Windows Precompiled EXEs

[D R. <da...@gm...>]

Didn't know sleuthkit had a windows exe.   I always have use Linux or I
just haven't paid attention in the past several years.   But to get to the
point,  it would be much easier to have everything compiled and shipped in
one place.

Dean

On Tue, Oct 11, 2016, 14:45 Danilo Marques <da...@gm...> wrote:

> Hi.
>
> I choose the option (1).
>
>
> 2016-10-11 17:59 GMT-03:00 Brian Carrier <ca...@sl...>:
>
> As was mentioned in another thread, we’re looking at having The Sleuth Kit
> use a more modern Visual Studio compiler.  I need some feedback on how
> people are using the precompiled EXEs that we ship.
>
> One of the big changes is that executables now depend on a lot more
> official microsoft dlls (C runtime dlls).
>
> Historically, I’ve always tried to make it be so that you can open a ZIP
> file with the EXEs and run them with no prerequisites. We did this by
> shipping the 2 or 3 run time dlls in the same folder as the exes.
>
> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> two options:
>
> 1) We copy all 40+ dlls into the folder like we did before and it is now
> just a bit more work to find the exe tools in there among all of the dlls
> (they are all similarly named, which makes it somewhat easy).
>
> 2) We make the user run a Microsoft Redistributable Installer to install
> the needed dlls into the c:\windows folder if they are not already there.
> We can then ship a ZIP file and the user has to know to install the MS
> redist or we start shipping a TSK installer that also installs the MS
> redist.
>
>
> Opinions?    Do people care more about prerequisites or big folders?
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
>
> --
> ---
> Danilo Caio Marcucci Marques
> Computer Forensic Investigator - ICCE-DGPTC/PCERJ/Brazil
> Linux user #419162
> [image: MyFreeCopyright.com Registered & Protected]
> <http://www.myfreecopyright.com/registered_mcn/CEM82_BNX21_KQM8A>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>

Re: [sleuthkit-users] Windows Precompiled EXEs

[Danilo M. <da...@gm...>]

Hi.

I choose the option (1).


2016-10-11 17:59 GMT-03:00 Brian Carrier <ca...@sl...>:

> As was mentioned in another thread, we’re looking at having The Sleuth Kit
> use a more modern Visual Studio compiler.  I need some feedback on how
> people are using the precompiled EXEs that we ship.
>
> One of the big changes is that executables now depend on a lot more
> official microsoft dlls (C runtime dlls).
>
> Historically, I’ve always tried to make it be so that you can open a ZIP
> file with the EXEs and run them with no prerequisites. We did this by
> shipping the 2 or 3 run time dlls in the same folder as the exes.
>
> With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are
> two options:
>
> 1) We copy all 40+ dlls into the folder like we did before and it is now
> just a bit more work to find the exe tools in there among all of the dlls
> (they are all similarly named, which makes it somewhat easy).
>
> 2) We make the user run a Microsoft Redistributable Installer to install
> the needed dlls into the c:\windows folder if they are not already there.
> We can then ship a ZIP file and the user has to know to install the MS
> redist or we start shipping a TSK installer that also installs the MS
> redist.
>
>
> Opinions?    Do people care more about prerequisites or big folders?
>
>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>



-- 
---
Danilo Caio Marcucci Marques
Computer Forensic Investigator - ICCE-DGPTC/PCERJ/Brazil
Linux user #419162
[image: MyFreeCopyright.com Registered & Protected]
<http://www.myfreecopyright.com/registered_mcn/CEM82_BNX21_KQM8A>

[sleuthkit-users] Windows Precompiled EXEs

[Brian C. <ca...@sl...>]

As was mentioned in another thread, we’re looking at having The Sleuth Kit use a more modern Visual Studio compiler.  I need some feedback on how people are using the precompiled EXEs that we ship.

One of the big changes is that executables now depend on a lot more official microsoft dlls (C runtime dlls). 

Historically, I’ve always tried to make it be so that you can open a ZIP file with the EXEs and run them with no prerequisites. We did this by shipping the 2 or 3 run time dlls in the same folder as the exes. 

With Visual Studio 2015, there are over 40 dlls, not 2!   So, there are two options:

1) We copy all 40+ dlls into the folder like we did before and it is now just a bit more work to find the exe tools in there among all of the dlls (they are all similarly named, which makes it somewhat easy).

2) We make the user run a Microsoft Redistributable Installer to install the needed dlls into the c:\windows folder if they are not already there. We can then ship a ZIP file and the user has to know to install the MS redist or we start shipping a TSK installer that also installs the MS redist.


Opinions?    Do people care more about prerequisites or big folders?

Re: [sleuthkit-users] [sleuthkit-developers] Upgrade to Visual Studio 2015

[Roberto M. <rma...@ch...>]

I recently built 64bit versions of libvmdk, libvhdi, zlib, libewf, etc under VS2015 community edition.

Here are some of the steps that had to happen on individual projects:


- Change Platform Toolset from Windows7.1 SDK to Visual Studio 2015 (v140)

- Both libvmdk/libvhdi use config_winapi.h for setup, expect WINVER to be set (rarely is), I used _WIN32_WINNT_WIN7

- Switching Configuration Manager from Win32 to x64 (copy+minor tweaks)

- Running PowerShell scripts to pull down dependencies

- libewf has some post-build copies that use ENV VARS for dependency paths

- libtsk uses ENV VARS for dependencies that need to be setup


While it has now all built and it seems to be working, there are several images (mostly E01/EWF type) that come back with:

"Cannot determine file system type".

The files (E01) where generated with EnCase and FTK.


I'd love to contribute back my changes for this build to happen but I may have screwed something that is causing this problem.

Has anybody encountered this before?


Expected output from MMLS:

DOS Partition Table

Offset Sector: 0

Units are in 512-byte sectors


      Slot      Start        End          Length       Description

000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)

001:  -------   0000000000   0000002047   0000002048   Unallocated

002:  000:000   0000002048   0000206847   0000204800   NTFS / exFAT (0x07)

003:  000:001   0000206848   0156246015   0156039168   NTFS / exFAT (0x07)

004:  -------   0156246016   0156249999   0000003984   Unallocated


Output seen on build's MMLS:

tsk_img_open: Type: 0   NumImg: 1  Img1: \Temp\AEX-pretest.e01
ewf_open: found 0 segment files via libewf_glob
Not an EWF file
Error opening vmdk file
Error checking file signature for vhd file
tsk_img_findFiles: \Temp\AEX-pretest.e01 found
tsk_img_findFiles: 1 total segments found
raw_open: segment: 0  size: 19031471881  max offset: 19031471881  path: \Temp\AEX-pretest.e01
dos_load_prim: Table Sector: 0
raw_read: byte offset: 0 len: 65536
raw_read: found in image 0 relative offset: 0 len: 65536
raw_read_segment: opening file into slot 0: \Temp\AEX-pretest.e01
File is not a DOS partition (invalid primary magic) (Sector: 0)bsd_load_table: Table Sector: 1
gpt_load_table: Sector: 0
gpt_open: Trying other sector sizes
gpt_open: Trying sector size: 512
gpt_load_table: Sector: 0
gpt_open: Trying sector size: 1024
gpt_load_table: Sector: 0
gpt_open: Trying sector size: 2048
gpt_load_table: Sector: 0
gpt_open: Trying sector size: 4096
gpt_load_table: Sector: 0
gpt_open: Trying sector size: 8192
gpt_load_table: Sector: 0
sun_load_table: Trying sector: 0
sun_load_table: Trying sector: 1
mac_load_table: Sector: 1
mac_load: Missing initial magic value
mac_open: Trying 4096-byte sector size instead of 512-byte
mac_load_table: Sector: 1
mac_load: Missing initial magic value
Cannot determine partition type




Roberto Machorro   Software Developer, Child Rescue Coalition


Phone: (561) 226-9690<tel:%28561%29%20226-9690>
Email: rma...@ch...<mailto:rma...@ch...>
Address: 4530 Conference Way S
Boca Raton, FL  33431




________________________________
From: Richard Cordovano <rco...@ba...>
Sent: Tuesday, October 11, 2016 9:04 AM
To: Lloyd
Cc: sleuthkit-users
Subject: Re: [sleuthkit-users] [sleuthkit-developers] Upgrade to Visual Studio 2015

We have an engineer here at Basis currently working on completing an update of the Windows platform build of the SleuthKit for Microsoft Visual Studio 2015. We will also be updating the companion 64-bit versions of libewf, libvmdk, and libvhdi to build with VS 2015.

On Sun, Oct 9, 2016 at 10:24 AM, Lloyd <llo...@gm...<mailto:llo...@gm...>> wrote:
It would be great if sleuthkit is supported on vs2015 also.

On Tue, Oct 4, 2016 at 1:21 PM, Alessandro De Vito <ale...@gm...<mailto:ale...@gm...>> wrote:
Hi Brian,
Is there any update about this?
I would like to use tsk but I can not find VS10 on the web. On this link:
http://www.microsoft.com/express/vc/

only 2015 version is available.

Thanks
Alessandro

2016-06-09 18:44 GMT+02:00 Michael Cohen <scu...@gm...<mailto:scu...@gm...>>:

Hi Brian,
Just as an FYI, pytsk uses VS 9.0 since that is the only supported compiler for python 2.7. But we do not use any of the project files since python has its own build system.

https://wiki.python.org/moin/WindowsCompilers

It would be good to keep the code itself compilable under this old version which does not support later c standards.

Thanks
Michael.

On 9 Jun 2016 08:46, "Brian Carrier" <ca...@sl...<mailto:ca...@sl...>> wrote:
If you compile TSK with Visual Studio, you have to have use 2010, which has become dated and is a pain to get 64-bit builds out of. We're thinking about moving to VS 2015 (still the free version).  Does this impact anyone?   Anyone building for source on Windows and want it to remain in 2010?

brian


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
sleuthkit-developers mailing list
sle...@li...<mailto:sle...@li...>
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org

Re: [sleuthkit-users] [sleuthkit-developers] Upgrade to Visual Studio 2015

[Richard C. <rco...@ba...>]

We have an engineer here at Basis currently working on completing an update
of the Windows platform build of the SleuthKit for Microsoft Visual Studio
2015. We will also be updating the companion 64-bit versions of libewf,
libvmdk, and libvhdi to build with VS 2015.

On Sun, Oct 9, 2016 at 10:24 AM, Lloyd <llo...@gm...> wrote:

> It would be great if sleuthkit is supported on vs2015 also.
>
> On Tue, Oct 4, 2016 at 1:21 PM, Alessandro De Vito <
> ale...@gm...> wrote:
>
>> Hi Brian,
>> Is there any update about this?
>> I would like to use tsk but I can not find VS10 on the web. On this link:
>> http://www.microsoft.com/express/vc/
>> only 2015 version is available.
>>
>> Thanks
>> Alessandro
>>
>> 2016-06-09 18:44 GMT+02:00 Michael Cohen <scu...@gm...>:
>>
>>> Hi Brian,
>>> Just as an FYI, pytsk uses VS 9.0 since that is the only supported
>>> compiler for python 2.7. But we do not use any of the project files since
>>> python has its own build system.
>>>
>>> https://wiki.python.org/moin/WindowsCompilers
>>>
>>> It would be good to keep the code itself compilable under this old
>>> version which does not support later c standards.
>>>
>>> Thanks
>>> Michael.
>>>
>>> On 9 Jun 2016 08:46, "Brian Carrier" <ca...@sl...> wrote:
>>>
>>>> If you compile TSK with Visual Studio, you have to have use 2010, which
>>>> has become dated and is a pain to get 64-bit builds out of. We’re thinking
>>>> about moving to VS 2015 (still the free version).  Does this impact
>>>> anyone?   Anyone building for source on Windows and want it to remain in
>>>> 2010?
>>>>
>>>> brian
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>>>> traffic
>>>> patterns at an interface-level. Reveals which users, apps, and
>>>> protocols are
>>>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>>>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>>>> planning reports. https://ad.doubleclick.net/ddm
>>>> /clk/305295220;132659582;e
>>>> _______________________________________________
>>>> sleuthkit-developers mailing list
>>>> sle...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
>>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>>> traffic
>>> patterns at an interface-level. Reveals which users, apps, and protocols
>>> are
>>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>>> planning reports. https://ad.doubleclick.net/ddm
>>> /clk/305295220;132659582;e
>>> _______________________________________________
>>> sleuthkit-users mailing list
>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> http://www.sleuthkit.org
>>>
>>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>>
>>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] [sleuthkit-developers] Upgrade to Visual Studio 2015

[Lloyd <llo...@gm...>]

It would be great if sleuthkit is supported on vs2015 also.

On Tue, Oct 4, 2016 at 1:21 PM, Alessandro De Vito <
ale...@gm...> wrote:

> Hi Brian,
> Is there any update about this?
> I would like to use tsk but I can not find VS10 on the web. On this link:
> http://www.microsoft.com/express/vc/
> only 2015 version is available.
>
> Thanks
> Alessandro
>
> 2016-06-09 18:44 GMT+02:00 Michael Cohen <scu...@gm...>:
>
>> Hi Brian,
>> Just as an FYI, pytsk uses VS 9.0 since that is the only supported
>> compiler for python 2.7. But we do not use any of the project files since
>> python has its own build system.
>>
>> https://wiki.python.org/moin/WindowsCompilers
>>
>> It would be good to keep the code itself compilable under this old
>> version which does not support later c standards.
>>
>> Thanks
>> Michael.
>>
>> On 9 Jun 2016 08:46, "Brian Carrier" <ca...@sl...> wrote:
>>
>>> If you compile TSK with Visual Studio, you have to have use 2010, which
>>> has become dated and is a pain to get 64-bit builds out of. We’re thinking
>>> about moving to VS 2015 (still the free version).  Does this impact
>>> anyone?   Anyone building for source on Windows and want it to remain in
>>> 2010?
>>>
>>> brian
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>>> traffic
>>> patterns at an interface-level. Reveals which users, apps, and protocols
>>> are
>>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>>> planning reports. https://ad.doubleclick.net/ddm
>>> /clk/305295220;132659582;e
>>> _______________________________________________
>>> sleuthkit-developers mailing list
>>> sle...@li...
>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
>>>
>>
>> ------------------------------------------------------------
>> ------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>> traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols
>> are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>> planning reports. https://ad.doubleclick.net/ddm
>> /clk/305295220;132659582;e
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>>
>>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Slow Ingest

[Ketil F. <ke...@fr...>]

If you have the time, opportunity and disk space, could you convert the e01
to raw/dd and retry? I suspect that in some cases, libewf may be spending a
lot of time seeking in compressed e01 files. If you're traversing a complex
file system that requires a lot of seeking, my hunch is that libewf, which
is also single threaded as far as I understand, may be your bottleneck.
It'd be interesting to hear if simply changing to a raw image makes much of
a difference.

https://github.com/libyal/libewf

Regards, Ketil

On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote:

> I would like to circle back on this.  I made the suggested modifications
> to reflect the physical cores of my system versus the hyper thread count.
> It honestly didn’t make much a difference.  When I reran the ingest it took
> 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search
> was at 3 Percent.
>
>
>
> One problem might be that my case was hosted on an OCZ350 Revo drive and
> it was waring of Overheat.   I circled back and deleted the case, then
> reran it with the case on 15k SAS raid0 and the source file on 10k SAS.
>
>
>
>
>
>
>
>
>
> As I read the perfmon data the system is just ticking over, not really
> pushing hard.  Are  my expectations off?  The source drive image is 488 Gig
> in eo1 format.  How long should an ingest take for such a drive.  What are
> the system performance specs I should be working to improve to bring this
> down.  My goal with migrating to the new system was to get a case ingest
> down to under 24 hours.
>
>
>
>
>
> *From:* MATT PIERCE
> *Sent:* Wednesday, September 14, 2016 6:25 PM
> *To:* Richard Cordovano <rco...@ba...>
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> Thank you.  I'm restarting the run tomorrow.  I don't have any keywords
> defined so I might skip that module.
>
>
>
> Sent Mobily
> ------------------------------
>
> *From:* Richard Cordovano <rco...@ba...>
> *Sent:* Sep 14, 2016 6:03 PM
> *To:* MATT PIERCE
> *Cc:* sle...@li...
> *Subject:* Re: [sleuthkit-users] Slow Ingest
>
>
>
> It looks like you are overtaxing your system with too many file ingest
> threads.The error notifications you are getting should not be considered to
> be typical and look to be symptomatic of a system struggling with out of
> memory errors. Based on your ingest progress snapshot, all file level
> analysis has been completed except keyword search, which is the most memory
> intensive aspect of Autopsy data source ingest.
>
>
>
> >> The system has dual Xeon x5550 2.66 quad core processors...I’ve set
> number of threads to 12 as suggested by the Options dialog.
>
>
>
> We generally recommend at most one file ingest thread per core, and the
> code for the Autopsy options dialog is even more conservative. For the
> system you describe, I would expect the Java library we use to detect the
> available processors (java.lang.Runtime.getRuntime().availableProcessors())
> would return eight (http://ark.intel.com/products/37106/Intel-Xeon-
> Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed
> give you a recommendation of only six file ingest threads! However, it
> appears that the Java lib is reporting the "hyperthreads" (sixteen) rather
> than the cores, which would indeed result in a recommendation of twelve
> threads. We will need to look into this further! In the mean time, the best
> advice I can give you is back way off on file ingest threads.
>
>
>
> Sincerely,
>
> Richard Cordovano
>
> Basis Technology
>
>
>
> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...>
> wrote:
>
> I’m working a case and again have issues with performance using Autopsy.
> I have setup a dedicated server for running Autopsy.  In two days of ingest
> I’m at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can
> spot where my bottle neck is?
>
>
>
> The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.
> Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy
> is loaded on a Raid0 15k SAS volume.
>
>
>
> Autopsy Load.
>
> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0
> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit
> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on
> amd64; Cp1252; en_US (autopsy)
>
>
>
> The image is from a Windows 7 workstation.  FTKimager took the disk image
> in E01 format.  I have the NSRL known good hash database loaded.  I’ve set
> number of threads to 12 as suggested by the Options dialog.  I’m running
> the default ingest process with no 3rd party modules.
>
>
>
> Performance Diagnostics
>
>
>
> Ingest Progress Snapshot
>
>
>
> 1              IDLE                                       Wed Sep 14
> 01:10:44 CDT 2016   15:49:08.024       0
>
> 2              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 3              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.758         0
>
> 4              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798
> 2
>
> 5              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.764         0
>
> 6              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759
> 2
>
> 7              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
> 8              Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132
> 2
>
> 9              IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.742         0
>
> 10           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.812         0
>
> 11           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120
> 2
>
> 12           IDLE                                       Wed Sep 14
> 16:59:26 CDT 2016   0:00:25.811         0
>
> 13           Keyword Search               2016-08-31-1-1.E01
> image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155
> 2
>
>
>
> 2              2016-08-31-1-1.E01          15:23:00               193034
> 2.0938940654524942       84           2              36
> 34           0
>
>
>
> Keyword Search               155:52:59.626 (36%)
>
> File Type Identification  139:55:17.562 (32%)
>
> Hash Lookup      48:49:44.445 (11%)
>
> Embedded File Extractor              46:01:20.323 (10%)
>
> Email Parser       28:37:58.461 (6%)
>
> Extension Mismatch Detector    4:51:51.763 (1%)
>
> Exif Parser           1:41:39.597 (0%)
>
> PhotoRec Carver              0:00:04.762 (0%)
>
> Interesting Files Identifier            0:00:00.105 (0%)
>
>
>
> I get a bunch of errors but that is pretty typical.
>
>
>
>
> ------------------------------------------------------------
> ------------------
>
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Slow Ingest

[MATT P. <mat...@ad...>]

I would like to circle back on this.  I made the suggested modifications to reflect the physical cores of my system versus the hyper thread count.  It honestly didn't make much a difference.  When I reran the ingest it took 96 hours to complete the Analyzing Files stage.  Periodic Keyword Search was at 3 Percent.

One problem might be that my case was hosted on an OCZ350 Revo drive and it was waring of Overheat.   I circled back and deleted the case, then reran it with the case on 15k SAS raid0 and the source file on 10k SAS.

[cid:image007.jpg@01D21FCA.6B9D4710]

[cid:image008.jpg@01D21FCA.6B9D4710]

[cid:image009.jpg@01D21FCA.6B9D4710]

As I read the perfmon data the system is just ticking over, not really pushing hard.  Are  my expectations off?  The source drive image is 488 Gig in eo1 format.  How long should an ingest take for such a drive.  What are the system performance specs I should be working to improve to bring this down.  My goal with migrating to the new system was to get a case ingest down to under 24 hours.


From: MATT PIERCE
Sent: Wednesday, September 14, 2016 6:25 PM
To: Richard Cordovano <rco...@ba...>
Subject: Re: [sleuthkit-users] Slow Ingest

Thank you.  I'm restarting the run tomorrow.  I don't have any keywords defined so I might skip that module.

Sent Mobily
________________________________
From: Richard Cordovano <rco...@ba...<mailto:rco...@ba...>>
Sent: Sep 14, 2016 6:03 PM
To: MATT PIERCE
Cc: sle...@li...<mailto:sle...@li...>
Subject: Re: [sleuthkit-users] Slow Ingest

It looks like you are overtaxing your system with too many file ingest threads.The error notifications you are getting should not be considered to be typical and look to be symptomatic of a system struggling with out of memory errors. Based on your ingest progress snapshot, all file level analysis has been completed except keyword search, which is the most memory intensive aspect of Autopsy data source ingest.

>> The system has dual Xeon x5550 2.66 quad core processors...I've set number of threads to 12 as suggested by the Options dialog.

We generally recommend at most one file ingest thread per core, and the code for the Autopsy options dialog is even more conservative. For the system you describe, I would expect the Java library we use to detect the available processors (java.lang.Runtime.getRuntime().availableProcessors()) would return eight (http://ark.intel.com/products/37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), which would indeed give you a recommendation of only six file ingest threads! However, it appears that the Java lib is reporting the "hyperthreads" (sixteen) rather than the cores, which would indeed result in a recommendation of twelve threads. We will need to look into this further! In the mean time, the best advice I can give you is back way off on file ingest threads.

Sincerely,
Richard Cordovano
Basis Technology

On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...<mailto:mat...@ad...>> wrote:
I'm working a case and again have issues with performance using Autopsy.  I have setup a dedicated server for running Autopsy.  In two days of ingest I'm at 45%.  In 8 hours it has only progressed 7%. I was hoping someone can spot where my bottle neck is?

The system has dual Xeon x5550 2.66 quad core processors.  24 GB RAM.  Windows 2012 R2 x64.   The case drive is an OCZ Revo 350 PCIe SSD.  Autopsy is loaded on a Raid0 15k SAS volume.

Autopsy Load.
Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on amd64; Cp1252; en_US (autopsy)

The image is from a Windows 7 workstation.  FTKimager took the disk image in E01 format.  I have the NSRL known good hash database loaded.  I've set number of threads to 12 as suggested by the Options dialog.  I'm running the default ingest process with no 3rd party modules.

Performance Diagnostics
[cid:image001.png@01D21FC8.72F0CE30]

Ingest Progress Snapshot

1              IDLE                                       Wed Sep 14 01:10:44 CDT 2016   15:49:08.024       0
2              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759                2
3              IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.758         0
4              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.798                2
5              IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.764         0
6              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:25.759                2
7              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155                2
8              Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.132                2
9              IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.742         0
10           IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.812         0
11           Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.120                2
12           IDLE                                       Wed Sep 14 16:59:26 CDT 2016   0:00:25.811         0
13           Keyword Search               2016-08-31-1-1.E01          image1.emf        Wed Sep 14 16:59:26 CDT 2016   0:00:26.155                2

2              2016-08-31-1-1.E01          15:23:00               193034  2.0938940654524942       84           2              36           34           0

Keyword Search               155:52:59.626 (36%)
File Type Identification  139:55:17.562 (32%)
Hash Lookup      48:49:44.445 (11%)
Embedded File Extractor              46:01:20.323 (10%)
Email Parser       28:37:58.461 (6%)
Extension Mismatch Detector    4:51:51.763 (1%)
Exif Parser           1:41:39.597 (0%)
PhotoRec Carver              0:00:04.762 (0%)
Interesting Files Identifier            0:00:00.105 (0%)

I get a bunch of errors but that is pretty typical.

[cid:image002.png@01D21FC8.72F0CE30]

------------------------------------------------------------------------------

_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org

Re: [sleuthkit-users] [sleuthkit-developers] Upgrade to Visual Studio 2015

[Alessandro De V. <ale...@gm...>]

Hi Brian,
Is there any update about this?
I would like to use tsk but I can not find VS10 on the web. On this link:
http://www.microsoft.com/express/vc/
only 2015 version is available.

Thanks
Alessandro

2016-06-09 18:44 GMT+02:00 Michael Cohen <scu...@gm...>:

> Hi Brian,
> Just as an FYI, pytsk uses VS 9.0 since that is the only supported
> compiler for python 2.7. But we do not use any of the project files since
> python has its own build system.
>
> https://wiki.python.org/moin/WindowsCompilers
>
> It would be good to keep the code itself compilable under this old version
> which does not support later c standards.
>
> Thanks
> Michael.
>
> On 9 Jun 2016 08:46, "Brian Carrier" <ca...@sl...> wrote:
>
>> If you compile TSK with Visual Studio, you have to have use 2010, which
>> has become dated and is a pain to get 64-bit builds out of. We’re thinking
>> about moving to VS 2015 (still the free version).  Does this impact
>> anyone?   Anyone building for source on Windows and want it to remain in
>> 2010?
>>
>> brian
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>> traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols
>> are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>> planning reports. https://ad.doubleclick.net/
>> ddm/clk/305295220;132659582;e
>> _______________________________________________
>> sleuthkit-developers mailing list
>> sle...@li...
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
>>
>
> ------------------------------------------------------------
> ------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and
> traffic
> patterns at an interface-level. Reveals which users, apps, and protocols
> are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

[sleuthkit-users] Does the sleuth kit support Encrypted EnCase image?

[Ramesh S. <ram...@gm...>]

Hi all,

Does The Sleuth Kit/ Autospy support encrypted EnCase image file?

Also does it support *.Ex01* image file?
I only have *.E01* image file which it seems to support but don't have any
*.Ex01* image file hence can't verified whether it is supported or not.

Thanks in advance

-- 
ramesh

[sleuthkit-users] ISO9660 is not processed properly. Only a part of files is enumerated by TSK

[Paulo W. <pau...@gm...>]

Hi,

Having problem with TSK e ISO9660. Found it was already reported. Any news?

Rgds,

Paulo.

[sleuthkit-users] OSDFCon Early Registration Ends Soon

[Brian C. <ca...@sl...>]

OSDFCon is a month away and the early bird rate and discounted hotel rates are ending soon.  Register now to get your packed day of open source digital forensics talks at the lowest price. 

OSDFCon will be held October 26 in Herndon, VA. There are hands-on workshops the day before and a 1-day Autopsy course the day after.  Come see talks from Facebook, Google, Mozilla, and Basis Technology on topics about memory forensics, incident response, cloud computing, and more. The full program is online. 

    http://www.osdfcon.org/2016-event/2016-agenda/

Early bird registration ends October 1.  The conference is free for government employees. 

    http://www.osdfcon.org/2016-event/2016-registration/

The discounted hotel rate ends October 3.

    http://www.osdfcon.org/2016-event/2016-venue/

Autopsy challenge submissions are due October 17 and details are available here. We have received over 12 submissions, so the the prizes have doubled!

[sleuthkit-users] Open ab files with Autopsy?

[Joyce N. <joy...@gm...>]

Is it possible to open Android Backup Files with Autopsy?

[sleuthkit-users] Sleuthkit implementation inconsistent with API documentation

[noxdafox <nox...@gm...>]

Greetings,


I am integrating TSK within a tool to add its forensics capabilities.

I am writing an API capable of retrieving the inode of a given block in 
a similar manner as for the command:

ifind -d 1232 win.dd

I am using the ifind logic as a reference implementation and I stumbled 
over some inconsistency with the API documentation.


According to the documentation, the tsk_fs_attr_walk function calls a 
callback function passing the address (TSK_DADDR_T) of the data block 
being analysed.

http://www.sleuthkit.org/sleuthkit/docs/api-docs/4.2/tsk__fs_8h.html#a90bf87426f9ff3b898a6bdc61a8eff76

The documentation states that the User must check the flags 
(<http://www.sleuthkit.org/sleuthkit/docs/api-docs/4.2/tsk__fs_8h.html#a1e6bf157f5d258191bf5d8ae31ee7148>TSK_FS_BLOCK_FLAG_ENUM) 
and verify the block is raw (TSK_FS_BLOCK_FLAG_RAW).

Yet the implementation seems to check only whether the block is sparse 
or not (TSK_FS_BLOCK_FLAG_SPARSE):

https://github.com/sleuthkit/sleuthkit/blob/develop/tsk/fs/ifind_lib.c#L479

I guess that check could be avoided by passing the no sparse 
(TSK_FS_FILE_WALK_FLAG_NOSPARSE) flag to the tsk_fs_attr_walk function.


After further reading, it seems to me that enforcing the blocks to be 
raw would skip compressed ones which would be valid as well.


What do you thing is the correct approach? Shall I just skip sparse 
blocks or shall I report only raw blocks?

Thank you.

[sleuthkit-users] tsk_loaddb error -1073741571

[Aristeu G. A. Jr <ari...@gm...>]

Hi folks,

I'm trying to use tsk_loaddb (sleuthkit-4.3) on an E01 image of an external
disk with HFS+.
After 1 hour trying to decode the image, the tsk_loaddb returns the error
-1073741571.

Does someone knows something about this error?
The image seems OK, because I can open it and navigate on it using FTK
imager.

Thanks and regards,
--
Aristeu Jr