Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6
Brought to you by:
carrier
Menu
▾
▴
Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6
|
From: Brian C. <ca...@sl...> - 2016-10-14 13:38:27
|
Excellent point. Given that libewf et al. are GPL, they cannot be statically linked in. > On Oct 14, 2016, at 1:01 AM, Joachim Metz <joa...@gm...> wrote: > > > It would be better to build static (or mostly static binaries) even through they end up being very large. > > Redistributing these might not be possible with certain licenses. > > > On Fri, Oct 14, 2016 at 12:19 AM, <sle...@li...> wrote: > Send sleuthkit-users mailing list submissions to > sle...@li... > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > or, via email, send a message with subject or body 'help' to > sle...@li... > > You can reach the person managing the list at > sle...@li... > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of sleuthkit-users digest..." > > > Today's Topics: > > 1. Re: Windows Precompiled EXEs (Mark W. Jeanmougin) > 2. Re: Windows Precompiled EXEs (Michael Cohen) > 3. Re: Windows Precompiled EXEs (Brian Carrier) > 4. Re: Slow Ingest (Richard Cordovano) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 11 Oct 2016 21:21:42 -0400 > From: "Mark W. Jeanmougin" <ma...@gm...> > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs > To: lfc...@gm... > Cc: "sle...@li... users" > <sle...@li...> > Message-ID: > <CAMMgxRWnCpP7HB7WTq0uo=Wwn...@ma...> > Content-Type: text/plain; charset=UTF-8 > > I vote #1 for the same reasons as Luis. > > MJ > > > On Tue, Oct 11, 2016 at 9:09 PM, Lu?s Filipe Nassif <lfc...@gm...> wrote: > > I prefer option 1, it is much more portable, you can run tsk tools from a > > thumb drive in the Field for example. > > > > Luis Nassif > > > > > > Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...> > > escreveu: > >> > >> As was mentioned in another thread, we?re looking at having The Sleuth Kit > >> use a more modern Visual Studio compiler. I need some feedback on how > >> people are using the precompiled EXEs that we ship. > >> > >> One of the big changes is that executables now depend on a lot more > >> official microsoft dlls (C runtime dlls). > >> > >> Historically, I?ve always tried to make it be so that you can open a ZIP > >> file with the EXEs and run them with no prerequisites. We did this by > >> shipping the 2 or 3 run time dlls in the same folder as the exes. > >> > >> With Visual Studio 2015, there are over 40 dlls, not 2! So, there are > >> two options: > >> > >> 1) We copy all 40+ dlls into the folder like we did before and it is now > >> just a bit more work to find the exe tools in there among all of the dlls > >> (they are all similarly named, which makes it somewhat easy). > >> > >> 2) We make the user run a Microsoft Redistributable Installer to install > >> the needed dlls into the c:\windows folder if they are not already there. We > >> can then ship a ZIP file and the user has to know to install the MS redist > >> or we start shipping a TSK installer that also installs the MS redist. > >> > >> > >> Opinions? Do people care more about prerequisites or big folders? > >> > >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> sleuthkit-users mailing list > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > >> http://www.sleuthkit.org > > > > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > sleuthkit-users mailing list > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > http://www.sleuthkit.org > > > > > > ------------------------------ > > Message: 2 > Date: Tue, 11 Oct 2016 22:44:37 -0700 > From: Michael Cohen <scu...@gm...> > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs > To: Brian Carrier <ca...@sl...> > Cc: "sle...@li... users" > <sle...@li...> > Message-ID: > <CAJ...@ma...> > Content-Type: text/plain; charset=UTF-8 > > It would be better to build static (or mostly static binaries) even > through they end up being very large. Depending on the system you have > to run on, SxS configurations can cause huge problems, even when the > right dlls are in the same folder as the binary (the system might > insist on using some other version which it decides is better). > > For pytsk we use the python build system to build the TSK library into > one big python module with no external dependencies. Having to ship > extra dlls that link at runtime to a python program is a pain and it > is error prone. It might make more sense to have a shared library > between all the tsk tools (libtsk) but I would opt to keep that as > dependency free as possible even if it means building it with static > linked MSVCRT. FWIW we have to continue using MSVC 2008 for pytsk > (Because this is still the standard for python 2.7) so it would be > nice if the code remains portable. > > Thanks > Michael. > > On 11/10/2016, Brian Carrier <ca...@sl...> wrote: > > As was mentioned in another thread, we?re looking at having The Sleuth Kit > > use a more modern Visual Studio compiler. I need some feedback on how > > people are using the precompiled EXEs that we ship. > > > > One of the big changes is that executables now depend on a lot more official > > microsoft dlls (C runtime dlls). > > > > Historically, I?ve always tried to make it be so that you can open a ZIP > > file with the EXEs and run them with no prerequisites. We did this by > > shipping the 2 or 3 run time dlls in the same folder as the exes. > > > > With Visual Studio 2015, there are over 40 dlls, not 2! So, there are two > > options: > > > > 1) We copy all 40+ dlls into the folder like we did before and it is now > > just a bit more work to find the exe tools in there among all of the dlls > > (they are all similarly named, which makes it somewhat easy). > > > > 2) We make the user run a Microsoft Redistributable Installer to install the > > needed dlls into the c:\windows folder if they are not already there. We can > > then ship a ZIP file and the user has to know to install the MS redist or we > > start shipping a TSK installer that also installs the MS redist. > > > > > > Opinions? Do people care more about prerequisites or big folders? > > > > > > > > > > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > sleuthkit-users mailing list > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > http://www.sleuthkit.org > > > > > > ------------------------------ > > Message: 3 > Date: Wed, 12 Oct 2016 09:15:22 -0400 > From: Brian Carrier <ca...@sl...> > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs > To: Michael Cohen <scu...@gm...> > Cc: "sle...@li... users" > <sle...@li...> > Message-ID: <31B...@sl...> > Content-Type: text/plain; charset=utf-8 > > Hi Michael, > > We thought about the entirely static approach, but we?d also then need libewf, libvhdi, zlib, etc. to be static and they don?t ship with a static library option. We could obviously change their visual studio projects to make them, but we were first trying to do this as simply as possible. > > FWIW: There is now a Release_NoLibs build target for The Sleuth Kit that is static (i.e. no runtime dlls), but it doesn?t support E01 files. We use that for our incident response Cyber Triage collection tool (?agent?). > > We?ll continue to keep it portable. There are still a lot of gcc compilers out there that don?t like new fancy things. > > brian > > > > > On Oct 12, 2016, at 1:44 AM, Michael Cohen <scu...@gm...> wrote: > > > > It would be better to build static (or mostly static binaries) even > > through they end up being very large. Depending on the system you have > > to run on, SxS configurations can cause huge problems, even when the > > right dlls are in the same folder as the binary (the system might > > insist on using some other version which it decides is better). > > > > For pytsk we use the python build system to build the TSK library into > > one big python module with no external dependencies. Having to ship > > extra dlls that link at runtime to a python program is a pain and it > > is error prone. It might make more sense to have a shared library > > between all the tsk tools (libtsk) but I would opt to keep that as > > dependency free as possible even if it means building it with static > > linked MSVCRT. FWIW we have to continue using MSVC 2008 for pytsk > > (Because this is still the standard for python 2.7) so it would be > > nice if the code remains portable. > > > > Thanks > > Michael. > > > > On 11/10/2016, Brian Carrier <ca...@sl...> wrote: > >> As was mentioned in another thread, we?re looking at having The Sleuth Kit > >> use a more modern Visual Studio compiler. I need some feedback on how > >> people are using the precompiled EXEs that we ship. > >> > >> One of the big changes is that executables now depend on a lot more official > >> microsoft dlls (C runtime dlls). > >> > >> Historically, I?ve always tried to make it be so that you can open a ZIP > >> file with the EXEs and run them with no prerequisites. We did this by > >> shipping the 2 or 3 run time dlls in the same folder as the exes. > >> > >> With Visual Studio 2015, there are over 40 dlls, not 2! So, there are two > >> options: > >> > >> 1) We copy all 40+ dlls into the folder like we did before and it is now > >> just a bit more work to find the exe tools in there among all of the dlls > >> (they are all similarly named, which makes it somewhat easy). > >> > >> 2) We make the user run a Microsoft Redistributable Installer to install the > >> needed dlls into the c:\windows folder if they are not already there. We can > >> then ship a ZIP file and the user has to know to install the MS redist or we > >> start shipping a TSK installer that also installs the MS redist. > >> > >> > >> Opinions? Do people care more about prerequisites or big folders? > >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> sleuthkit-users mailing list > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > >> http://www.sleuthkit.org > >> > > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > sleuthkit-users mailing list > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > http://www.sleuthkit.org > > > > > ------------------------------ > > Message: 4 > Date: Thu, 13 Oct 2016 18:19:26 -0400 > From: Richard Cordovano <rco...@ba...> > Subject: Re: [sleuthkit-users] Slow Ingest > To: Ketil Froyn <ke...@fr...> > Cc: sleuthkit-users <sle...@li...> > Message-ID: > <CAP...@ma...> > Content-Type: text/plain; charset="utf-8" > > Ketil's hypothesis is very interesting, as is the remark about overheating. > > Does the image have a lot of unallocated space? > > Another possible test would be to reduce the number of file ingest threads > to two and compare processing times. > > On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote: > > > If you have the time, opportunity and disk space, could you convert the > > e01 to raw/dd and retry? I suspect that in some cases, libewf may be > > spending a lot of time seeking in compressed e01 files. If you're > > traversing a complex file system that requires a lot of seeking, my hunch > > is that libewf, which is also single threaded as far as I understand, may > > be your bottleneck. It'd be interesting to hear if simply changing to a raw > > image makes much of a difference. > > > > https://github.com/libyal/libewf > > > > Regards, Ketil > > > > On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote: > > > >> I would like to circle back on this. I made the suggested modifications > >> to reflect the physical cores of my system versus the hyper thread count. > >> It honestly didn?t make much a difference. When I reran the ingest it took > >> 96 hours to complete the Analyzing Files stage. Periodic Keyword Search > >> was at 3 Percent. > >> > >> > >> > >> One problem might be that my case was hosted on an OCZ350 Revo drive and > >> it was waring of Overheat. I circled back and deleted the case, then > >> reran it with the case on 15k SAS raid0 and the source file on 10k SAS. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> As I read the perfmon data the system is just ticking over, not really > >> pushing hard. Are my expectations off? The source drive image is 488 Gig > >> in eo1 format. How long should an ingest take for such a drive. What are > >> the system performance specs I should be working to improve to bring this > >> down. My goal with migrating to the new system was to get a case ingest > >> down to under 24 hours. > >> > >> > >> > >> > >> > >> *From:* MATT PIERCE > >> *Sent:* Wednesday, September 14, 2016 6:25 PM > >> *To:* Richard Cordovano <rco...@ba...> > >> *Subject:* Re: [sleuthkit-users] Slow Ingest > >> > >> > >> > >> Thank you. I'm restarting the run tomorrow. I don't have any keywords > >> defined so I might skip that module. > >> > >> > >> > >> Sent Mobily > >> ------------------------------ > >> > >> *From:* Richard Cordovano <rco...@ba...> > >> *Sent:* Sep 14, 2016 6:03 PM > >> *To:* MATT PIERCE > >> *Cc:* sle...@li... > >> *Subject:* Re: [sleuthkit-users] Slow Ingest > >> > >> > >> > >> It looks like you are overtaxing your system with too many file ingest > >> threads.The error notifications you are getting should not be considered to > >> be typical and look to be symptomatic of a system struggling with out of > >> memory errors. Based on your ingest progress snapshot, all file level > >> analysis has been completed except keyword search, which is the most memory > >> intensive aspect of Autopsy data source ingest. > >> > >> > >> > >> >> The system has dual Xeon x5550 2.66 quad core processors...I?ve set > >> number of threads to 12 as suggested by the Options dialog. > >> > >> > >> > >> We generally recommend at most one file ingest thread per core, and the > >> code for the Autopsy options dialog is even more conservative. For the > >> system you describe, I would expect the Java library we use to detect the > >> available processors (java.lang.Runtime.getRuntime().availableProcessors()) > >> would return eight (http://ark.intel.com/products > >> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40-GTs-Intel-QPI), > >> which would indeed give you a recommendation of only six file ingest > >> threads! However, it appears that the Java lib is reporting the > >> "hyperthreads" (sixteen) rather than the cores, which would indeed result > >> in a recommendation of twelve threads. We will need to look into this > >> further! In the mean time, the best advice I can give you is back way off > >> on file ingest threads. > >> > >> > >> > >> Sincerely, > >> > >> Richard Cordovano > >> > >> Basis Technology > >> > >> > >> > >> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...> > >> wrote: > >> > >> I?m working a case and again have issues with performance using Autopsy. > >> I have setup a dedicated server for running Autopsy. In two days of ingest > >> I?m at 45%. In 8 hours it has only progressed 7%. I was hoping someone can > >> spot where my bottle neck is? > >> > >> > >> > >> The system has dual Xeon x5550 2.66 quad core processors. 24 GB RAM. > >> Windows 2012 R2 x64. The case drive is an OCZ Revo 350 PCIe SSD. Autopsy > >> is loaded on a Raid0 15k SAS volume. > >> > >> > >> > >> Autopsy Load. > >> > >> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 > >> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) 64-Bit > >> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 running on > >> amd64; Cp1252; en_US (autopsy) > >> > >> > >> > >> The image is from a Windows 7 workstation. FTKimager took the disk image > >> in E01 format. I have the NSRL known good hash database loaded. I?ve set > >> number of threads to 12 as suggested by the Options dialog. I?m running > >> the default ingest process with no 3rd party modules. > >> > >> > >> > >> Performance Diagnostics > >> > >> > >> > >> Ingest Progress Snapshot > >> > >> > >> > >> 1 IDLE Wed Sep 14 > >> 01:10:44 CDT 2016 15:49:08.024 0 > >> > >> 2 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 > >> 2 > >> > >> 3 IDLE Wed Sep 14 > >> 16:59:26 CDT 2016 0:00:25.758 0 > >> > >> 4 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.798 > >> 2 > >> > >> 5 IDLE Wed Sep 14 > >> 16:59:26 CDT 2016 0:00:25.764 0 > >> > >> 6 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 > >> 2 > >> > >> 7 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 > >> 2 > >> > >> 8 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.132 > >> 2 > >> > >> 9 IDLE Wed Sep 14 > >> 16:59:26 CDT 2016 0:00:25.742 0 > >> > >> 10 IDLE Wed Sep 14 > >> 16:59:26 CDT 2016 0:00:25.812 0 > >> > >> 11 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.120 > >> 2 > >> > >> 12 IDLE Wed Sep 14 > >> 16:59:26 CDT 2016 0:00:25.811 0 > >> > >> 13 Keyword Search 2016-08-31-1-1.E01 > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 > >> 2 > >> > >> > >> > >> 2 2016-08-31-1-1.E01 15:23:00 193034 > >> 2.0938940654524942 84 2 36 > >> 34 0 > >> > >> > >> > >> Keyword Search 155:52:59.626 (36%) > >> > >> File Type Identification 139:55:17.562 (32%) > >> > >> Hash Lookup 48:49:44.445 (11%) > >> > >> Embedded File Extractor 46:01:20.323 (10%) > >> > >> Email Parser 28:37:58.461 (6%) > >> > >> Extension Mismatch Detector 4:51:51.763 (1%) > >> > >> Exif Parser 1:41:39.597 (0%) > >> > >> PhotoRec Carver 0:00:04.762 (0%) > >> > >> Interesting Files Identifier 0:00:00.105 (0%) > >> > >> > >> > >> I get a bunch of errors but that is pretty typical. > >> > >> > >> > >> > >> ------------------------------------------------------------ > >> ------------------ > >> > >> _______________________________________________ > >> sleuthkit-users mailing list > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > >> http://www.sleuthkit.org > >> > >> > >> > >> ------------------------------------------------------------ > >> ------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> sleuthkit-users mailing list > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > >> http://www.sleuthkit.org > >> > >> > > ------------------------------------------------------------ > > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > sleuthkit-users mailing list > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > http://www.sleuthkit.org > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image008.jpg > Type: image/jpeg > Size: 38407 bytes > Desc: not available > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image007.jpg > Type: image/jpeg > Size: 12346 bytes > Desc: not available > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image002.png > Type: image/png > Size: 19142 bytes > Desc: not available > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image009.jpg > Type: image/jpeg > Size: 93667 bytes > Desc: not available > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image001.png > Type: image/png > Size: 13311 bytes > Desc: not available > > ------------------------------ > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > ------------------------------ > > _______________________________________________ > sleuthkit-users mailing list > sle...@li... > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > > End of sleuthkit-users Digest, Vol 124, Issue 6 > *********************************************** > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot_______________________________________________ > sleuthkit-users mailing list > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > http://www.sleuthkit.org |
