Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6
Brought to you by:
carrier
Menu
▾
▴
Re: [sleuthkit-users] sleuthkit-users Digest, Vol 124, Issue 6
|
From: Joachim M. <joa...@gm...> - 2016-10-14 14:31:05
|
They are LGPL v3 not GPL, they can be linked in statically, there is no issue if the rest of the code is LGPL compatible https://en.wikipedia.org/wiki/License_compatibility However there more licenses to consider such as that of the VS C runtime. On Fri, Oct 14, 2016 at 3:38 PM, Brian Carrier <ca...@sl...> wrote: > Excellent point. Given that libewf et al. are GPL, they cannot be > statically linked in. > > > > On Oct 14, 2016, at 1:01 AM, Joachim Metz <joa...@gm...> > wrote: > > > > > It would be better to build static (or mostly static binaries) even > through they end up being very large. > > > > Redistributing these might not be possible with certain licenses. > > > > > > On Fri, Oct 14, 2016 at 12:19 AM, <sleuthkit-users-request@ > lists.sourceforge.net> wrote: > > Send sleuthkit-users mailing list submissions to > > sle...@li... > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > or, via email, send a message with subject or body 'help' to > > sle...@li... > > > > You can reach the person managing the list at > > sle...@li... > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of sleuthkit-users digest..." > > > > > > Today's Topics: > > > > 1. Re: Windows Precompiled EXEs (Mark W. Jeanmougin) > > 2. Re: Windows Precompiled EXEs (Michael Cohen) > > 3. Re: Windows Precompiled EXEs (Brian Carrier) > > 4. Re: Slow Ingest (Richard Cordovano) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Tue, 11 Oct 2016 21:21:42 -0400 > > From: "Mark W. Jeanmougin" <ma...@gm...> > > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs > > To: lfc...@gm... > > Cc: "sle...@li... users" > > <sle...@li...> > > Message-ID: > > <CAMMgxRWnCpP7HB7WTq0uo=WwnnPBCTzs3QUCoHbX-9xkNM_eDg@mail. > gmail.com> > > Content-Type: text/plain; charset=UTF-8 > > > > I vote #1 for the same reasons as Luis. > > > > MJ > > > > > > On Tue, Oct 11, 2016 at 9:09 PM, Lu?s Filipe Nassif <lfc...@gm...> > wrote: > > > I prefer option 1, it is much more portable, you can run tsk tools > from a > > > thumb drive in the Field for example. > > > > > > Luis Nassif > > > > > > > > > Em 11 de out de 2016 18:04, "Brian Carrier" <ca...@sl...> > > > escreveu: > > >> > > >> As was mentioned in another thread, we?re looking at having The > Sleuth Kit > > >> use a more modern Visual Studio compiler. I need some feedback on how > > >> people are using the precompiled EXEs that we ship. > > >> > > >> One of the big changes is that executables now depend on a lot more > > >> official microsoft dlls (C runtime dlls). > > >> > > >> Historically, I?ve always tried to make it be so that you can open a > ZIP > > >> file with the EXEs and run them with no prerequisites. We did this by > > >> shipping the 2 or 3 run time dlls in the same folder as the exes. > > >> > > >> With Visual Studio 2015, there are over 40 dlls, not 2! So, there > are > > >> two options: > > >> > > >> 1) We copy all 40+ dlls into the folder like we did before and it is > now > > >> just a bit more work to find the exe tools in there among all of the > dlls > > >> (they are all similarly named, which makes it somewhat easy). > > >> > > >> 2) We make the user run a Microsoft Redistributable Installer to > install > > >> the needed dlls into the c:\windows folder if they are not already > there. We > > >> can then ship a ZIP file and the user has to know to install the MS > redist > > >> or we start shipping a TSK installer that also installs the MS redist. > > >> > > >> > > >> Opinions? Do people care more about prerequisites or big folders? > > >> > > >> > > >> > > >> > > >> > > >> > > >> ------------------------------------------------------------ > ------------------ > > >> Check out the vibrant tech community on one of the world's most > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > >> _______________________________________________ > > >> sleuthkit-users mailing list > > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > >> http://www.sleuthkit.org > > > > > > > > > ------------------------------------------------------------ > ------------------ > > > Check out the vibrant tech community on one of the world's most > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > sleuthkit-users mailing list > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > > http://www.sleuthkit.org > > > > > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Tue, 11 Oct 2016 22:44:37 -0700 > > From: Michael Cohen <scu...@gm...> > > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs > > To: Brian Carrier <ca...@sl...> > > Cc: "sle...@li... users" > > <sle...@li...> > > Message-ID: > > <CAJ...@ma...ail. > com> > > Content-Type: text/plain; charset=UTF-8 > > > > It would be better to build static (or mostly static binaries) even > > through they end up being very large. Depending on the system you have > > to run on, SxS configurations can cause huge problems, even when the > > right dlls are in the same folder as the binary (the system might > > insist on using some other version which it decides is better). > > > > For pytsk we use the python build system to build the TSK library into > > one big python module with no external dependencies. Having to ship > > extra dlls that link at runtime to a python program is a pain and it > > is error prone. It might make more sense to have a shared library > > between all the tsk tools (libtsk) but I would opt to keep that as > > dependency free as possible even if it means building it with static > > linked MSVCRT. FWIW we have to continue using MSVC 2008 for pytsk > > (Because this is still the standard for python 2.7) so it would be > > nice if the code remains portable. > > > > Thanks > > Michael. > > > > On 11/10/2016, Brian Carrier <ca...@sl...> wrote: > > > As was mentioned in another thread, we?re looking at having The Sleuth > Kit > > > use a more modern Visual Studio compiler. I need some feedback on how > > > people are using the precompiled EXEs that we ship. > > > > > > One of the big changes is that executables now depend on a lot more > official > > > microsoft dlls (C runtime dlls). > > > > > > Historically, I?ve always tried to make it be so that you can open a > ZIP > > > file with the EXEs and run them with no prerequisites. We did this by > > > shipping the 2 or 3 run time dlls in the same folder as the exes. > > > > > > With Visual Studio 2015, there are over 40 dlls, not 2! So, there > are two > > > options: > > > > > > 1) We copy all 40+ dlls into the folder like we did before and it is > now > > > just a bit more work to find the exe tools in there among all of the > dlls > > > (they are all similarly named, which makes it somewhat easy). > > > > > > 2) We make the user run a Microsoft Redistributable Installer to > install the > > > needed dlls into the c:\windows folder if they are not already there. > We can > > > then ship a ZIP file and the user has to know to install the MS redist > or we > > > start shipping a TSK installer that also installs the MS redist. > > > > > > > > > Opinions? Do people care more about prerequisites or big folders? > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > ------------------ > > > Check out the vibrant tech community on one of the world's most > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > sleuthkit-users mailing list > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > > http://www.sleuthkit.org > > > > > > > > > > > ------------------------------ > > > > Message: 3 > > Date: Wed, 12 Oct 2016 09:15:22 -0400 > > From: Brian Carrier <ca...@sl...> > > Subject: Re: [sleuthkit-users] Windows Precompiled EXEs > > To: Michael Cohen <scu...@gm...> > > Cc: "sle...@li... users" > > <sle...@li...> > > Message-ID: <31B...@sl...> > > Content-Type: text/plain; charset=utf-8 > > > > Hi Michael, > > > > We thought about the entirely static approach, but we?d also then need > libewf, libvhdi, zlib, etc. to be static and they don?t ship with a static > library option. We could obviously change their visual studio projects to > make them, but we were first trying to do this as simply as possible. > > > > FWIW: There is now a Release_NoLibs build target for The Sleuth Kit that > is static (i.e. no runtime dlls), but it doesn?t support E01 files. We > use that for our incident response Cyber Triage collection tool (?agent?). > > > > We?ll continue to keep it portable. There are still a lot of gcc > compilers out there that don?t like new fancy things. > > > > brian > > > > > > > > > On Oct 12, 2016, at 1:44 AM, Michael Cohen <scu...@gm...> wrote: > > > > > > It would be better to build static (or mostly static binaries) even > > > through they end up being very large. Depending on the system you have > > > to run on, SxS configurations can cause huge problems, even when the > > > right dlls are in the same folder as the binary (the system might > > > insist on using some other version which it decides is better). > > > > > > For pytsk we use the python build system to build the TSK library into > > > one big python module with no external dependencies. Having to ship > > > extra dlls that link at runtime to a python program is a pain and it > > > is error prone. It might make more sense to have a shared library > > > between all the tsk tools (libtsk) but I would opt to keep that as > > > dependency free as possible even if it means building it with static > > > linked MSVCRT. FWIW we have to continue using MSVC 2008 for pytsk > > > (Because this is still the standard for python 2.7) so it would be > > > nice if the code remains portable. > > > > > > Thanks > > > Michael. > > > > > > On 11/10/2016, Brian Carrier <ca...@sl...> wrote: > > >> As was mentioned in another thread, we?re looking at having The > Sleuth Kit > > >> use a more modern Visual Studio compiler. I need some feedback on how > > >> people are using the precompiled EXEs that we ship. > > >> > > >> One of the big changes is that executables now depend on a lot more > official > > >> microsoft dlls (C runtime dlls). > > >> > > >> Historically, I?ve always tried to make it be so that you can open a > ZIP > > >> file with the EXEs and run them with no prerequisites. We did this by > > >> shipping the 2 or 3 run time dlls in the same folder as the exes. > > >> > > >> With Visual Studio 2015, there are over 40 dlls, not 2! So, there > are two > > >> options: > > >> > > >> 1) We copy all 40+ dlls into the folder like we did before and it is > now > > >> just a bit more work to find the exe tools in there among all of the > dlls > > >> (they are all similarly named, which makes it somewhat easy). > > >> > > >> 2) We make the user run a Microsoft Redistributable Installer to > install the > > >> needed dlls into the c:\windows folder if they are not already there. > We can > > >> then ship a ZIP file and the user has to know to install the MS > redist or we > > >> start shipping a TSK installer that also installs the MS redist. > > >> > > >> > > >> Opinions? Do people care more about prerequisites or big folders? > > >> > > >> > > >> > > >> > > >> > > >> ------------------------------------------------------------ > ------------------ > > >> Check out the vibrant tech community on one of the world's most > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > >> _______________________________________________ > > >> sleuthkit-users mailing list > > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > >> http://www.sleuthkit.org > > >> > > > > > > ------------------------------------------------------------ > ------------------ > > > Check out the vibrant tech community on one of the world's most > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > sleuthkit-users mailing list > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > > http://www.sleuthkit.org > > > > > > > > > > ------------------------------ > > > > Message: 4 > > Date: Thu, 13 Oct 2016 18:19:26 -0400 > > From: Richard Cordovano <rco...@ba...> > > Subject: Re: [sleuthkit-users] Slow Ingest > > To: Ketil Froyn <ke...@fr...> > > Cc: sleuthkit-users <sle...@li...> > > Message-ID: > > <CAPuuEWQ9FU6zo+0t4iVdQ4SmFa8S_XjeARcSCPa2c3ttXP+VQA@mail. > gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Ketil's hypothesis is very interesting, as is the remark about > overheating. > > > > Does the image have a lot of unallocated space? > > > > Another possible test would be to reduce the number of file ingest > threads > > to two and compare processing times. > > > > On Thu, Oct 6, 2016 at 6:16 PM, Ketil Froyn <ke...@fr...> wrote: > > > > > If you have the time, opportunity and disk space, could you convert the > > > e01 to raw/dd and retry? I suspect that in some cases, libewf may be > > > spending a lot of time seeking in compressed e01 files. If you're > > > traversing a complex file system that requires a lot of seeking, my > hunch > > > is that libewf, which is also single threaded as far as I understand, > may > > > be your bottleneck. It'd be interesting to hear if simply changing to > a raw > > > image makes much of a difference. > > > > > > https://github.com/libyal/libewf > > > > > > Regards, Ketil > > > > > > On 6 Oct 2016 19:28, "MATT PIERCE" <mat...@ad...> wrote: > > > > > >> I would like to circle back on this. I made the suggested > modifications > > >> to reflect the physical cores of my system versus the hyper thread > count. > > >> It honestly didn?t make much a difference. When I reran the ingest > it took > > >> 96 hours to complete the Analyzing Files stage. Periodic Keyword > Search > > >> was at 3 Percent. > > >> > > >> > > >> > > >> One problem might be that my case was hosted on an OCZ350 Revo drive > and > > >> it was waring of Overheat. I circled back and deleted the case, then > > >> reran it with the case on 15k SAS raid0 and the source file on 10k > SAS. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> As I read the perfmon data the system is just ticking over, not really > > >> pushing hard. Are my expectations off? The source drive image is > 488 Gig > > >> in eo1 format. How long should an ingest take for such a drive. > What are > > >> the system performance specs I should be working to improve to bring > this > > >> down. My goal with migrating to the new system was to get a case > ingest > > >> down to under 24 hours. > > >> > > >> > > >> > > >> > > >> > > >> *From:* MATT PIERCE > > >> *Sent:* Wednesday, September 14, 2016 6:25 PM > > >> *To:* Richard Cordovano <rco...@ba...> > > >> *Subject:* Re: [sleuthkit-users] Slow Ingest > > >> > > >> > > >> > > >> Thank you. I'm restarting the run tomorrow. I don't have any > keywords > > >> defined so I might skip that module. > > >> > > >> > > >> > > >> Sent Mobily > > >> ------------------------------ > > >> > > >> *From:* Richard Cordovano <rco...@ba...> > > >> *Sent:* Sep 14, 2016 6:03 PM > > >> *To:* MATT PIERCE > > >> *Cc:* sle...@li... > > >> *Subject:* Re: [sleuthkit-users] Slow Ingest > > >> > > >> > > >> > > >> It looks like you are overtaxing your system with too many file ingest > > >> threads.The error notifications you are getting should not be > considered to > > >> be typical and look to be symptomatic of a system struggling with out > of > > >> memory errors. Based on your ingest progress snapshot, all file level > > >> analysis has been completed except keyword search, which is the most > memory > > >> intensive aspect of Autopsy data source ingest. > > >> > > >> > > >> > > >> >> The system has dual Xeon x5550 2.66 quad core processors...I?ve set > > >> number of threads to 12 as suggested by the Options dialog. > > >> > > >> > > >> > > >> We generally recommend at most one file ingest thread per core, and > the > > >> code for the Autopsy options dialog is even more conservative. For the > > >> system you describe, I would expect the Java library we use to detect > the > > >> available processors (java.lang.Runtime.getRuntime( > ).availableProcessors()) > > >> would return eight (http://ark.intel.com/products > > >> /37106/Intel-Xeon-Processor-X5550-8M-Cache-2_66-GHz-6_40- > GTs-Intel-QPI), > > >> which would indeed give you a recommendation of only six file ingest > > >> threads! However, it appears that the Java lib is reporting the > > >> "hyperthreads" (sixteen) rather than the cores, which would indeed > result > > >> in a recommendation of twelve threads. We will need to look into this > > >> further! In the mean time, the best advice I can give you is back way > off > > >> on file ingest threads. > > >> > > >> > > >> > > >> Sincerely, > > >> > > >> Richard Cordovano > > >> > > >> Basis Technology > > >> > > >> > > >> > > >> On Wed, Sep 14, 2016 at 6:04 PM, MATT PIERCE <mat...@ad...> > > >> wrote: > > >> > > >> I?m working a case and again have issues with performance using > Autopsy. > > >> I have setup a dedicated server for running Autopsy. In two days of > ingest > > >> I?m at 45%. In 8 hours it has only progressed 7%. I was hoping > someone can > > >> spot where my bottle neck is? > > >> > > >> > > >> > > >> The system has dual Xeon x5550 2.66 quad core processors. 24 GB RAM. > > >> Windows 2012 R2 x64. The case drive is an OCZ Revo 350 PCIe SSD. > Autopsy > > >> is loaded on a Raid0 15k SAS volume. > > >> > > >> > > >> > > >> Autopsy Load. > > >> > > >> Product Version: Autopsy 4.1.1 (RELEASE) Sleuth Kit Version: 4.2.0 > > >> Netbeans RCP Build: 201510222201 Java: 1.8.0_92; Java HotSpot(TM) > 64-Bit > > >> Server VM 25.92-b14 System: Windows Server 2012 R2 version 6.3 > running on > > >> amd64; Cp1252; en_US (autopsy) > > >> > > >> > > >> > > >> The image is from a Windows 7 workstation. FTKimager took the disk > image > > >> in E01 format. I have the NSRL known good hash database loaded. > I?ve set > > >> number of threads to 12 as suggested by the Options dialog. I?m > running > > >> the default ingest process with no 3rd party modules. > > >> > > >> > > >> > > >> Performance Diagnostics > > >> > > >> > > >> > > >> Ingest Progress Snapshot > > >> > > >> > > >> > > >> 1 IDLE Wed Sep 14 > > >> 01:10:44 CDT 2016 15:49:08.024 0 > > >> > > >> 2 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 > > >> 2 > > >> > > >> 3 IDLE Wed Sep 14 > > >> 16:59:26 CDT 2016 0:00:25.758 0 > > >> > > >> 4 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.798 > > >> 2 > > >> > > >> 5 IDLE Wed Sep 14 > > >> 16:59:26 CDT 2016 0:00:25.764 0 > > >> > > >> 6 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:25.759 > > >> 2 > > >> > > >> 7 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 > > >> 2 > > >> > > >> 8 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.132 > > >> 2 > > >> > > >> 9 IDLE Wed Sep 14 > > >> 16:59:26 CDT 2016 0:00:25.742 0 > > >> > > >> 10 IDLE Wed Sep 14 > > >> 16:59:26 CDT 2016 0:00:25.812 0 > > >> > > >> 11 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.120 > > >> 2 > > >> > > >> 12 IDLE Wed Sep 14 > > >> 16:59:26 CDT 2016 0:00:25.811 0 > > >> > > >> 13 Keyword Search 2016-08-31-1-1.E01 > > >> image1.emf Wed Sep 14 16:59:26 CDT 2016 0:00:26.155 > > >> 2 > > >> > > >> > > >> > > >> 2 2016-08-31-1-1.E01 15:23:00 > 193034 > > >> 2.0938940654524942 84 2 36 > > >> 34 0 > > >> > > >> > > >> > > >> Keyword Search 155:52:59.626 (36%) > > >> > > >> File Type Identification 139:55:17.562 (32%) > > >> > > >> Hash Lookup 48:49:44.445 (11%) > > >> > > >> Embedded File Extractor 46:01:20.323 (10%) > > >> > > >> Email Parser 28:37:58.461 (6%) > > >> > > >> Extension Mismatch Detector 4:51:51.763 (1%) > > >> > > >> Exif Parser 1:41:39.597 (0%) > > >> > > >> PhotoRec Carver 0:00:04.762 (0%) > > >> > > >> Interesting Files Identifier 0:00:00.105 (0%) > > >> > > >> > > >> > > >> I get a bunch of errors but that is pretty typical. > > >> > > >> > > >> > > >> > > >> ------------------------------------------------------------ > > >> ------------------ > > >> > > >> _______________________________________________ > > >> sleuthkit-users mailing list > > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > >> http://www.sleuthkit.org > > >> > > >> > > >> > > >> ------------------------------------------------------------ > > >> ------------------ > > >> Check out the vibrant tech community on one of the world's most > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > >> _______________________________________________ > > >> sleuthkit-users mailing list > > >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > >> http://www.sleuthkit.org > > >> > > >> > > > ------------------------------------------------------------ > > > ------------------ > > > Check out the vibrant tech community on one of the world's most > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > sleuthkit-users mailing list > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > > http://www.sleuthkit.org > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: image008.jpg > > Type: image/jpeg > > Size: 38407 bytes > > Desc: not available > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: image007.jpg > > Type: image/jpeg > > Size: 12346 bytes > > Desc: not available > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: image002.png > > Type: image/png > > Size: 19142 bytes > > Desc: not available > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: image009.jpg > > Type: image/jpeg > > Size: 93667 bytes > > Desc: not available > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: image001.png > > Type: image/png > > Size: 13311 bytes > > Desc: not available > > > > ------------------------------ > > > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > > ------------------------------ > > > > _______________________________________________ > > sleuthkit-users mailing list > > sle...@li... > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > > > > > End of sleuthkit-users Digest, Vol 124, Issue 6 > > *********************************************** > > > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot______ > _________________________________________ > > sleuthkit-users mailing list > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users > > http://www.sleuthkit.org > > |
