Re: [sleuthkit-users] fiwalk

[Simson G. <si...@ac...>]

My take:

- MD5 collisions are generally not an issue in digital forensics if you are looking for known content. If you have the MD5 of a piece of stolen IP, it is unlikely that an adversary will attempt to craft multiple files to have the same MD5 as that stolen IP. 

- MD5 collisions may be an issue if you are using MD5s as an "ignore" list rather than as an "alert" list. That is, if you have a list of MD5s that you routinely ignore (e.g. executables), then there is a chance that an adversary may modify the MD5 of one of their files to match one of the files that you ignore. However it's hard. Currently it's possible to make two files have the same MD5, but I'm not aware that it is possible to tune a second file to match the MD5 of an arbitrary first file. That is, MD5 no longer has collision resistance, but I believe it still has preimage resistance.

In your case below, using MD5 for filtering known bads from unknowns is fine. However, using them for known goods is potentially problematic if you think that the adversary can get their known good into your known goods list.

Simson



> On Nov 12, 2014, at 12:19 PM, Jason Wright <jwr...@gm...> wrote:
> 
> Separately from this, but somewhat related, given the chatter regarding MD5 of late, e.g. http://arstechnica.com/security/2014/11/crypto-attack-that-hijacked-windows-update-goes-mainstream-in-amazon-cloud/ <http://arstechnica.com/security/2014/11/crypto-attack-that-hijacked-windows-update-goes-mainstream-in-amazon-cloud/>, is there any concern for relying on MD5 in digital forensics? 
> 
> I ask because it seems that tsk_loaddb only calculates MD5s for an image.Typically, we use fiwalk to gather MD5, SHA1, and SHA256 from files in a drive image. We use those for whitelisting/blacklisting, but do turn immediately to MD5 for filtering knowns (good and bad) from unknowns. SHA1, and then of course SHA256, are secondary to that process. Outside of this article, has there been any other MD5 collisions encountered? 
> 
> 
> On Wed, Nov 12, 2014 at 11:59 AM, Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
> Hi, Jason.
> 
> Sadly, tsk_loaddb does not grab file signatures. It can hash, but I don't know if it just does MD5 or if it does SHA1. It does not do SHA256.  The byte runs are available.
> 
> I gather you are running fiwalk on Linux?
> 
> 
> 
>> On Nov 12, 2014, at 11:23 AM, Jason Wright <jwr...@gm... <mailto:jwr...@gm...>> wrote:
>> 
>> Simson et al,
>> 
>> In my organization, we still use fiwalk too. We use it to gather as much file metadata as possible, to include file signatures, MD5, SHA1, and SHA256, and byte runs for file allocation along with the regular complement from the MFT indices. We have not used tsk_loaddb and if need be we can switch to that, provided we still get all of that info. Does it provide that breadth of detail?
>> 
>> V/R,
>> 
>> Jason
>> 
>> On Mon, Nov 10, 2014 at 3:43 PM, Alex Nelson <ajn...@cs... <mailto:ajn...@cs...>> wrote:
>> Hi Simson, all,
>> 
>> I'm making extensive use of Fiwalk.  I believe the BitCurator folks are as well.
>> 
>> --Alex
>> 
>> 
>> 
>> On Mon, Nov 10, 2014 at 3:18 PM, Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
>> Derrick,
>> 
>> My recommendation is that you transition away from fiwalk.
>> 
>> My one concern with tsk_loaddb is that the filenames it produces are not necessarily UTF-8. It seems to be putting in the database whatever is on the disk, which can cause problems in post-analysis.  I'm not sure how others are dealing with this. My problem is that on Windows, I'm reading these values with Python and I'm getting exceptions when I attempt to write them to a file.
>> 
>> 
>> 
>> > On Nov 10, 2014, at 1:17 PM, Derrick Karpo <dk...@gm... <mailto:dk...@gm...>> wrote:
>> >
>> > I am still using fiwalk but have been transitioning to tsk_loaddb.
>> >
>> > My primary reason is that my forensic indexer (Xapian) automatically
>> > indexes fiwalk text output and I haven't configured it to include
>> > sqlite files yet.  It's a simple config change for me to fully
>> > transition over.
>> >
>> > Derrick
>> >
>> >
>> > On Mon, Nov 10, 2014 at 10:43 AM, Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
>> >> I see.
>> >>
>> >> The other approach would be to abandon fiwalk and move the things that use it over to using the database produced by tsk_loaddb.
>> >>
>> >> Is anyone other than me using fiwalk at this point?
>> >>
>> >> Simson
>> >>
>> >>
>> >>> On Nov 10, 2014, at 12:40 PM, Brian Carrier <ca...@sl... <mailto:ca...@sl...>> wrote:
>> >>>
>> >>> We don't use mingw for the TSK packaging.  Just Visual Studio.  So, it would be much easier to include if there were a visual studio project for it.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Nov 10, 2014, at 12:16 PM, Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
>> >>>
>> >>>> Hi. I see that fiwalk.exe is not being compiled as part of the pre-compiled SleuthKit download.
>> >>>>
>> >>>> Is there some reason why fiwalk is not included, and is there any packaging change that I could make to make it more likely to include the executable in the future?
>> >>>>
>> >>>> Simson
>> >>>>
>> >>>>
>> >>>> ------------------------------------------------------------------------------
>> >>>> _______________________________________________
>> >>>> sleuthkit-users mailing list
>> >>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>> >>>> http://www.sleuthkit.org <http://www.sleuthkit.org/>
>> >>>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> _______________________________________________
>> >> sleuthkit-users mailing list
>> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>> >> http://www.sleuthkit.org <http://www.sleuthkit.org/>
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk <http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk>
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>> http://www.sleuthkit.org <http://www.sleuthkit.org/>
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk <http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk>
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>> http://www.sleuthkit.org <http://www.sleuthkit.org/>
>> 
>> 
> 
>