Re: [sleuthkit-users] fiwalk

[Simson G. <si...@ac...>]

Hi, Brian.

fiwalk uses regular expressions to allow people to specify glob patterns of which files to process and which to ignore, as well as for processing plug-ins.   

For compiling under Windows, the correct thing to do would be to modify fiwalk to use the Microsoft regular expression library. As you indicate, this requires someone building and maintaining a Visual Studio profile. 

I don't recall the exact discussion we had about the file and path names  that TSK returns. I thought that the TSK functions were returning UTF-8, even when data in the file system are stored in UCS-2.  There is a function called tsk_UTF16toUTF8  which receives significant use. Clearly there is some transformation happening, even now.

As Alex indicated, it seems unlikely that we will be able to get current DFXML users to move to SQLite at this point.

I suspect that the correct way forward is your #3 - modify tsk_loaddb to output DFXML and incorporate the other features that are in fiwalk.  But realistically I don't think that anyone is going to do that at this point, since most of the DFXML users are on Linux, not Windows. What they have works.




> On Nov 11, 2014, at 12:26 PM, Brian Carrier <ca...@sl...> wrote:
> 
> Re Encoding:  At one point in TSK's life, we did cleanup of the names to ensure that they were stored in a 'printable' way.  The request was then make (from Simson, as I recall) to not do this cleanup so that there was an accurate representation of what was on disk and it was up to the user of the data to then clean it up as it saw fit for writing to disk or displaying to the user.   If people would rather return to the days of cleaning up in TSK, then we can certainly revisit that. 
> 
> There are certainly things that fiwalk can do that tsk_loaddb doesn't do:
> - SHA-1
> - Can limit output to certain files (based on arguments defined here: http://sancho.ccd.uniroma2.it/cgi-bin/man/man2html?fiwalk+1). 
> 
> I thought there used to be a bunch of regular expression and bloom filter stuff too, but don't see them mentioned in the man page. 
> 
> There seem to be three options:
> 1) Update fiwalk to compile with Visual Studio (i thought the regular expression libraries were an issue here, but I could be wrong -- it's been a while since I tried)
> 2) Force everyone to start using SQLite
> 3) Expand  tsk_loaddb to output DFXML instead of SQLite and as many of the other features as needed that are in fiwalk and not in tsk_loaddb that people want to have transferred.
> 
> 
> On Nov 10, 2014, at 3:18 PM, Simson Garfinkel <si...@ac...> wrote:
> 
>> Derrick,
>> 
>> My recommendation is that you transition away from fiwalk.
>> 
>> My one concern with tsk_loaddb is that the filenames it produces are not necessarily UTF-8. It seems to be putting in the database whatever is on the disk, which can cause problems in post-analysis.  I'm not sure how others are dealing with this. My problem is that on Windows, I'm reading these values with Python and I'm getting exceptions when I attempt to write them to a file. 
>> 
>> 
>> 
>>> On Nov 10, 2014, at 1:17 PM, Derrick Karpo <dk...@gm...> wrote:
>>> 
>>> I am still using fiwalk but have been transitioning to tsk_loaddb.
>>> 
>>> My primary reason is that my forensic indexer (Xapian) automatically
>>> indexes fiwalk text output and I haven't configured it to include
>>> sqlite files yet.  It's a simple config change for me to fully
>>> transition over.
>>> 
>>> Derrick
>>> 
>>> 
>>> On Mon, Nov 10, 2014 at 10:43 AM, Simson Garfinkel <si...@ac...> wrote:
>>>> I see.
>>>> 
>>>> The other approach would be to abandon fiwalk and move the things that use it over to using the database produced by tsk_loaddb.
>>>> 
>>>> Is anyone other than me using fiwalk at this point?
>>>> 
>>>> Simson
>>>> 
>>>> 
>>>>> On Nov 10, 2014, at 12:40 PM, Brian Carrier <ca...@sl...> wrote:
>>>>> 
>>>>> We don't use mingw for the TSK packaging.  Just Visual Studio.  So, it would be much easier to include if there were a visual studio project for it.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Nov 10, 2014, at 12:16 PM, Simson Garfinkel <si...@ac...> wrote:
>>>>> 
>>>>>> Hi. I see that fiwalk.exe is not being compiled as part of the pre-compiled SleuthKit download.
>>>>>> 
>>>>>> Is there some reason why fiwalk is not included, and is there any packaging change that I could make to make it more likely to include the executable in the future?
>>>>>> 
>>>>>> Simson
>>>>>> 
>>>>>> 
>>>>>> ------------------------------------------------------------------------------
>>>>>> _______________________________________________
>>>>>> sleuthkit-users mailing list
>>>>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>>>> http://www.sleuthkit.org
>>>>> 
>>>> 
>>>> 
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> sleuthkit-users mailing list
>>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> http://www.sleuthkit.org
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
>> _______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
>