Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi,

The recent changes (from commit a725989 to 0432598) resolved the bug for my
2 problematic images and for my colleague's image, thank you all very much!
But with another image, the changes are causing a few files to be
discarded, returning the following message: "Error: Database Error
(TskDbSqlite::findParObjId: Error selecting file id by meta_addr: unknown
error (result code 101)"

For some reason the sql query did not find some parents in the database...
I don't know why (maybe the parents are not there?) Might those lost files
go into Orphan dir?

Thanks,
Luis

2016-06-06 18:09 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi,
>
> It is not related directly to this issue, but I discovered, based on
> Microsoft docs (
> https://msdn.microsoft.com/en-us/library/bb470124(v=vs.85).aspx), that
> the sequence number is incremented when the file record is freed and not
> when it is reused. I think that may affect the relationship between deleted
> children and their parent, but I do not tried to confirm it.
>
> Best,
> Luis
>
> 2016-05-02 15:19 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>
>> Hi,
>>
>> I applied the patch locally and it resolved the problem. Tested with one
>> of my cases.
>>
>> Thank you very much for all the attention!
>> Luis
>>
>>
>> 2016-05-02 9:20 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>>
>>> Hi Brian,
>>>
>>> I will gladly build and test the db_cache_uses_path branch and post
>>> results here.
>>>
>>> Thank you,
>>> Luis
>>>
>>> 2016-05-01 21:32 GMT-03:00 Brian Carrier <ca...@sl...>:
>>>
>>>> Hi Luis,
>>>>
>>>> I realized I forgot to circle back on this.  I committed a fix a while
>>>> back to the db_cache_uses_path branch on github. It would be great if you
>>>> could test that before we do another release. Can you build from github or
>>>> should I send you and exe?
>>>>
>>>> thanks,
>>>> brian
>>>>
>>>>
>>>> > On Apr 12, 2016, at 1:14 PM, Luís Filipe Nassif <lfc...@gm...>
>>>> wrote:
>>>> >
>>>> > Hi Brian,
>>>> >
>>>> > Yes, in my colleague's case, the deleted folder (dir2) name is "New
>>>> Folder (2)" and the true parent folder has a user specific name, so I think
>>>> some rename operation was done. But I am not sure if it was an unique
>>>> rename that have caused the problem. In other cases it is not so clear to
>>>> me, one has a Windows.old folder involved and looks like a system backup
>>>> was done. In another case the false and true parent folders are at
>>>> different levels in the FS tree.
>>>> >
>>>> > Thank you for your investigation,
>>>> > Luis
>>>> >
>>>> > 2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:
>>>> > Hi Luis,
>>>> >
>>>> > I want to understand how we got into this state.  Based on the names
>>>> of your folders, is it likely that the deleted folder (“dir2” in our
>>>> example below) was moved to be “dir1”?    That is the easiest explanation
>>>> of why a deleted name and an allocated name would point to the same entry
>>>> and have the same sequence.
>>>> >
>>>> > It still means there is an issue in the caching code that you pointed
>>>> out, I just want to make sure the solution makes sense for this general
>>>> problem.
>>>> >
>>>> > thanks,
>>>> > brian
>>>> >
>>>> > > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <
>>>> lfc...@gm...> wrote:
>>>> > >
>>>> > > Hi Brian,
>>>> > >
>>>> > > I've attached the istat output for dir1. I do not know how to get
>>>> istat result for dir2, because they have the same meta_addr. I searched the
>>>> $MFT for dir2 filename and did not find it, I was able to find only dir1
>>>> filename in $MFT exactly into the record at meta_addr offset.
>>>> > >
>>>> > > But I have found dir2 filename into $index_allocation ($I30) of its
>>>> parent directory. It is attached too. Finally, dir2 is shown as (realloc)
>>>> in fls output.
>>>> > >
>>>> > > Please let me know if you need any other info.
>>>> > >
>>>> > > Thank you very much,
>>>> > > Luis
>>>> > >
>>>> > > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
>>>> > > Hi Luis,
>>>> > >
>>>> > > Thanks for this info. Can you send me the ‘istat’ output for dir1
>>>> and dir2? I’m curious to see how they are different.
>>>> > >
>>>> > > thanks,
>>>> > > brian
>>>> > >
>>>> > >
>>>> > > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <
>>>> lfc...@gm...> wrote:
>>>> > > >
>>>> > > > Hi Brian,
>>>> > > >
>>>> > > > I have just confirmed fls shows the file path as /dir1/file1.txt
>>>> based on your example.
>>>> > > >
>>>> > > > But in tsk_objects table the parent of file1.txt is dir2, which
>>>> has the same meta_addr and meta_seq of dir1. I think the problem is in the
>>>> parent cache map used in TskDbSqlite::storeObjId() and
>>>> TskDbSqlite::findParObjId()
>>>> > > >
>>>> > > > I detected the issue in my cases searching for allocated files
>>>> under unallocated folders, maybe you could find that in your test cases.
>>>> > > >
>>>> > > > Thank you very much for your time,
>>>> > > > Luis
>>>> > > >
>>>> > > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...
>>>> >:
>>>> > > > Hi Brian,
>>>> > > >
>>>> > > > Your example illustrates the issue, except that file1.txt and
>>>> dir1 are allocated and dir2 is unallocated.
>>>> > > >
>>>> > > > I will run fls on one image to check what it reports and post
>>>> results.
>>>> > > >
>>>> > > > Thank you,
>>>> > > > Luis
>>>> > > >
>>>> > > > Em 23 de mar de 2016 23:44, "Brian Carrier" <
>>>> ca...@sl...> escreveu:
>>>> > > > Hi Luis,
>>>> > > >
>>>> > > > Can you provide a sanitized example of what you are seeing?  Is
>>>> it that:
>>>> > > >
>>>> > > > - You have an unallocated file.  If you look at its parent_path
>>>> and file name in tsk_file, then you get something like /dir1/ and file1.txt
>>>> (i.e. /dir1/file1.txt)
>>>> > > > - When you use the DB / Autopsy, the file is actually shown as a
>>>> child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
>>>> > > > - dir1 and dir2 have the same meta_add and meta_seq values in the
>>>> database.
>>>> > > > - If you were to run ‘fls’ on the image, it would show the file
>>>> as (/dir1/file1.txt).
>>>> > > >
>>>> > > > Is this correct?
>>>> > > >
>>>> > > > At some point in the past couple of years, we added a bunch of
>>>> sequence checks because we were incorrectly mapping files to parent
>>>> folders.  In theory, the sequence number is incremented each time an entry
>>>> is reallocated, so it should be a reliable way of mapping it to the correct
>>>> path.
>>>> > > >
>>>> > > > thanks,
>>>> > > > brian
>>>> > > >
>>>> > > >
>>>> > > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <
>>>> lfc...@gm...> wrote:
>>>> > > > >
>>>> > > > > Hi,
>>>> > > > >
>>>> > > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think
>>>> Autopsy too) incorrectly decoding the file system tree of some NTFS images.
>>>> I analysed one sqlite sent by him and tsk_loaddb is putting a number of
>>>> (very important) files into a deleted directory with the same meta_addr and
>>>> meta_seq of the true (not deleted) parent directory of those files,
>>>> according to tsk_objects table. The parent_path of the child files are
>>>> populated correctly into sqlite.
>>>> > > > >
>>>> > > > > Maybe the parent cache table logic used internally by
>>>> tsk_loaddb should be updated to handle that situation (NTFS files with same
>>>> meta_addr and meta_seq)?
>>>> > > > >
>>>> > > > > We will gladly provide any other information to help solve the
>>>> problem.
>>>> > > > >
>>>> > > > > Thank you very much for your attention,
>>>> > > > > Luis
>>>> > > > >
>>>> > > > >
>>>> ------------------------------------------------------------------------------
>>>> > > > > Transform Data into Opportunity.
>>>> > > > > Accelerate data analysis in your applications with
>>>> > > > > Intel Data Analytics Acceleration Library.
>>>> > > > > Click to learn more.
>>>> > > > >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>>> > > > > sleuthkit-users mailing list
>>>> > > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > > > > http://www.sleuthkit.org
>>>> > > >
>>>> > > >
>>>> > > >
>>>> ------------------------------------------------------------------------------
>>>> > > > Transform Data into Opportunity.
>>>> > > > Accelerate data analysis in your applications with
>>>> > > > Intel Data Analytics Acceleration Library.
>>>> > > > Click to learn more.
>>>> > > >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>>> > > > sleuthkit-users mailing list
>>>> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > > > http://www.sleuthkit.org
>>>> > >
>>>> > >
>>>> > >
>>>> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
>>>> > > Transform Data into Opportunity.
>>>> > > Accelerate data analysis in your applications with
>>>> > > Intel Data Analytics Acceleration Library.
>>>> > > Click to learn more.
>>>> > >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
>>>> > > sleuthkit-users mailing list
>>>> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > > http://www.sleuthkit.org
>>>> >
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Find and fix application performance issues faster with Applications
>>>> Manager
>>>> > Applications Manager provides deep performance insights into multiple
>>>> tiers of
>>>> > your business applications. It resolves application problems quickly
>>>> and
>>>> > reduces your MTTR. Get your free trial!
>>>> >
>>>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
>>>> > sleuthkit-users mailing list
>>>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > http://www.sleuthkit.org
>>>>
>>>>
>>>
>>
>