sleuthkit-users

[sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi,

A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too)
incorrectly decoding the file system tree of some NTFS images. I analysed
one sqlite sent by him and tsk_loaddb is putting a number of (very
important) files into a deleted directory with the same meta_addr and
meta_seq of the true (not deleted) parent directory of those files,
according to tsk_objects table. The parent_path of the child files are
populated correctly into sqlite.

Maybe the parent cache table logic used internally by tsk_loaddb should be
updated to handle that situation (NTFS files with same meta_addr and
meta_seq)?

We will gladly provide any other information to help solve the problem.

Thank you very much for your attention,
Luis

[sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Mauricio L. <mau...@gm...>]

Hi,
I have verified the same issue with a case I am currently working on: many
allocated files were put into a wrong deleted parent folder. But some of
those wrong deleted parent folders have exactly the same name and path of
the true parent folders. For example, there are two "Program Files" folders
in the root, one allocated and one deleted, and many allocated children
were put below the deleted "Program Files". So I think the current fix,
that uses the parent paths to decide into which folder they will be put,
will not work for some cases like mine.

thanks,

-- 
Mauricio S. Lage

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi Mauricio,

I think you are right. The fix uses parent paths to choose the correct
parent, so it will not work if the possible parents have the same path and
name. I think there should be another approach for the fix, because the
child paths always were ok, so tsk knows the correct parent at some point.

Luis

2016-06-24 15:10 GMT-03:00 Mauricio Lage <mau...@gm...>:

> Hi,
> I have verified the same issue with a case I am currently working on: many
> allocated files were put into a wrong deleted parent folder. But some of
> those wrong deleted parent folders have exactly the same name and path of
> the true parent folders. For example, there are two "Program Files" folders
> in the root, one allocated and one deleted, and many allocated children
> were put below the deleted "Program Files". So I think the current fix,
> that uses the parent paths to decide into which folder they will be put,
> will not work for some cases like mine.
>
> thanks,
>
> --
> Mauricio S. Lage
>
>
>
> ------------------------------------------------------------------------------
> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
> present their vision of the future. This family event has something for
> everyone, including kids. Get more information and register today.
> http://sdm.link/attshape
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

I have just confirmed this issue with 2 (of 3) ntfs images processed with
tsk_loaddb I am working right now, so the problem seems quite common. There
are a number of allocated files below some deleted directories in the file
system tree and the path of those files shows different parents. Looking
into the true parent directories (not deleted), there is no children. My
colleague observed this behaviour at least in 3 ntfs images.

I modified the logic of TskDbSqlite::storeObjId() and
TskDbSqlite::findParObjId in db_sqlite.cpp to ignore the NTFS sequence
number and to use the file paths when there are multiple items pointing to
the same meta_addr, like is already done for other file systems. That
change solved the problem, but I not sure if it could cause side effects
for NTFS?

Best regards,
Luis

2016-03-22 13:10 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi,
>
> A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too)
> incorrectly decoding the file system tree of some NTFS images. I analysed
> one sqlite sent by him and tsk_loaddb is putting a number of (very
> important) files into a deleted directory with the same meta_addr and
> meta_seq of the true (not deleted) parent directory of those files,
> according to tsk_objects table. The parent_path of the child files are
> populated correctly into sqlite.
>
> Maybe the parent cache table logic used internally by tsk_loaddb should be
> updated to handle that situation (NTFS files with same meta_addr and
> meta_seq)?
>
> We will gladly provide any other information to help solve the problem.
>
> Thank you very much for your attention,
> Luis
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Simson G. <si...@ac...>]

Luis,

This is the point at which someone (Alex?) should be asking a few pointed questions, to wit:

• Does anybody know of any formal validation that's been done to compare the output of FTK, EnCase, TSK, and Autopsy?


> On Mar 23, 2016, at 7:17 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
> I have just confirmed this issue with 2 (of 3) ntfs images processed with tsk_loaddb I am working right now, so the problem seems quite common. There are a number of allocated files below some deleted directories in the file system tree and the path of those files shows different parents. Looking into the true parent directories (not deleted), there is no children. My colleague observed this behaviour at least in 3 ntfs images.
> 
> I modified the logic of TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId in db_sqlite.cpp to ignore the NTFS sequence number and to use the file paths when there are multiple items pointing to the same meta_addr, like is already done for other file systems. That change solved the problem, but I not sure if it could cause side effects for NTFS?
> 
> Best regards,
> Luis 
> 
> 2016-03-22 13:10 GMT-03:00 Luís Filipe Nassif <lfc...@gm... <mailto:lfc...@gm...>>:
> Hi,
> 
> A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
> 
> Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
> 
> We will gladly provide any other information to help solve the problem.
> 
> Thank you very much for your attention,
> Luis
> 
> 
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Mike H. <mi...@zu...>]

Depending on how you traverse NTFS it is possible to come up with multiple paths for the same file.  Parents have pointers to their children and children have pointers to their parents, and (in the case of deleted directories) they don't have to match.  A top down traversal of the tree via the Index Root and Index Allocation attributes will result in allocated files under deleted directories.  A flat (or bottom up) traversal of the MFT that uses the parent reference in the File Name attribute is going to give you a file's true parent directory, which should always be allocated if the file was allocated.  Most tools show the results of the bottom up approach as the "path" of a file.

> On Mar 23, 2016, at 7:32 PM, Simson Garfinkel <si...@ac...> wrote:
> 
> Luis,
> 
> This is the point at which someone (Alex?) should be asking a few pointed questions, to wit:
> 
> • Does anybody know of any formal validation that's been done to compare the output of FTK, EnCase, TSK, and Autopsy?
> 
> 
>> On Mar 23, 2016, at 7:17 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
>> 
>> I have just confirmed this issue with 2 (of 3) ntfs images processed with tsk_loaddb I am working right now, so the problem seems quite common. There are a number of allocated files below some deleted directories in the file system tree and the path of those files shows different parents. Looking into the true parent directories (not deleted), there is no children. My colleague observed this behaviour at least in 3 ntfs images.
>> 
>> I modified the logic of TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId in db_sqlite.cpp to ignore the NTFS sequence number and to use the file paths when there are multiple items pointing to the same meta_addr, like is already done for other file systems. That change solved the problem, but I not sure if it could cause side effects for NTFS?
>> 
>> Best regards,
>> Luis 
>> 
>> 2016-03-22 13:10 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>>> Hi,
>>> 
>>> A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
>>> 
>>> Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
>>> 
>>> We will gladly provide any other information to help solve the problem.
>>> 
>>> Thank you very much for your attention,
>>> Luis
>>> 
>> 
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
> 
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Simson G. <si...@ac...>]

Mike,

That's fine.  If multiple paths work, then we should either have a canonical path or a way for validating that a path is correct. However, I think that another thing you would want to check is to make sure that the different tools consistently report whether or not a file is allocated, the file's physical location in the disk image, file length, etc.

Alex Nelson was looking at ways of doing inter-tool verification and had a published article on this:
	https://www.soe.ucsc.edu/events/event/2981 <https://www.soe.ucsc.edu/events/event/2981>

I'm curious as to whether or not the TSK developers have such a validation process in place.

Simson


> On Mar 23, 2016, at 8:42 PM, Mike Haaf <mi...@zu...> wrote:
> 
> Depending on how you traverse NTFS it is possible to come up with multiple paths for the same file.  Parents have pointers to their children and children have pointers to their parents, and (in the case of deleted directories) they don't have to match.  A top down traversal of the tree via the Index Root and Index Allocation attributes will result in allocated files under deleted directories.  A flat (or bottom up) traversal of the MFT that uses the parent reference in the File Name attribute is going to give you a file's true parent directory, which should always be allocated if the file was allocated.  Most tools show the results of the bottom up approach as the "path" of a file.
> 
> On Mar 23, 2016, at 7:32 PM, Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
> 
>> Luis,
>> 
>> This is the point at which someone (Alex?) should be asking a few pointed questions, to wit:
>> 
>> • Does anybody know of any formal validation that's been done to compare the output of FTK, EnCase, TSK, and Autopsy?
>> 
>> 
>>> On Mar 23, 2016, at 7:17 PM, Luís Filipe Nassif <lfc...@gm... <mailto:lfc...@gm...>> wrote:
>>> 
>>> I have just confirmed this issue with 2 (of 3) ntfs images processed with tsk_loaddb I am working right now, so the problem seems quite common. There are a number of allocated files below some deleted directories in the file system tree and the path of those files shows different parents. Looking into the true parent directories (not deleted), there is no children. My colleague observed this behaviour at least in 3 ntfs images.
>>> 
>>> I modified the logic of TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId in db_sqlite.cpp to ignore the NTFS sequence number and to use the file paths when there are multiple items pointing to the same meta_addr, like is already done for other file systems. That change solved the problem, but I not sure if it could cause side effects for NTFS?
>>> 
>>> Best regards,
>>> Luis 
>>> 
>>> 2016-03-22 13:10 GMT-03:00 Luís Filipe Nassif <lfc...@gm... <mailto:lfc...@gm...>>:
>>> Hi,
>>> 
>>> A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
>>> 
>>> Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
>>> 
>>> We will gladly provide any other information to help solve the problem.
>>> 
>>> Thank you very much for your attention,
>>> Luis
>>> 
>>> 
>>> ------------------------------------------------------------------------------
>>> Transform Data into Opportunity.
>>> Accelerate data analysis in your applications with
>>> Intel Data Analytics Acceleration Library.
>>> Click to learn more.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________ <http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________>
>>> sleuthkit-users mailing list
>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>>> http://www.sleuthkit.org <http://www.sleuthkit.org/>
>> 
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 <http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140>_______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>> http://www.sleuthkit.org <http://www.sleuthkit.org/>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Alex N. <ajn...@cs...>]

Hi all,

I’m just seeing tonight’s thread.

I did publish an article on the topic of multi-tool verification of file system state, but Simson provided the wrong link.
<https://www.dfrws.org/2014/proceedings/DFRWS2014-6.pdf <https://www.dfrws.org/2014/proceedings/DFRWS2014-6.pdf>>

If anybody is curious on verifying NTFS with The SleuthKit’s code base and EnCase, there is an approach you can take based on Digital Forensics XML.  Try using Fiwalk and Sebastien Bourdon-Richard’s EnScript to generate DFXML:
<https://github.com/Sebastienbr/DFXML-EnCase <https://github.com/Sebastienbr/DFXML-EnCase>>

You can compare results using this tool, written for the DFRWS paper:
<https://github.com/ajnelson/fsnview <https://github.com/ajnelson/fsnview>>

I hope that helps, and I’ll be curious to hear what you find.  If you have weird issues based on deletion, they may be related to this:
<https://github.com/dfxml-working-group/dfxml_schema/issues/14 <https://github.com/dfxml-working-group/dfxml_schema/issues/14>>

I’m still interested in the topic of tool verification, but I’m wrapping up an unrelated multi-year project in the next two months, so I’ve been quiet on this front.

—Alex


> On Mar 23, 2016, at 21:06 , Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
> 
> Mike,
> 
> That's fine.  If multiple paths work, then we should either have a canonical path or a way for validating that a path is correct. However, I think that another thing you would want to check is to make sure that the different tools consistently report whether or not a file is allocated, the file's physical location in the disk image, file length, etc.
> 
> Alex Nelson was looking at ways of doing inter-tool verification and had a published article on this:
> 	https://www.soe.ucsc.edu/events/event/2981 <https://www.soe.ucsc.edu/events/event/2981>
> 
> I'm curious as to whether or not the TSK developers have such a validation process in place.
> 
> Simson
> 
> 
>> On Mar 23, 2016, at 8:42 PM, Mike Haaf <mi...@zu... <mailto:mi...@zu...>> wrote:
>> 
>> Depending on how you traverse NTFS it is possible to come up with multiple paths for the same file.  Parents have pointers to their children and children have pointers to their parents, and (in the case of deleted directories) they don't have to match.  A top down traversal of the tree via the Index Root and Index Allocation attributes will result in allocated files under deleted directories.  A flat (or bottom up) traversal of the MFT that uses the parent reference in the File Name attribute is going to give you a file's true parent directory, which should always be allocated if the file was allocated.  Most tools show the results of the bottom up approach as the "path" of a file.
>> 
>> On Mar 23, 2016, at 7:32 PM, Simson Garfinkel <si...@ac... <mailto:si...@ac...>> wrote:
>> 
>>> Luis,
>>> 
>>> This is the point at which someone (Alex?) should be asking a few pointed questions, to wit:
>>> 
>>> • Does anybody know of any formal validation that's been done to compare the output of FTK, EnCase, TSK, and Autopsy?
>>> 
>>> 
>>>> On Mar 23, 2016, at 7:17 PM, Luís Filipe Nassif <lfc...@gm... <mailto:lfc...@gm...>> wrote:
>>>> 
>>>> I have just confirmed this issue with 2 (of 3) ntfs images processed with tsk_loaddb I am working right now, so the problem seems quite common. There are a number of allocated files below some deleted directories in the file system tree and the path of those files shows different parents. Looking into the true parent directories (not deleted), there is no children. My colleague observed this behaviour at least in 3 ntfs images.
>>>> 
>>>> I modified the logic of TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId in db_sqlite.cpp to ignore the NTFS sequence number and to use the file paths when there are multiple items pointing to the same meta_addr, like is already done for other file systems. That change solved the problem, but I not sure if it could cause side effects for NTFS?
>>>> 
>>>> Best regards,
>>>> Luis 
>>>> 
>>>> 2016-03-22 13:10 GMT-03:00 Luís Filipe Nassif <lfc...@gm... <mailto:lfc...@gm...>>:
>>>> Hi,
>>>> 
>>>> A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
>>>> 
>>>> Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
>>>> 
>>>> We will gladly provide any other information to help solve the problem.
>>>> 
>>>> Thank you very much for your attention,
>>>> Luis
>>>> 
>>>> 
>>>> ------------------------------------------------------------------------------
>>>> Transform Data into Opportunity.
>>>> Accelerate data analysis in your applications with
>>>> Intel Data Analytics Acceleration Library.
>>>> Click to learn more.
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________ <http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________>
>>>> sleuthkit-users mailing list
>>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>>>> http://www.sleuthkit.org <http://www.sleuthkit.org/>
>>> 
>>> ------------------------------------------------------------------------------
>>> Transform Data into Opportunity.
>>> Accelerate data analysis in your applications with
>>> Intel Data Analytics Acceleration Library.
>>> Click to learn more.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 <http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140>_______________________________________________
>>> sleuthkit-users mailing list
>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users <https://lists.sourceforge.net/lists/listinfo/sleuthkit-users>
>>> http://www.sleuthkit.org <http://www.sleuthkit.org/>
> 
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________ <http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________>
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Brian C. <ca...@sl...>]

Hi Luis,

Can you provide a sanitized example of what you are seeing?  Is it that:

- You have an unallocated file.  If you look at its parent_path and file name in tsk_file, then you get something like /dir1/ and file1.txt (i.e. /dir1/file1.txt)
- When you use the DB / Autopsy, the file is actually shown as a child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
- dir1 and dir2 have the same meta_add and meta_seq values in the database.
- If you were to run ‘fls’ on the image, it would show the file as (/dir1/file1.txt).

Is this correct?

At some point in the past couple of years, we added a bunch of sequence checks because we were incorrectly mapping files to parent folders.  In theory, the sequence number is incremented each time an entry is reallocated, so it should be a reliable way of mapping it to the correct path. 

thanks,
brian


> On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
> Hi,
> 
> A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
> 
> Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
> 
> We will gladly provide any other information to help solve the problem.
> 
> Thank you very much for your attention,
> Luis
> 
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi Brian,

Your example illustrates the issue, except that file1.txt and dir1 are
allocated and dir2 is unallocated.

I will run fls on one image to check what it reports and post results.

Thank you,
Luis
Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
escreveu:

> Hi Luis,
>
> Can you provide a sanitized example of what you are seeing?  Is it that:
>
> - You have an unallocated file.  If you look at its parent_path and file
> name in tsk_file, then you get something like /dir1/ and file1.txt (i.e.
> /dir1/file1.txt)
> - When you use the DB / Autopsy, the file is actually shown as a child of
> another directory, dir2 for example.  (i.e. /dir2/file1.txt).
> - dir1 and dir2 have the same meta_add and meta_seq values in the database.
> - If you were to run ‘fls’ on the image, it would show the file as
> (/dir1/file1.txt).
>
> Is this correct?
>
> At some point in the past couple of years, we added a bunch of sequence
> checks because we were incorrectly mapping files to parent folders.  In
> theory, the sequence number is incremented each time an entry is
> reallocated, so it should be a reliable way of mapping it to the correct
> path.
>
> thanks,
> brian
>
>
> > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> >
> > Hi,
> >
> > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy
> too) incorrectly decoding the file system tree of some NTFS images. I
> analysed one sqlite sent by him and tsk_loaddb is putting a number of (very
> important) files into a deleted directory with the same meta_addr and
> meta_seq of the true (not deleted) parent directory of those files,
> according to tsk_objects table. The parent_path of the child files are
> populated correctly into sqlite.
> >
> > Maybe the parent cache table logic used internally by tsk_loaddb should
> be updated to handle that situation (NTFS files with same meta_addr and
> meta_seq)?
> >
> > We will gladly provide any other information to help solve the problem.
> >
> > Thank you very much for your attention,
> > Luis
> >
> >
> ------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi Brian,

I have just confirmed fls shows the file path as /dir1/file1.txt based on
your example.

But in tsk_objects table the parent of file1.txt is dir2, which has the
same meta_addr and meta_seq of dir1. I think the problem is in the parent
cache map used in TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId()

I detected the issue in my cases searching for allocated files under
unallocated folders, maybe you could find that in your test cases.

Thank you very much for your time,
Luis

2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi Brian,
>
> Your example illustrates the issue, except that file1.txt and dir1 are
> allocated and dir2 is unallocated.
>
> I will run fls on one image to check what it reports and post results.
>
> Thank you,
> Luis
> Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
> escreveu:
>
>> Hi Luis,
>>
>> Can you provide a sanitized example of what you are seeing?  Is it that:
>>
>> - You have an unallocated file.  If you look at its parent_path and file
>> name in tsk_file, then you get something like /dir1/ and file1.txt (i.e.
>> /dir1/file1.txt)
>> - When you use the DB / Autopsy, the file is actually shown as a child of
>> another directory, dir2 for example.  (i.e. /dir2/file1.txt).
>> - dir1 and dir2 have the same meta_add and meta_seq values in the
>> database.
>> - If you were to run ‘fls’ on the image, it would show the file as
>> (/dir1/file1.txt).
>>
>> Is this correct?
>>
>> At some point in the past couple of years, we added a bunch of sequence
>> checks because we were incorrectly mapping files to parent folders.  In
>> theory, the sequence number is incremented each time an entry is
>> reallocated, so it should be a reliable way of mapping it to the correct
>> path.
>>
>> thanks,
>> brian
>>
>>
>> > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...>
>> wrote:
>> >
>> > Hi,
>> >
>> > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy
>> too) incorrectly decoding the file system tree of some NTFS images. I
>> analysed one sqlite sent by him and tsk_loaddb is putting a number of (very
>> important) files into a deleted directory with the same meta_addr and
>> meta_seq of the true (not deleted) parent directory of those files,
>> according to tsk_objects table. The parent_path of the child files are
>> populated correctly into sqlite.
>> >
>> > Maybe the parent cache table logic used internally by tsk_loaddb should
>> be updated to handle that situation (NTFS files with same meta_addr and
>> meta_seq)?
>> >
>> > We will gladly provide any other information to help solve the problem.
>> >
>> > Thank you very much for your attention,
>> > Luis
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Transform Data into Opportunity.
>> > Accelerate data analysis in your applications with
>> > Intel Data Analytics Acceleration Library.
>> > Click to learn more.
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>> > sleuthkit-users mailing list
>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > http://www.sleuthkit.org
>>
>>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

One more info, the obj_id of dir2 is less than obj_id of dir1, so I think
obj_id of dir2 (unallocated) is saved in the parent cache map before
processing dir1.

2016-03-24 10:29 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi Brian,
>
> I have just confirmed fls shows the file path as /dir1/file1.txt based on
> your example.
>
> But in tsk_objects table the parent of file1.txt is dir2, which has the
> same meta_addr and meta_seq of dir1. I think the problem is in the parent
> cache map used in TskDbSqlite::storeObjId() and
> TskDbSqlite::findParObjId()
>
> I detected the issue in my cases searching for allocated files under
> unallocated folders, maybe you could find that in your test cases.
>
> Thank you very much for your time,
> Luis
>
> 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>
>> Hi Brian,
>>
>> Your example illustrates the issue, except that file1.txt and dir1 are
>> allocated and dir2 is unallocated.
>>
>> I will run fls on one image to check what it reports and post results.
>>
>> Thank you,
>> Luis
>> Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
>> escreveu:
>>
>>> Hi Luis,
>>>
>>> Can you provide a sanitized example of what you are seeing?  Is it that:
>>>
>>> - You have an unallocated file.  If you look at its parent_path and file
>>> name in tsk_file, then you get something like /dir1/ and file1.txt (i.e.
>>> /dir1/file1.txt)
>>> - When you use the DB / Autopsy, the file is actually shown as a child
>>> of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
>>> - dir1 and dir2 have the same meta_add and meta_seq values in the
>>> database.
>>> - If you were to run ‘fls’ on the image, it would show the file as
>>> (/dir1/file1.txt).
>>>
>>> Is this correct?
>>>
>>> At some point in the past couple of years, we added a bunch of sequence
>>> checks because we were incorrectly mapping files to parent folders.  In
>>> theory, the sequence number is incremented each time an entry is
>>> reallocated, so it should be a reliable way of mapping it to the correct
>>> path.
>>>
>>> thanks,
>>> brian
>>>
>>>
>>> > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...>
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy
>>> too) incorrectly decoding the file system tree of some NTFS images. I
>>> analysed one sqlite sent by him and tsk_loaddb is putting a number of (very
>>> important) files into a deleted directory with the same meta_addr and
>>> meta_seq of the true (not deleted) parent directory of those files,
>>> according to tsk_objects table. The parent_path of the child files are
>>> populated correctly into sqlite.
>>> >
>>> > Maybe the parent cache table logic used internally by tsk_loaddb
>>> should be updated to handle that situation (NTFS files with same meta_addr
>>> and meta_seq)?
>>> >
>>> > We will gladly provide any other information to help solve the problem.
>>> >
>>> > Thank you very much for your attention,
>>> > Luis
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Transform Data into Opportunity.
>>> > Accelerate data analysis in your applications with
>>> > Intel Data Analytics Acceleration Library.
>>> > Click to learn more.
>>> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>> > sleuthkit-users mailing list
>>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > http://www.sleuthkit.org
>>>
>>>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Brian C. <ca...@sl...>]

Hi Luis,

Thanks for this info. Can you send me the ‘istat’ output for dir1 and dir2? I’m curious to see how they are different.

thanks,
brian


> On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
> Hi Brian,
> 
> I have just confirmed fls shows the file path as /dir1/file1.txt based on your example.
> 
> But in tsk_objects table the parent of file1.txt is dir2, which has the same meta_addr and meta_seq of dir1. I think the problem is in the parent cache map used in TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId()
> 
> I detected the issue in my cases searching for allocated files under unallocated folders, maybe you could find that in your test cases.
> 
> Thank you very much for your time,
> Luis
> 
> 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> Hi Brian,
> 
> Your example illustrates the issue, except that file1.txt and dir1 are allocated and dir2 is unallocated.
> 
> I will run fls on one image to check what it reports and post results.
> 
> Thank you,
> Luis
> 
> Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...> escreveu:
> Hi Luis,
> 
> Can you provide a sanitized example of what you are seeing?  Is it that:
> 
> - You have an unallocated file.  If you look at its parent_path and file name in tsk_file, then you get something like /dir1/ and file1.txt (i.e. /dir1/file1.txt)
> - When you use the DB / Autopsy, the file is actually shown as a child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
> - dir1 and dir2 have the same meta_add and meta_seq values in the database.
> - If you were to run ‘fls’ on the image, it would show the file as (/dir1/file1.txt).
> 
> Is this correct?
> 
> At some point in the past couple of years, we added a bunch of sequence checks because we were incorrectly mapping files to parent folders.  In theory, the sequence number is incremented each time an entry is reallocated, so it should be a reliable way of mapping it to the correct path.
> 
> thanks,
> brian
> 
> 
> > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> >
> > Hi,
> >
> > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
> >
> > Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
> >
> > We will gladly provide any other information to help solve the problem.
> >
> > Thank you very much for your attention,
> > Luis
> >
> > ------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> 
> 
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

MFT Entry Header Values:
Entry: 24765        Sequence: 514
$LogFile Sequence Number: 18426169393
Allocated Directory
Links: 2

$STANDARD_INFORMATION Attribute Values:
Flags: 
Owner ID: 0
Security ID: 1915  (S-1-5-21-1420377728-1435070833-5377028-1000)
Last User Journal Update Sequence Number: 3021062824
Created:	2014-01-21 22:25:14.139789800 (HorFile Modified:	2014-03-10 13:56:09.287806600 (Hora oficial do Brasil)
MFT Modified:	2014-03-23 16:48:56.241901700 (Hora oficial do Brasil)
Accessed:	2014-03-10 13:56:09.287806600 (Hora oficial do Brasil)

$FILE_NAME Attribute Values:
Flags: Directory
Name: 2001 FI
Parent MFT Entry: 260633 	Sequence: 2
Allocated Size: 0   	Actual Size: 0
Created:	2014-01-21 22:25:14.139789800 (HorFile Modified:	2014-03-10 13:56:09.287806600 (Hora oficial do Brasil)
MFT Modified:	2014-03-10 13:56:09.287806600 (Hora oficial do Brasil)
Accessed:	2014-03-10 13:56:09.287806600 (Hora oficial do Brasil)

$OBJECT_ID Attribute Values:
Object Id: abdbf634-2b90-34a7-11e3-88342f05a13b

Attributes: 
Type: $STANDARD_INFORMATION (16-0)   Name: N/A   Resident   size: 72
Type: $FILE_NAME (48-13)   Name: N/A   Resident   size: 80
Type: $FILE_NAME (48-14)   Name: N/A   Resident   size: 82
Type: $OBJECT_ID (64-10)   Name: N/A   Resident   size: 16
Type: $INDEX_ROOT (144-9)   Name: $I30   Resident   size: 176
Type: $INDEX_ALLOCATION (160-7)   Name: $I30   Non-Resident   size: 12288  init_size: 12288
1037942 1037943 1037944 
Type: $BITMAP (176-8)   Name: $I30   Resident   size: 8

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Brian C. <ca...@sl...>]

Hi Luis,

I want to understand how we got into this state.  Based on the names of your folders, is it likely that the deleted folder (“dir2” in our example below) was moved to be “dir1”?    That is the easiest explanation of why a deleted name and an allocated name would point to the same entry and have the same sequence. 

It still means there is an issue in the caching code that you pointed out, I just want to make sure the solution makes sense for this general problem. 

thanks,
brian

> On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
> Hi Brian,
> 
> I've attached the istat output for dir1. I do not know how to get istat result for dir2, because they have the same meta_addr. I searched the $MFT for dir2 filename and did not find it, I was able to find only dir1 filename in $MFT exactly into the record at meta_addr offset.
> 
> But I have found dir2 filename into $index_allocation ($I30) of its parent directory. It is attached too. Finally, dir2 is shown as (realloc) in fls output.
> 
> Please let me know if you need any other info.
> 
> Thank you very much,
> Luis
> 
> 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
> Hi Luis,
> 
> Thanks for this info. Can you send me the ‘istat’ output for dir1 and dir2? I’m curious to see how they are different.
> 
> thanks,
> brian
> 
> 
> > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
> >
> > Hi Brian,
> >
> > I have just confirmed fls shows the file path as /dir1/file1.txt based on your example.
> >
> > But in tsk_objects table the parent of file1.txt is dir2, which has the same meta_addr and meta_seq of dir1. I think the problem is in the parent cache map used in TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId()
> >
> > I detected the issue in my cases searching for allocated files under unallocated folders, maybe you could find that in your test cases.
> >
> > Thank you very much for your time,
> > Luis
> >
> > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> > Hi Brian,
> >
> > Your example illustrates the issue, except that file1.txt and dir1 are allocated and dir2 is unallocated.
> >
> > I will run fls on one image to check what it reports and post results.
> >
> > Thank you,
> > Luis
> >
> > Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...> escreveu:
> > Hi Luis,
> >
> > Can you provide a sanitized example of what you are seeing?  Is it that:
> >
> > - You have an unallocated file.  If you look at its parent_path and file name in tsk_file, then you get something like /dir1/ and file1.txt (i.e. /dir1/file1.txt)
> > - When you use the DB / Autopsy, the file is actually shown as a child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
> > - dir1 and dir2 have the same meta_add and meta_seq values in the database.
> > - If you were to run ‘fls’ on the image, it would show the file as (/dir1/file1.txt).
> >
> > Is this correct?
> >
> > At some point in the past couple of years, we added a bunch of sequence checks because we were incorrectly mapping files to parent folders.  In theory, the sequence number is incremented each time an entry is reallocated, so it should be a reliable way of mapping it to the correct path.
> >
> > thanks,
> > brian
> >
> >
> > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> > >
> > > Hi,
> > >
> > > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
> > >
> > > Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
> > >
> > > We will gladly provide any other information to help solve the problem.
> > >
> > > Thank you very much for your attention,
> > > Luis
> > >
> > > ------------------------------------------------------------------------------
> > > Transform Data into Opportunity.
> > > Accelerate data analysis in your applications with
> > > Intel Data Analytics Acceleration Library.
> > > Click to learn more.
> > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> >
> >
> > ------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> 
> 
> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi Brian,

Yes, in my colleague's case, the deleted folder (dir2) name is "New Folder
(2)" and the true parent folder has a user specific name, so I think some
rename operation was done. But I am not sure if it was an unique rename
that have caused the problem. In other cases it is not so clear to me, one
has a Windows.old folder involved and looks like a system backup was done.
In another case the false and true parent folders are at different levels
in the FS tree.

Thank you for your investigation,
Luis

2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:

> Hi Luis,
>
> I want to understand how we got into this state.  Based on the names of
> your folders, is it likely that the deleted folder (“dir2” in our example
> below) was moved to be “dir1”?    That is the easiest explanation of why a
> deleted name and an allocated name would point to the same entry and have
> the same sequence.
>
> It still means there is an issue in the caching code that you pointed out,
> I just want to make sure the solution makes sense for this general problem.
>
> thanks,
> brian
>
> > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> >
> > Hi Brian,
> >
> > I've attached the istat output for dir1. I do not know how to get istat
> result for dir2, because they have the same meta_addr. I searched the $MFT
> for dir2 filename and did not find it, I was able to find only dir1
> filename in $MFT exactly into the record at meta_addr offset.
> >
> > But I have found dir2 filename into $index_allocation ($I30) of its
> parent directory. It is attached too. Finally, dir2 is shown as (realloc)
> in fls output.
> >
> > Please let me know if you need any other info.
> >
> > Thank you very much,
> > Luis
> >
> > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
> > Hi Luis,
> >
> > Thanks for this info. Can you send me the ‘istat’ output for dir1 and
> dir2? I’m curious to see how they are different.
> >
> > thanks,
> > brian
> >
> >
> > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> > >
> > > Hi Brian,
> > >
> > > I have just confirmed fls shows the file path as /dir1/file1.txt based
> on your example.
> > >
> > > But in tsk_objects table the parent of file1.txt is dir2, which has
> the same meta_addr and meta_seq of dir1. I think the problem is in the
> parent cache map used in TskDbSqlite::storeObjId() and
> TskDbSqlite::findParObjId()
> > >
> > > I detected the issue in my cases searching for allocated files under
> unallocated folders, maybe you could find that in your test cases.
> > >
> > > Thank you very much for your time,
> > > Luis
> > >
> > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> > > Hi Brian,
> > >
> > > Your example illustrates the issue, except that file1.txt and dir1 are
> allocated and dir2 is unallocated.
> > >
> > > I will run fls on one image to check what it reports and post results.
> > >
> > > Thank you,
> > > Luis
> > >
> > > Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
> escreveu:
> > > Hi Luis,
> > >
> > > Can you provide a sanitized example of what you are seeing?  Is it
> that:
> > >
> > > - You have an unallocated file.  If you look at its parent_path and
> file name in tsk_file, then you get something like /dir1/ and file1.txt
> (i.e. /dir1/file1.txt)
> > > - When you use the DB / Autopsy, the file is actually shown as a child
> of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
> > > - dir1 and dir2 have the same meta_add and meta_seq values in the
> database.
> > > - If you were to run ‘fls’ on the image, it would show the file as
> (/dir1/file1.txt).
> > >
> > > Is this correct?
> > >
> > > At some point in the past couple of years, we added a bunch of
> sequence checks because we were incorrectly mapping files to parent
> folders.  In theory, the sequence number is incremented each time an entry
> is reallocated, so it should be a reliable way of mapping it to the correct
> path.
> > >
> > > thanks,
> > > brian
> > >
> > >
> > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <
> lfc...@gm...> wrote:
> > > >
> > > > Hi,
> > > >
> > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy
> too) incorrectly decoding the file system tree of some NTFS images. I
> analysed one sqlite sent by him and tsk_loaddb is putting a number of (very
> important) files into a deleted directory with the same meta_addr and
> meta_seq of the true (not deleted) parent directory of those files,
> according to tsk_objects table. The parent_path of the child files are
> populated correctly into sqlite.
> > > >
> > > > Maybe the parent cache table logic used internally by tsk_loaddb
> should be updated to handle that situation (NTFS files with same meta_addr
> and meta_seq)?
> > > >
> > > > We will gladly provide any other information to help solve the
> problem.
> > > >
> > > > Thank you very much for your attention,
> > > > Luis
> > > >
> > > >
> ------------------------------------------------------------------------------
> > > > Transform Data into Opportunity.
> > > > Accelerate data analysis in your applications with
> > > > Intel Data Analytics Acceleration Library.
> > > > Click to learn more.
> > > >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > > sleuthkit-users mailing list
> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > > http://www.sleuthkit.org
> > >
> > >
> > >
> ------------------------------------------------------------------------------
> > > Transform Data into Opportunity.
> > > Accelerate data analysis in your applications with
> > > Intel Data Analytics Acceleration Library.
> > > Click to learn more.
> > >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> >
> >
> >
> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Brian C. <ca...@sl...>]

Hi Luis,

I realized I forgot to circle back on this.  I committed a fix a while back to the db_cache_uses_path branch on github. It would be great if you could test that before we do another release. Can you build from github or should I send you and exe?

thanks,
brian


> On Apr 12, 2016, at 1:14 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
> Hi Brian,
> 
> Yes, in my colleague's case, the deleted folder (dir2) name is "New Folder (2)" and the true parent folder has a user specific name, so I think some rename operation was done. But I am not sure if it was an unique rename that have caused the problem. In other cases it is not so clear to me, one has a Windows.old folder involved and looks like a system backup was done. In another case the false and true parent folders are at different levels in the FS tree.
> 
> Thank you for your investigation,
> Luis
> 
> 2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:
> Hi Luis,
> 
> I want to understand how we got into this state.  Based on the names of your folders, is it likely that the deleted folder (“dir2” in our example below) was moved to be “dir1”?    That is the easiest explanation of why a deleted name and an allocated name would point to the same entry and have the same sequence.
> 
> It still means there is an issue in the caching code that you pointed out, I just want to make sure the solution makes sense for this general problem.
> 
> thanks,
> brian
> 
> > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
> >
> > Hi Brian,
> >
> > I've attached the istat output for dir1. I do not know how to get istat result for dir2, because they have the same meta_addr. I searched the $MFT for dir2 filename and did not find it, I was able to find only dir1 filename in $MFT exactly into the record at meta_addr offset.
> >
> > But I have found dir2 filename into $index_allocation ($I30) of its parent directory. It is attached too. Finally, dir2 is shown as (realloc) in fls output.
> >
> > Please let me know if you need any other info.
> >
> > Thank you very much,
> > Luis
> >
> > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
> > Hi Luis,
> >
> > Thanks for this info. Can you send me the ‘istat’ output for dir1 and dir2? I’m curious to see how they are different.
> >
> > thanks,
> > brian
> >
> >
> > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
> > >
> > > Hi Brian,
> > >
> > > I have just confirmed fls shows the file path as /dir1/file1.txt based on your example.
> > >
> > > But in tsk_objects table the parent of file1.txt is dir2, which has the same meta_addr and meta_seq of dir1. I think the problem is in the parent cache map used in TskDbSqlite::storeObjId() and TskDbSqlite::findParObjId()
> > >
> > > I detected the issue in my cases searching for allocated files under unallocated folders, maybe you could find that in your test cases.
> > >
> > > Thank you very much for your time,
> > > Luis
> > >
> > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> > > Hi Brian,
> > >
> > > Your example illustrates the issue, except that file1.txt and dir1 are allocated and dir2 is unallocated.
> > >
> > > I will run fls on one image to check what it reports and post results.
> > >
> > > Thank you,
> > > Luis
> > >
> > > Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...> escreveu:
> > > Hi Luis,
> > >
> > > Can you provide a sanitized example of what you are seeing?  Is it that:
> > >
> > > - You have an unallocated file.  If you look at its parent_path and file name in tsk_file, then you get something like /dir1/ and file1.txt (i.e. /dir1/file1.txt)
> > > - When you use the DB / Autopsy, the file is actually shown as a child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
> > > - dir1 and dir2 have the same meta_add and meta_seq values in the database.
> > > - If you were to run ‘fls’ on the image, it would show the file as (/dir1/file1.txt).
> > >
> > > Is this correct?
> > >
> > > At some point in the past couple of years, we added a bunch of sequence checks because we were incorrectly mapping files to parent folders.  In theory, the sequence number is incremented each time an entry is reallocated, so it should be a reliable way of mapping it to the correct path.
> > >
> > > thanks,
> > > brian
> > >
> > >
> > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> > > >
> > > > Hi,
> > > >
> > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think Autopsy too) incorrectly decoding the file system tree of some NTFS images. I analysed one sqlite sent by him and tsk_loaddb is putting a number of (very important) files into a deleted directory with the same meta_addr and meta_seq of the true (not deleted) parent directory of those files, according to tsk_objects table. The parent_path of the child files are populated correctly into sqlite.
> > > >
> > > > Maybe the parent cache table logic used internally by tsk_loaddb should be updated to handle that situation (NTFS files with same meta_addr and meta_seq)?
> > > >
> > > > We will gladly provide any other information to help solve the problem.
> > > >
> > > > Thank you very much for your attention,
> > > > Luis
> > > >
> > > > ------------------------------------------------------------------------------
> > > > Transform Data into Opportunity.
> > > > Accelerate data analysis in your applications with
> > > > Intel Data Analytics Acceleration Library.
> > > > Click to learn more.
> > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > > sleuthkit-users mailing list
> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > > http://www.sleuthkit.org
> > >
> > >
> > > ------------------------------------------------------------------------------
> > > Transform Data into Opportunity.
> > > Accelerate data analysis in your applications with
> > > Intel Data Analytics Acceleration Library.
> > > Click to learn more.
> > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> >
> >
> > <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> > http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> 
> 
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications Manager
> Applications Manager provides deep performance insights into multiple tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi Brian,

I will gladly build and test the db_cache_uses_path branch and post results
here.

Thank you,
Luis

2016-05-01 21:32 GMT-03:00 Brian Carrier <ca...@sl...>:

> Hi Luis,
>
> I realized I forgot to circle back on this.  I committed a fix a while
> back to the db_cache_uses_path branch on github. It would be great if you
> could test that before we do another release. Can you build from github or
> should I send you and exe?
>
> thanks,
> brian
>
>
> > On Apr 12, 2016, at 1:14 PM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> >
> > Hi Brian,
> >
> > Yes, in my colleague's case, the deleted folder (dir2) name is "New
> Folder (2)" and the true parent folder has a user specific name, so I think
> some rename operation was done. But I am not sure if it was an unique
> rename that have caused the problem. In other cases it is not so clear to
> me, one has a Windows.old folder involved and looks like a system backup
> was done. In another case the false and true parent folders are at
> different levels in the FS tree.
> >
> > Thank you for your investigation,
> > Luis
> >
> > 2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:
> > Hi Luis,
> >
> > I want to understand how we got into this state.  Based on the names of
> your folders, is it likely that the deleted folder (“dir2” in our example
> below) was moved to be “dir1”?    That is the easiest explanation of why a
> deleted name and an allocated name would point to the same entry and have
> the same sequence.
> >
> > It still means there is an issue in the caching code that you pointed
> out, I just want to make sure the solution makes sense for this general
> problem.
> >
> > thanks,
> > brian
> >
> > > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> > >
> > > Hi Brian,
> > >
> > > I've attached the istat output for dir1. I do not know how to get
> istat result for dir2, because they have the same meta_addr. I searched the
> $MFT for dir2 filename and did not find it, I was able to find only dir1
> filename in $MFT exactly into the record at meta_addr offset.
> > >
> > > But I have found dir2 filename into $index_allocation ($I30) of its
> parent directory. It is attached too. Finally, dir2 is shown as (realloc)
> in fls output.
> > >
> > > Please let me know if you need any other info.
> > >
> > > Thank you very much,
> > > Luis
> > >
> > > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
> > > Hi Luis,
> > >
> > > Thanks for this info. Can you send me the ‘istat’ output for dir1 and
> dir2? I’m curious to see how they are different.
> > >
> > > thanks,
> > > brian
> > >
> > >
> > > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> > > >
> > > > Hi Brian,
> > > >
> > > > I have just confirmed fls shows the file path as /dir1/file1.txt
> based on your example.
> > > >
> > > > But in tsk_objects table the parent of file1.txt is dir2, which has
> the same meta_addr and meta_seq of dir1. I think the problem is in the
> parent cache map used in TskDbSqlite::storeObjId() and
> TskDbSqlite::findParObjId()
> > > >
> > > > I detected the issue in my cases searching for allocated files under
> unallocated folders, maybe you could find that in your test cases.
> > > >
> > > > Thank you very much for your time,
> > > > Luis
> > > >
> > > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> > > > Hi Brian,
> > > >
> > > > Your example illustrates the issue, except that file1.txt and dir1
> are allocated and dir2 is unallocated.
> > > >
> > > > I will run fls on one image to check what it reports and post
> results.
> > > >
> > > > Thank you,
> > > > Luis
> > > >
> > > > Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
> escreveu:
> > > > Hi Luis,
> > > >
> > > > Can you provide a sanitized example of what you are seeing?  Is it
> that:
> > > >
> > > > - You have an unallocated file.  If you look at its parent_path and
> file name in tsk_file, then you get something like /dir1/ and file1.txt
> (i.e. /dir1/file1.txt)
> > > > - When you use the DB / Autopsy, the file is actually shown as a
> child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
> > > > - dir1 and dir2 have the same meta_add and meta_seq values in the
> database.
> > > > - If you were to run ‘fls’ on the image, it would show the file as
> (/dir1/file1.txt).
> > > >
> > > > Is this correct?
> > > >
> > > > At some point in the past couple of years, we added a bunch of
> sequence checks because we were incorrectly mapping files to parent
> folders.  In theory, the sequence number is incremented each time an entry
> is reallocated, so it should be a reliable way of mapping it to the correct
> path.
> > > >
> > > > thanks,
> > > > brian
> > > >
> > > >
> > > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <
> lfc...@gm...> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think
> Autopsy too) incorrectly decoding the file system tree of some NTFS images.
> I analysed one sqlite sent by him and tsk_loaddb is putting a number of
> (very important) files into a deleted directory with the same meta_addr and
> meta_seq of the true (not deleted) parent directory of those files,
> according to tsk_objects table. The parent_path of the child files are
> populated correctly into sqlite.
> > > > >
> > > > > Maybe the parent cache table logic used internally by tsk_loaddb
> should be updated to handle that situation (NTFS files with same meta_addr
> and meta_seq)?
> > > > >
> > > > > We will gladly provide any other information to help solve the
> problem.
> > > > >
> > > > > Thank you very much for your attention,
> > > > > Luis
> > > > >
> > > > >
> ------------------------------------------------------------------------------
> > > > > Transform Data into Opportunity.
> > > > > Accelerate data analysis in your applications with
> > > > > Intel Data Analytics Acceleration Library.
> > > > > Click to learn more.
> > > > >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > > > sleuthkit-users mailing list
> > > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > > > http://www.sleuthkit.org
> > > >
> > > >
> > > >
> ------------------------------------------------------------------------------
> > > > Transform Data into Opportunity.
> > > > Accelerate data analysis in your applications with
> > > > Intel Data Analytics Acceleration Library.
> > > > Click to learn more.
> > > >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
> > > > sleuthkit-users mailing list
> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > > http://www.sleuthkit.org
> > >
> > >
> > >
> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
> > > Transform Data into Opportunity.
> > > Accelerate data analysis in your applications with
> > > Intel Data Analytics Acceleration Library.
> > > Click to learn more.
> > >
> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
> > > sleuthkit-users mailing list
> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > > http://www.sleuthkit.org
> >
> >
> >
> ------------------------------------------------------------------------------
> > Find and fix application performance issues faster with Applications
> Manager
> > Applications Manager provides deep performance insights into multiple
> tiers of
> > your business applications. It resolves application problems quickly and
> > reduces your MTTR. Get your free trial!
> >
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi,

I applied the patch locally and it resolved the problem. Tested with one of
my cases.

Thank you very much for all the attention!
Luis

2016-05-02 9:20 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi Brian,
>
> I will gladly build and test the db_cache_uses_path branch and post
> results here.
>
> Thank you,
> Luis
>
> 2016-05-01 21:32 GMT-03:00 Brian Carrier <ca...@sl...>:
>
>> Hi Luis,
>>
>> I realized I forgot to circle back on this.  I committed a fix a while
>> back to the db_cache_uses_path branch on github. It would be great if you
>> could test that before we do another release. Can you build from github or
>> should I send you and exe?
>>
>> thanks,
>> brian
>>
>>
>> > On Apr 12, 2016, at 1:14 PM, Luís Filipe Nassif <lfc...@gm...>
>> wrote:
>> >
>> > Hi Brian,
>> >
>> > Yes, in my colleague's case, the deleted folder (dir2) name is "New
>> Folder (2)" and the true parent folder has a user specific name, so I think
>> some rename operation was done. But I am not sure if it was an unique
>> rename that have caused the problem. In other cases it is not so clear to
>> me, one has a Windows.old folder involved and looks like a system backup
>> was done. In another case the false and true parent folders are at
>> different levels in the FS tree.
>> >
>> > Thank you for your investigation,
>> > Luis
>> >
>> > 2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:
>> > Hi Luis,
>> >
>> > I want to understand how we got into this state.  Based on the names of
>> your folders, is it likely that the deleted folder (“dir2” in our example
>> below) was moved to be “dir1”?    That is the easiest explanation of why a
>> deleted name and an allocated name would point to the same entry and have
>> the same sequence.
>> >
>> > It still means there is an issue in the caching code that you pointed
>> out, I just want to make sure the solution makes sense for this general
>> problem.
>> >
>> > thanks,
>> > brian
>> >
>> > > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <lfc...@gm...>
>> wrote:
>> > >
>> > > Hi Brian,
>> > >
>> > > I've attached the istat output for dir1. I do not know how to get
>> istat result for dir2, because they have the same meta_addr. I searched the
>> $MFT for dir2 filename and did not find it, I was able to find only dir1
>> filename in $MFT exactly into the record at meta_addr offset.
>> > >
>> > > But I have found dir2 filename into $index_allocation ($I30) of its
>> parent directory. It is attached too. Finally, dir2 is shown as (realloc)
>> in fls output.
>> > >
>> > > Please let me know if you need any other info.
>> > >
>> > > Thank you very much,
>> > > Luis
>> > >
>> > > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
>> > > Hi Luis,
>> > >
>> > > Thanks for this info. Can you send me the ‘istat’ output for dir1 and
>> dir2? I’m curious to see how they are different.
>> > >
>> > > thanks,
>> > > brian
>> > >
>> > >
>> > > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <
>> lfc...@gm...> wrote:
>> > > >
>> > > > Hi Brian,
>> > > >
>> > > > I have just confirmed fls shows the file path as /dir1/file1.txt
>> based on your example.
>> > > >
>> > > > But in tsk_objects table the parent of file1.txt is dir2, which has
>> the same meta_addr and meta_seq of dir1. I think the problem is in the
>> parent cache map used in TskDbSqlite::storeObjId() and
>> TskDbSqlite::findParObjId()
>> > > >
>> > > > I detected the issue in my cases searching for allocated files
>> under unallocated folders, maybe you could find that in your test cases.
>> > > >
>> > > > Thank you very much for your time,
>> > > > Luis
>> > > >
>> > > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>> > > > Hi Brian,
>> > > >
>> > > > Your example illustrates the issue, except that file1.txt and dir1
>> are allocated and dir2 is unallocated.
>> > > >
>> > > > I will run fls on one image to check what it reports and post
>> results.
>> > > >
>> > > > Thank you,
>> > > > Luis
>> > > >
>> > > > Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
>> escreveu:
>> > > > Hi Luis,
>> > > >
>> > > > Can you provide a sanitized example of what you are seeing?  Is it
>> that:
>> > > >
>> > > > - You have an unallocated file.  If you look at its parent_path and
>> file name in tsk_file, then you get something like /dir1/ and file1.txt
>> (i.e. /dir1/file1.txt)
>> > > > - When you use the DB / Autopsy, the file is actually shown as a
>> child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
>> > > > - dir1 and dir2 have the same meta_add and meta_seq values in the
>> database.
>> > > > - If you were to run ‘fls’ on the image, it would show the file as
>> (/dir1/file1.txt).
>> > > >
>> > > > Is this correct?
>> > > >
>> > > > At some point in the past couple of years, we added a bunch of
>> sequence checks because we were incorrectly mapping files to parent
>> folders.  In theory, the sequence number is incremented each time an entry
>> is reallocated, so it should be a reliable way of mapping it to the correct
>> path.
>> > > >
>> > > > thanks,
>> > > > brian
>> > > >
>> > > >
>> > > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <
>> lfc...@gm...> wrote:
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think
>> Autopsy too) incorrectly decoding the file system tree of some NTFS images.
>> I analysed one sqlite sent by him and tsk_loaddb is putting a number of
>> (very important) files into a deleted directory with the same meta_addr and
>> meta_seq of the true (not deleted) parent directory of those files,
>> according to tsk_objects table. The parent_path of the child files are
>> populated correctly into sqlite.
>> > > > >
>> > > > > Maybe the parent cache table logic used internally by tsk_loaddb
>> should be updated to handle that situation (NTFS files with same meta_addr
>> and meta_seq)?
>> > > > >
>> > > > > We will gladly provide any other information to help solve the
>> problem.
>> > > > >
>> > > > > Thank you very much for your attention,
>> > > > > Luis
>> > > > >
>> > > > >
>> ------------------------------------------------------------------------------
>> > > > > Transform Data into Opportunity.
>> > > > > Accelerate data analysis in your applications with
>> > > > > Intel Data Analytics Acceleration Library.
>> > > > > Click to learn more.
>> > > > >
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>> > > > > sleuthkit-users mailing list
>> > > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > > > > http://www.sleuthkit.org
>> > > >
>> > > >
>> > > >
>> ------------------------------------------------------------------------------
>> > > > Transform Data into Opportunity.
>> > > > Accelerate data analysis in your applications with
>> > > > Intel Data Analytics Acceleration Library.
>> > > > Click to learn more.
>> > > >
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>> > > > sleuthkit-users mailing list
>> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > > > http://www.sleuthkit.org
>> > >
>> > >
>> > >
>> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
>> > > Transform Data into Opportunity.
>> > > Accelerate data analysis in your applications with
>> > > Intel Data Analytics Acceleration Library.
>> > > Click to learn more.
>> > >
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
>> > > sleuthkit-users mailing list
>> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > > http://www.sleuthkit.org
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Find and fix application performance issues faster with Applications
>> Manager
>> > Applications Manager provides deep performance insights into multiple
>> tiers of
>> > your business applications. It resolves application problems quickly and
>> > reduces your MTTR. Get your free trial!
>> >
>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
>> > sleuthkit-users mailing list
>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > http://www.sleuthkit.org
>>
>>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi,

It is not related directly to this issue, but I discovered, based on
Microsoft docs (
https://msdn.microsoft.com/en-us/library/bb470124(v=vs.85).aspx), that the
sequence number is incremented when the file record is freed and not when
it is reused. I think that may affect the relationship between deleted
children and their parent, but I do not tried to confirm it.

Best,
Luis

2016-05-02 15:19 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi,
>
> I applied the patch locally and it resolved the problem. Tested with one
> of my cases.
>
> Thank you very much for all the attention!
> Luis
>
>
> 2016-05-02 9:20 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>
>> Hi Brian,
>>
>> I will gladly build and test the db_cache_uses_path branch and post
>> results here.
>>
>> Thank you,
>> Luis
>>
>> 2016-05-01 21:32 GMT-03:00 Brian Carrier <ca...@sl...>:
>>
>>> Hi Luis,
>>>
>>> I realized I forgot to circle back on this.  I committed a fix a while
>>> back to the db_cache_uses_path branch on github. It would be great if you
>>> could test that before we do another release. Can you build from github or
>>> should I send you and exe?
>>>
>>> thanks,
>>> brian
>>>
>>>
>>> > On Apr 12, 2016, at 1:14 PM, Luís Filipe Nassif <lfc...@gm...>
>>> wrote:
>>> >
>>> > Hi Brian,
>>> >
>>> > Yes, in my colleague's case, the deleted folder (dir2) name is "New
>>> Folder (2)" and the true parent folder has a user specific name, so I think
>>> some rename operation was done. But I am not sure if it was an unique
>>> rename that have caused the problem. In other cases it is not so clear to
>>> me, one has a Windows.old folder involved and looks like a system backup
>>> was done. In another case the false and true parent folders are at
>>> different levels in the FS tree.
>>> >
>>> > Thank you for your investigation,
>>> > Luis
>>> >
>>> > 2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:
>>> > Hi Luis,
>>> >
>>> > I want to understand how we got into this state.  Based on the names
>>> of your folders, is it likely that the deleted folder (“dir2” in our
>>> example below) was moved to be “dir1”?    That is the easiest explanation
>>> of why a deleted name and an allocated name would point to the same entry
>>> and have the same sequence.
>>> >
>>> > It still means there is an issue in the caching code that you pointed
>>> out, I just want to make sure the solution makes sense for this general
>>> problem.
>>> >
>>> > thanks,
>>> > brian
>>> >
>>> > > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <
>>> lfc...@gm...> wrote:
>>> > >
>>> > > Hi Brian,
>>> > >
>>> > > I've attached the istat output for dir1. I do not know how to get
>>> istat result for dir2, because they have the same meta_addr. I searched the
>>> $MFT for dir2 filename and did not find it, I was able to find only dir1
>>> filename in $MFT exactly into the record at meta_addr offset.
>>> > >
>>> > > But I have found dir2 filename into $index_allocation ($I30) of its
>>> parent directory. It is attached too. Finally, dir2 is shown as (realloc)
>>> in fls output.
>>> > >
>>> > > Please let me know if you need any other info.
>>> > >
>>> > > Thank you very much,
>>> > > Luis
>>> > >
>>> > > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
>>> > > Hi Luis,
>>> > >
>>> > > Thanks for this info. Can you send me the ‘istat’ output for dir1
>>> and dir2? I’m curious to see how they are different.
>>> > >
>>> > > thanks,
>>> > > brian
>>> > >
>>> > >
>>> > > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <
>>> lfc...@gm...> wrote:
>>> > > >
>>> > > > Hi Brian,
>>> > > >
>>> > > > I have just confirmed fls shows the file path as /dir1/file1.txt
>>> based on your example.
>>> > > >
>>> > > > But in tsk_objects table the parent of file1.txt is dir2, which
>>> has the same meta_addr and meta_seq of dir1. I think the problem is in the
>>> parent cache map used in TskDbSqlite::storeObjId() and
>>> TskDbSqlite::findParObjId()
>>> > > >
>>> > > > I detected the issue in my cases searching for allocated files
>>> under unallocated folders, maybe you could find that in your test cases.
>>> > > >
>>> > > > Thank you very much for your time,
>>> > > > Luis
>>> > > >
>>> > > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...
>>> >:
>>> > > > Hi Brian,
>>> > > >
>>> > > > Your example illustrates the issue, except that file1.txt and dir1
>>> are allocated and dir2 is unallocated.
>>> > > >
>>> > > > I will run fls on one image to check what it reports and post
>>> results.
>>> > > >
>>> > > > Thank you,
>>> > > > Luis
>>> > > >
>>> > > > Em 23 de mar de 2016 23:44, "Brian Carrier" <ca...@sl...>
>>> escreveu:
>>> > > > Hi Luis,
>>> > > >
>>> > > > Can you provide a sanitized example of what you are seeing?  Is it
>>> that:
>>> > > >
>>> > > > - You have an unallocated file.  If you look at its parent_path
>>> and file name in tsk_file, then you get something like /dir1/ and file1.txt
>>> (i.e. /dir1/file1.txt)
>>> > > > - When you use the DB / Autopsy, the file is actually shown as a
>>> child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
>>> > > > - dir1 and dir2 have the same meta_add and meta_seq values in the
>>> database.
>>> > > > - If you were to run ‘fls’ on the image, it would show the file as
>>> (/dir1/file1.txt).
>>> > > >
>>> > > > Is this correct?
>>> > > >
>>> > > > At some point in the past couple of years, we added a bunch of
>>> sequence checks because we were incorrectly mapping files to parent
>>> folders.  In theory, the sequence number is incremented each time an entry
>>> is reallocated, so it should be a reliable way of mapping it to the correct
>>> path.
>>> > > >
>>> > > > thanks,
>>> > > > brian
>>> > > >
>>> > > >
>>> > > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <
>>> lfc...@gm...> wrote:
>>> > > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think
>>> Autopsy too) incorrectly decoding the file system tree of some NTFS images.
>>> I analysed one sqlite sent by him and tsk_loaddb is putting a number of
>>> (very important) files into a deleted directory with the same meta_addr and
>>> meta_seq of the true (not deleted) parent directory of those files,
>>> according to tsk_objects table. The parent_path of the child files are
>>> populated correctly into sqlite.
>>> > > > >
>>> > > > > Maybe the parent cache table logic used internally by tsk_loaddb
>>> should be updated to handle that situation (NTFS files with same meta_addr
>>> and meta_seq)?
>>> > > > >
>>> > > > > We will gladly provide any other information to help solve the
>>> problem.
>>> > > > >
>>> > > > > Thank you very much for your attention,
>>> > > > > Luis
>>> > > > >
>>> > > > >
>>> ------------------------------------------------------------------------------
>>> > > > > Transform Data into Opportunity.
>>> > > > > Accelerate data analysis in your applications with
>>> > > > > Intel Data Analytics Acceleration Library.
>>> > > > > Click to learn more.
>>> > > > >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>> > > > > sleuthkit-users mailing list
>>> > > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > > > > http://www.sleuthkit.org
>>> > > >
>>> > > >
>>> > > >
>>> ------------------------------------------------------------------------------
>>> > > > Transform Data into Opportunity.
>>> > > > Accelerate data analysis in your applications with
>>> > > > Intel Data Analytics Acceleration Library.
>>> > > > Click to learn more.
>>> > > >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>> > > > sleuthkit-users mailing list
>>> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > > > http://www.sleuthkit.org
>>> > >
>>> > >
>>> > >
>>> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
>>> > > Transform Data into Opportunity.
>>> > > Accelerate data analysis in your applications with
>>> > > Intel Data Analytics Acceleration Library.
>>> > > Click to learn more.
>>> > >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
>>> > > sleuthkit-users mailing list
>>> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > > http://www.sleuthkit.org
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Find and fix application performance issues faster with Applications
>>> Manager
>>> > Applications Manager provides deep performance insights into multiple
>>> tiers of
>>> > your business applications. It resolves application problems quickly
>>> and
>>> > reduces your MTTR. Get your free trial!
>>> >
>>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
>>> > sleuthkit-users mailing list
>>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > http://www.sleuthkit.org
>>>
>>>
>>
>

Re: [sleuthkit-users] Tsk_loaddb generating an inconsistent file system tree

[Luís F. N. <lfc...@gm...>]

Hi,

The recent changes (from commit a725989 to 0432598) resolved the bug for my
2 problematic images and for my colleague's image, thank you all very much!
But with another image, the changes are causing a few files to be
discarded, returning the following message: "Error: Database Error
(TskDbSqlite::findParObjId: Error selecting file id by meta_addr: unknown
error (result code 101)"

For some reason the sql query did not find some parents in the database...
I don't know why (maybe the parents are not there?) Might those lost files
go into Orphan dir?

Thanks,
Luis

2016-06-06 18:09 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Hi,
>
> It is not related directly to this issue, but I discovered, based on
> Microsoft docs (
> https://msdn.microsoft.com/en-us/library/bb470124(v=vs.85).aspx), that
> the sequence number is incremented when the file record is freed and not
> when it is reused. I think that may affect the relationship between deleted
> children and their parent, but I do not tried to confirm it.
>
> Best,
> Luis
>
> 2016-05-02 15:19 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>
>> Hi,
>>
>> I applied the patch locally and it resolved the problem. Tested with one
>> of my cases.
>>
>> Thank you very much for all the attention!
>> Luis
>>
>>
>> 2016-05-02 9:20 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>>
>>> Hi Brian,
>>>
>>> I will gladly build and test the db_cache_uses_path branch and post
>>> results here.
>>>
>>> Thank you,
>>> Luis
>>>
>>> 2016-05-01 21:32 GMT-03:00 Brian Carrier <ca...@sl...>:
>>>
>>>> Hi Luis,
>>>>
>>>> I realized I forgot to circle back on this.  I committed a fix a while
>>>> back to the db_cache_uses_path branch on github. It would be great if you
>>>> could test that before we do another release. Can you build from github or
>>>> should I send you and exe?
>>>>
>>>> thanks,
>>>> brian
>>>>
>>>>
>>>> > On Apr 12, 2016, at 1:14 PM, Luís Filipe Nassif <lfc...@gm...>
>>>> wrote:
>>>> >
>>>> > Hi Brian,
>>>> >
>>>> > Yes, in my colleague's case, the deleted folder (dir2) name is "New
>>>> Folder (2)" and the true parent folder has a user specific name, so I think
>>>> some rename operation was done. But I am not sure if it was an unique
>>>> rename that have caused the problem. In other cases it is not so clear to
>>>> me, one has a Windows.old folder involved and looks like a system backup
>>>> was done. In another case the false and true parent folders are at
>>>> different levels in the FS tree.
>>>> >
>>>> > Thank you for your investigation,
>>>> > Luis
>>>> >
>>>> > 2016-04-11 22:38 GMT-03:00 Brian Carrier <ca...@sl...>:
>>>> > Hi Luis,
>>>> >
>>>> > I want to understand how we got into this state.  Based on the names
>>>> of your folders, is it likely that the deleted folder (“dir2” in our
>>>> example below) was moved to be “dir1”?    That is the easiest explanation
>>>> of why a deleted name and an allocated name would point to the same entry
>>>> and have the same sequence.
>>>> >
>>>> > It still means there is an issue in the caching code that you pointed
>>>> out, I just want to make sure the solution makes sense for this general
>>>> problem.
>>>> >
>>>> > thanks,
>>>> > brian
>>>> >
>>>> > > On Mar 29, 2016, at 10:14 AM, Luís Filipe Nassif <
>>>> lfc...@gm...> wrote:
>>>> > >
>>>> > > Hi Brian,
>>>> > >
>>>> > > I've attached the istat output for dir1. I do not know how to get
>>>> istat result for dir2, because they have the same meta_addr. I searched the
>>>> $MFT for dir2 filename and did not find it, I was able to find only dir1
>>>> filename in $MFT exactly into the record at meta_addr offset.
>>>> > >
>>>> > > But I have found dir2 filename into $index_allocation ($I30) of its
>>>> parent directory. It is attached too. Finally, dir2 is shown as (realloc)
>>>> in fls output.
>>>> > >
>>>> > > Please let me know if you need any other info.
>>>> > >
>>>> > > Thank you very much,
>>>> > > Luis
>>>> > >
>>>> > > 2016-03-28 22:59 GMT-03:00 Brian Carrier <ca...@sl...>:
>>>> > > Hi Luis,
>>>> > >
>>>> > > Thanks for this info. Can you send me the ‘istat’ output for dir1
>>>> and dir2? I’m curious to see how they are different.
>>>> > >
>>>> > > thanks,
>>>> > > brian
>>>> > >
>>>> > >
>>>> > > > On Mar 24, 2016, at 9:29 AM, Luís Filipe Nassif <
>>>> lfc...@gm...> wrote:
>>>> > > >
>>>> > > > Hi Brian,
>>>> > > >
>>>> > > > I have just confirmed fls shows the file path as /dir1/file1.txt
>>>> based on your example.
>>>> > > >
>>>> > > > But in tsk_objects table the parent of file1.txt is dir2, which
>>>> has the same meta_addr and meta_seq of dir1. I think the problem is in the
>>>> parent cache map used in TskDbSqlite::storeObjId() and
>>>> TskDbSqlite::findParObjId()
>>>> > > >
>>>> > > > I detected the issue in my cases searching for allocated files
>>>> under unallocated folders, maybe you could find that in your test cases.
>>>> > > >
>>>> > > > Thank you very much for your time,
>>>> > > > Luis
>>>> > > >
>>>> > > > 2016-03-24 0:11 GMT-03:00 Luís Filipe Nassif <lfc...@gm...
>>>> >:
>>>> > > > Hi Brian,
>>>> > > >
>>>> > > > Your example illustrates the issue, except that file1.txt and
>>>> dir1 are allocated and dir2 is unallocated.
>>>> > > >
>>>> > > > I will run fls on one image to check what it reports and post
>>>> results.
>>>> > > >
>>>> > > > Thank you,
>>>> > > > Luis
>>>> > > >
>>>> > > > Em 23 de mar de 2016 23:44, "Brian Carrier" <
>>>> ca...@sl...> escreveu:
>>>> > > > Hi Luis,
>>>> > > >
>>>> > > > Can you provide a sanitized example of what you are seeing?  Is
>>>> it that:
>>>> > > >
>>>> > > > - You have an unallocated file.  If you look at its parent_path
>>>> and file name in tsk_file, then you get something like /dir1/ and file1.txt
>>>> (i.e. /dir1/file1.txt)
>>>> > > > - When you use the DB / Autopsy, the file is actually shown as a
>>>> child of another directory, dir2 for example.  (i.e. /dir2/file1.txt).
>>>> > > > - dir1 and dir2 have the same meta_add and meta_seq values in the
>>>> database.
>>>> > > > - If you were to run ‘fls’ on the image, it would show the file
>>>> as (/dir1/file1.txt).
>>>> > > >
>>>> > > > Is this correct?
>>>> > > >
>>>> > > > At some point in the past couple of years, we added a bunch of
>>>> sequence checks because we were incorrectly mapping files to parent
>>>> folders.  In theory, the sequence number is incremented each time an entry
>>>> is reallocated, so it should be a reliable way of mapping it to the correct
>>>> path.
>>>> > > >
>>>> > > > thanks,
>>>> > > > brian
>>>> > > >
>>>> > > >
>>>> > > > > On Mar 22, 2016, at 12:10 PM, Luís Filipe Nassif <
>>>> lfc...@gm...> wrote:
>>>> > > > >
>>>> > > > > Hi,
>>>> > > > >
>>>> > > > > A colleague of mine have observed tsk_loaddb 4.2 (so I think
>>>> Autopsy too) incorrectly decoding the file system tree of some NTFS images.
>>>> I analysed one sqlite sent by him and tsk_loaddb is putting a number of
>>>> (very important) files into a deleted directory with the same meta_addr and
>>>> meta_seq of the true (not deleted) parent directory of those files,
>>>> according to tsk_objects table. The parent_path of the child files are
>>>> populated correctly into sqlite.
>>>> > > > >
>>>> > > > > Maybe the parent cache table logic used internally by
>>>> tsk_loaddb should be updated to handle that situation (NTFS files with same
>>>> meta_addr and meta_seq)?
>>>> > > > >
>>>> > > > > We will gladly provide any other information to help solve the
>>>> problem.
>>>> > > > >
>>>> > > > > Thank you very much for your attention,
>>>> > > > > Luis
>>>> > > > >
>>>> > > > >
>>>> ------------------------------------------------------------------------------
>>>> > > > > Transform Data into Opportunity.
>>>> > > > > Accelerate data analysis in your applications with
>>>> > > > > Intel Data Analytics Acceleration Library.
>>>> > > > > Click to learn more.
>>>> > > > >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>>> > > > > sleuthkit-users mailing list
>>>> > > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > > > > http://www.sleuthkit.org
>>>> > > >
>>>> > > >
>>>> > > >
>>>> ------------------------------------------------------------------------------
>>>> > > > Transform Data into Opportunity.
>>>> > > > Accelerate data analysis in your applications with
>>>> > > > Intel Data Analytics Acceleration Library.
>>>> > > > Click to learn more.
>>>> > > >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140_______________________________________________
>>>> > > > sleuthkit-users mailing list
>>>> > > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > > > http://www.sleuthkit.org
>>>> > >
>>>> > >
>>>> > >
>>>> <istat_of_dir1.txt><$I30_of_dir2_parent>------------------------------------------------------------------------------
>>>> > > Transform Data into Opportunity.
>>>> > > Accelerate data analysis in your applications with
>>>> > > Intel Data Analytics Acceleration Library.
>>>> > > Click to learn more.
>>>> > >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140_______________________________________________
>>>> > > sleuthkit-users mailing list
>>>> > > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > > http://www.sleuthkit.org
>>>> >
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Find and fix application performance issues faster with Applications
>>>> Manager
>>>> > Applications Manager provides deep performance insights into multiple
>>>> tiers of
>>>> > your business applications. It resolves application problems quickly
>>>> and
>>>> > reduces your MTTR. Get your free trial!
>>>> >
>>>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
>>>> > sleuthkit-users mailing list
>>>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>>> > http://www.sleuthkit.org
>>>>
>>>>
>>>
>>
>