sleuthkit-users — List to discuss Autopsy and The Sleuth Kit.

Re: [sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

Fixing my last email, the test was run with the indexes AND Brian's fix.
Then I removed the index patch and loadDb took the same 1 hour to finish
with only Brian's fix. So the index patch did not help improving database
look up for parent_id.

Sorry for mistake,
Nassif


2014-05-02 10:54 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> I tested loadDb with a create index on meta_addr and fs_obj_id patch. The
> image with 433.321 files, previously taking 2h45min to load, now takes 1h
> to finish loadDb with the indexes. That is a good speed up, but completely
> disabling the database parent_id look up, it only takes 7min to finish. Is
> there another thing we can do to improve the parent_id database look up?
>
> Regards,
> Nassif
>
>
> 2014-05-02 9:35 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>
> Ok, tested in 2 images. Fix resolved a lot of misses:
>>
>> ntfs image w/ 127.408 files: from 19.558 to 6.511 misses
>> ntfs image w/ 433.321 files: from 182.256 to 19.908 misses
>>
>> I also think creating an index on tsk_files(meta_addr) and
>> tsk_files(fs_obj_id) could help improving the database look up for those
>> deleted files not found in local cache, what do you think? The database
>> look up seems too slow, as described in my first email.
>>
>> Thank you for taking a look so quickly.
>> Nassif
>>
>>
>> 2014-05-01 23:47 GMT-03:00 Brian Carrier <ca...@sl...>:
>>
>> Well that was an easy and embarrassing fix:
>>>
>>>      if (TSK_FS_TYPE_ISNTFS(fs_file->fs_info->ftype)) {
>>> -        seq = fs_file->name->meta_seq;
>>> +        seq = fs_file->name->par_seq;
>>>      }
>>>
>>> Turns out we've been having  a lot of cache misses because of this
>>> stupid bug. Can you replace that line and see if it helps.  It certainly
>>> did on my test image.
>>>
>>> thanks,
>>> brian
>>>
>>>
>>> On May 1, 2014, at 10:24 PM, Brian Carrier <ca...@sl...>
>>> wrote:
>>>
>>> > Thanks for the tests.  I wonder if it has to do with an incorrect
>>> sequence number. NTFS increments the sequence number each time a file is
>>> re-allocated. Deleted orphan files could be getting misses.  I'll add some
>>> logging on my system and see what kind of misses I get.
>>> >
>>> > brian
>>> >
>>> > On May 1, 2014, at 8:39 PM, Luís Filipe Nassif <lfc...@gm...>
>>> wrote:
>>> >
>>> >> Ok, tests 1 and 3 done. I do not have sleuthkit code inside an ide,
>>> so did not use breakpoints. Instead, I changed TskDbSqlite::findParObjId()
>>> to return the parent_meta_addr when it is not found and return 1 when it is
>>> found in the cache map.
>>> >>
>>> >> Performing queries on the generated sqlite, there were 19.558 cache
>>> misses from an image with 3 ntfs partitions and 127.408 files. I confirmed
>>> that many parent_meta_addr missed from cache (now stored in
>>> tsk_objects.par_obj_id) are into tsk_files.meta_addr. The complete paths
>>> corresponding to these meta_addr are parents of those files whose
>>> processing have not found them in cache.
>>> >>
>>> >> Other tests resulted in:
>>> >> 182.256 cache misses from 433.321 files (ntfs)
>>> >> 892.359 misses from 1.811.393 files (ntfs)
>>> >> 169.819 misses from 3.177.917 files (hfs)
>>> >>
>>> >> Luis Nassif
>>> >>
>>> >>
>>> >>
>>> >> 2014-05-01 16:14 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>>> >> Forgot to mention: we are using sleuthkit 4.1.3
>>> >>
>>> >> Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...>
>>> escreveu:
>>> >>
>>> >> Hi Brian,
>>> >>
>>> >> The 3 cases above were ntfs. I also tested with hfs and canceled
>>> loaddb after 1 day. The modified version finished after 8hours and added
>>> about 3 million entries. We will try to do the tests you have suggested.
>>> >>
>>> >> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...>
>>> escreveu:
>>> >> Hi Luis,
>>> >>
>>> >> What kind of file system was it? I fixed a bug a little while ago in
>>> that code for HFS file systems that resulted in a lot of cache misses.
>>> >>
>>> >> In theory, everything should be cached.  It sounds like a bug if you
>>> are getting so many misses.  The basic idea of this code is that everything
>>> in the DB gets assigned a unique object ID and we make associations between
>>> files and their parent folder's unique ID.
>>> >>
>>> >> Since you seem to be comfortable with a debugger in the code, can you
>>> set a breakpoint for when the miss happens and:
>>> >> 1) Determine the path of the file that was being added to the DB and
>>> the parent address that was trying to be found.
>>> >> 2) Use the 'ffind' TSK tool to then map that parent address to a
>>> path.  Is it a subset of the path from #1?
>>> >> 3) Open the DB in a SQLite tool and do something like this:
>>> >>
>>> >> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
>>> >>
>>> >> Is it in the DB?
>>> >>
>>> >> Thanks!
>>> >>
>>> >> brian
>>> >>
>>> >>
>>> >> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...>
>>> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> We have investigated a bit why the add image process is too slow in
>>> some cases. The add image process time seems to be quadratic with the
>>> number of files in the image.
>>> >>>
>>> >>> We detected that the function TskDbSqlite::findParObjId(), in
>>> db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
>>> mapping in the local cache for a lot of files, causing it to search for the
>>> mapping in the database (not sure if it is an non-indexed search?)
>>> >>>
>>> >>> For testing purposes, we added a "return 1;" line right after the
>>> cache look up, disabling the database look up, and this resulted in great
>>> speed ups:
>>> >>>
>>> >>> number of files / default load_db time / patched load_db time
>>> >>> ~80.000 / 20min / 2min
>>> >>> ~300.000 / 3h / 7min
>>> >>> ~700.000 / 48h / 27min
>>> >>>
>>> >>> We wonder if it is possible to store all par_meta_addr -> par_id
>>> mappings into local cache (better) or doing an improved (indexed?) search
>>> for the mapping in the database. We think that someone with more knowledge
>>> of load_db code could help a lot here.
>>> >>>
>>> ------------------------------------------------------------------------------
>>> >>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For
>>> FREE
>>> >>> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>>> >>> unparalleled scalability from the best Selenium testing platform
>>> available.
>>> >>> Simple to use. Nothing to install. Get started now for free."
>>> >>>
>>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>>> >>> sleuthkit-users mailing list
>>> >>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> >>> http://www.sleuthkit.org
>>> >>
>>> >>
>>> >>
>>> ------------------------------------------------------------------------------
>>> >> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>>> >> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>>> >> unparalleled scalability from the best Selenium testing platform
>>> available.
>>> >> Simple to use. Nothing to install. Get started now for free."
>>> >>
>>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>>> >> sleuthkit-users mailing list
>>> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> >> http://www.sleuthkit.org
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>>> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>>> > unparalleled scalability from the best Selenium testing platform
>>> available.
>>> > Simple to use. Nothing to install. Get started now for free."
>>> > http://p.sf.net/sfu/SauceLabs
>>> > _______________________________________________
>>> > sleuthkit-users mailing list
>>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > http://www.sleuthkit.org
>>>
>>>
>>
>

Re: [sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

I tested loadDb with a create index on meta_addr and fs_obj_id patch. The
image with 433.321 files, previously taking 2h45min to load, now takes 1h
to finish loadDb with the indexes. That is a good speed up, but completely
disabling the database parent_id look up, it only takes 7min to finish. Is
there another thing we can do to improve the parent_id database look up?

Regards,
Nassif


2014-05-02 9:35 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Ok, tested in 2 images. Fix resolved a lot of misses:
>
> ntfs image w/ 127.408 files: from 19.558 to 6.511 misses
> ntfs image w/ 433.321 files: from 182.256 to 19.908 misses
>
> I also think creating an index on tsk_files(meta_addr) and
> tsk_files(fs_obj_id) could help improving the database look up for those
> deleted files not found in local cache, what do you think? The database
> look up seems too slow, as described in my first email.
>
> Thank you for taking a look so quickly.
> Nassif
>
>
> 2014-05-01 23:47 GMT-03:00 Brian Carrier <ca...@sl...>:
>
> Well that was an easy and embarrassing fix:
>>
>>      if (TSK_FS_TYPE_ISNTFS(fs_file->fs_info->ftype)) {
>> -        seq = fs_file->name->meta_seq;
>> +        seq = fs_file->name->par_seq;
>>      }
>>
>> Turns out we've been having  a lot of cache misses because of this stupid
>> bug. Can you replace that line and see if it helps.  It certainly did on my
>> test image.
>>
>> thanks,
>> brian
>>
>>
>> On May 1, 2014, at 10:24 PM, Brian Carrier <ca...@sl...> wrote:
>>
>> > Thanks for the tests.  I wonder if it has to do with an incorrect
>> sequence number. NTFS increments the sequence number each time a file is
>> re-allocated. Deleted orphan files could be getting misses.  I'll add some
>> logging on my system and see what kind of misses I get.
>> >
>> > brian
>> >
>> > On May 1, 2014, at 8:39 PM, Luís Filipe Nassif <lfc...@gm...>
>> wrote:
>> >
>> >> Ok, tests 1 and 3 done. I do not have sleuthkit code inside an ide, so
>> did not use breakpoints. Instead, I changed TskDbSqlite::findParObjId() to
>> return the parent_meta_addr when it is not found and return 1 when it is
>> found in the cache map.
>> >>
>> >> Performing queries on the generated sqlite, there were 19.558 cache
>> misses from an image with 3 ntfs partitions and 127.408 files. I confirmed
>> that many parent_meta_addr missed from cache (now stored in
>> tsk_objects.par_obj_id) are into tsk_files.meta_addr. The complete paths
>> corresponding to these meta_addr are parents of those files whose
>> processing have not found them in cache.
>> >>
>> >> Other tests resulted in:
>> >> 182.256 cache misses from 433.321 files (ntfs)
>> >> 892.359 misses from 1.811.393 files (ntfs)
>> >> 169.819 misses from 3.177.917 files (hfs)
>> >>
>> >> Luis Nassif
>> >>
>> >>
>> >>
>> >> 2014-05-01 16:14 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>> >> Forgot to mention: we are using sleuthkit 4.1.3
>> >>
>> >> Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...>
>> escreveu:
>> >>
>> >> Hi Brian,
>> >>
>> >> The 3 cases above were ntfs. I also tested with hfs and canceled
>> loaddb after 1 day. The modified version finished after 8hours and added
>> about 3 million entries. We will try to do the tests you have suggested.
>> >>
>> >> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:
>> >> Hi Luis,
>> >>
>> >> What kind of file system was it? I fixed a bug a little while ago in
>> that code for HFS file systems that resulted in a lot of cache misses.
>> >>
>> >> In theory, everything should be cached.  It sounds like a bug if you
>> are getting so many misses.  The basic idea of this code is that everything
>> in the DB gets assigned a unique object ID and we make associations between
>> files and their parent folder's unique ID.
>> >>
>> >> Since you seem to be comfortable with a debugger in the code, can you
>> set a breakpoint for when the miss happens and:
>> >> 1) Determine the path of the file that was being added to the DB and
>> the parent address that was trying to be found.
>> >> 2) Use the 'ffind' TSK tool to then map that parent address to a path.
>>  Is it a subset of the path from #1?
>> >> 3) Open the DB in a SQLite tool and do something like this:
>> >>
>> >> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
>> >>
>> >> Is it in the DB?
>> >>
>> >> Thanks!
>> >>
>> >> brian
>> >>
>> >>
>> >> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> We have investigated a bit why the add image process is too slow in
>> some cases. The add image process time seems to be quadratic with the
>> number of files in the image.
>> >>>
>> >>> We detected that the function TskDbSqlite::findParObjId(), in
>> db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
>> mapping in the local cache for a lot of files, causing it to search for the
>> mapping in the database (not sure if it is an non-indexed search?)
>> >>>
>> >>> For testing purposes, we added a "return 1;" line right after the
>> cache look up, disabling the database look up, and this resulted in great
>> speed ups:
>> >>>
>> >>> number of files / default load_db time / patched load_db time
>> >>> ~80.000 / 20min / 2min
>> >>> ~300.000 / 3h / 7min
>> >>> ~700.000 / 48h / 27min
>> >>>
>> >>> We wonder if it is possible to store all par_meta_addr -> par_id
>> mappings into local cache (better) or doing an improved (indexed?) search
>> for the mapping in the database. We think that someone with more knowledge
>> of load_db code could help a lot here.
>> >>>
>> ------------------------------------------------------------------------------
>> >>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> >>> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>> >>> unparalleled scalability from the best Selenium testing platform
>> available.
>> >>> Simple to use. Nothing to install. Get started now for free."
>> >>>
>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>> >>> sleuthkit-users mailing list
>> >>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> >>> http://www.sleuthkit.org
>> >>
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> >> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>> >> unparalleled scalability from the best Selenium testing platform
>> available.
>> >> Simple to use. Nothing to install. Get started now for free."
>> >>
>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>> >> sleuthkit-users mailing list
>> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> >> http://www.sleuthkit.org
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>> > unparalleled scalability from the best Selenium testing platform
>> available.
>> > Simple to use. Nothing to install. Get started now for free."
>> > http://p.sf.net/sfu/SauceLabs
>> > _______________________________________________
>> > sleuthkit-users mailing list
>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > http://www.sleuthkit.org
>>
>>
>

[sleuthkit-users] a question: sleuthkit-users

[Conley, T. <tom...@us...>]

Is there a way to search through the undeleted items for keyword file names?
My problem is that there are over 100,000 and I can only see 10k of them.

Some noob 

Re: [sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

Ok, tested in 2 images. Fix resolved a lot of misses:

ntfs image w/ 127.408 files: from 19.558 to 6.511 misses
ntfs image w/ 433.321 files: from 182.256 to 19.908 misses

I also think creating an index on tsk_files(meta_addr) and
tsk_files(fs_obj_id) could help improving the database look up for those
deleted files not found in local cache, what do you think? The database
look up seems too slow, as described in my first email.

Thank you for taking a look so quickly.
Nassif


2014-05-01 23:47 GMT-03:00 Brian Carrier <ca...@sl...>:

> Well that was an easy and embarrassing fix:
>
>      if (TSK_FS_TYPE_ISNTFS(fs_file->fs_info->ftype)) {
> -        seq = fs_file->name->meta_seq;
> +        seq = fs_file->name->par_seq;
>      }
>
> Turns out we've been having  a lot of cache misses because of this stupid
> bug. Can you replace that line and see if it helps.  It certainly did on my
> test image.
>
> thanks,
> brian
>
>
> On May 1, 2014, at 10:24 PM, Brian Carrier <ca...@sl...> wrote:
>
> > Thanks for the tests.  I wonder if it has to do with an incorrect
> sequence number. NTFS increments the sequence number each time a file is
> re-allocated. Deleted orphan files could be getting misses.  I'll add some
> logging on my system and see what kind of misses I get.
> >
> > brian
> >
> > On May 1, 2014, at 8:39 PM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> >
> >> Ok, tests 1 and 3 done. I do not have sleuthkit code inside an ide, so
> did not use breakpoints. Instead, I changed TskDbSqlite::findParObjId() to
> return the parent_meta_addr when it is not found and return 1 when it is
> found in the cache map.
> >>
> >> Performing queries on the generated sqlite, there were 19.558 cache
> misses from an image with 3 ntfs partitions and 127.408 files. I confirmed
> that many parent_meta_addr missed from cache (now stored in
> tsk_objects.par_obj_id) are into tsk_files.meta_addr. The complete paths
> corresponding to these meta_addr are parents of those files whose
> processing have not found them in cache.
> >>
> >> Other tests resulted in:
> >> 182.256 cache misses from 433.321 files (ntfs)
> >> 892.359 misses from 1.811.393 files (ntfs)
> >> 169.819 misses from 3.177.917 files (hfs)
> >>
> >> Luis Nassif
> >>
> >>
> >>
> >> 2014-05-01 16:14 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> >> Forgot to mention: we are using sleuthkit 4.1.3
> >>
> >> Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...>
> escreveu:
> >>
> >> Hi Brian,
> >>
> >> The 3 cases above were ntfs. I also tested with hfs and canceled loaddb
> after 1 day. The modified version finished after 8hours and added about 3
> million entries. We will try to do the tests you have suggested.
> >>
> >> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:
> >> Hi Luis,
> >>
> >> What kind of file system was it? I fixed a bug a little while ago in
> that code for HFS file systems that resulted in a lot of cache misses.
> >>
> >> In theory, everything should be cached.  It sounds like a bug if you
> are getting so many misses.  The basic idea of this code is that everything
> in the DB gets assigned a unique object ID and we make associations between
> files and their parent folder's unique ID.
> >>
> >> Since you seem to be comfortable with a debugger in the code, can you
> set a breakpoint for when the miss happens and:
> >> 1) Determine the path of the file that was being added to the DB and
> the parent address that was trying to be found.
> >> 2) Use the 'ffind' TSK tool to then map that parent address to a path.
>  Is it a subset of the path from #1?
> >> 3) Open the DB in a SQLite tool and do something like this:
> >>
> >> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
> >>
> >> Is it in the DB?
> >>
> >> Thanks!
> >>
> >> brian
> >>
> >>
> >> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> We have investigated a bit why the add image process is too slow in
> some cases. The add image process time seems to be quadratic with the
> number of files in the image.
> >>>
> >>> We detected that the function TskDbSqlite::findParObjId(), in
> db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
> mapping in the local cache for a lot of files, causing it to search for the
> mapping in the database (not sure if it is an non-indexed search?)
> >>>
> >>> For testing purposes, we added a "return 1;" line right after the
> cache look up, disabling the database look up, and this resulted in great
> speed ups:
> >>>
> >>> number of files / default load_db time / patched load_db time
> >>> ~80.000 / 20min / 2min
> >>> ~300.000 / 3h / 7min
> >>> ~700.000 / 48h / 27min
> >>>
> >>> We wonder if it is possible to store all par_meta_addr -> par_id
> mappings into local cache (better) or doing an improved (indexed?) search
> for the mapping in the database. We think that someone with more knowledge
> of load_db code could help a lot here.
> >>>
> ------------------------------------------------------------------------------
> >>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> >>> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> >>> unparalleled scalability from the best Selenium testing platform
> available.
> >>> Simple to use. Nothing to install. Get started now for free."
> >>>
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> >>> sleuthkit-users mailing list
> >>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >>> http://www.sleuthkit.org
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> >> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> >> unparalleled scalability from the best Selenium testing platform
> available.
> >> Simple to use. Nothing to install. Get started now for free."
> >>
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> >> sleuthkit-users mailing list
> >> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> >> http://www.sleuthkit.org
> >
> >
> >
> ------------------------------------------------------------------------------
> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> > unparalleled scalability from the best Selenium testing platform
> available.
> > Simple to use. Nothing to install. Get started now for free."
> > http://p.sf.net/sfu/SauceLabs
> > _______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] failure to open autopsy.log

[Kalin K. <me....@gm...>]

On May 2, 2014 6:56 PM, <al...@ma...> wrote:
> Both were downloaded from the Ubuntu software repository and installed
> correctly, in the order SK then autopsy.
>
> I ran Autopsy from the Terminal, and got the message about opening an HTML
> session but then I got the error message (immediately):
>
> "Cannot open log: autopsy.log at /usr/share/autopsy/lib//Print.pm at line
> 383."
>
Most probably permission problem, most probably in /tmp ...

What is the output of `ls -lsd /tmp` ?

Note to devs: The message need to be changed to include the full path IMHO.

Kalin.

[sleuthkit-users] failure to open autopsy.log

[<al...@ma...>]

Dear users,

I need some assistance please.

I have installed autopsy and Sk for the first time yesterday to look at my
disk on my laptop.

Both were downloaded from the Ubuntu software repository and installed
correctly, in the order SK then autopsy.

I ran Autopsy from the Terminal, and got the message about opening an HTML
session but then I got the error message (immediately):

"Cannot open log: autopsy.log at /usr/share/autopsy/lib//Print.pm at line
383."

(Note that the number of '/' characters are exactly as per the error message.

No mention is made of which module is failing.

I am running the latest Autopsy and SK as per the ppa repository and am
running Ubuntu 12.04 LTS.

Anyone able to assist here?

many thanks
Alan Brown

Re: [sleuthkit-users] Slow Add Image Process Cause

[Brian C. <ca...@sl...>]

Well that was an easy and embarrassing fix:

     if (TSK_FS_TYPE_ISNTFS(fs_file->fs_info->ftype)) {
-        seq = fs_file->name->meta_seq;
+        seq = fs_file->name->par_seq;
     }

Turns out we've been having  a lot of cache misses because of this stupid bug. Can you replace that line and see if it helps.  It certainly did on my test image. 

thanks,
brian


On May 1, 2014, at 10:24 PM, Brian Carrier <ca...@sl...> wrote:

> Thanks for the tests.  I wonder if it has to do with an incorrect sequence number. NTFS increments the sequence number each time a file is re-allocated. Deleted orphan files could be getting misses.  I'll add some logging on my system and see what kind of misses I get.
> 
> brian
> 
> On May 1, 2014, at 8:39 PM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
>> Ok, tests 1 and 3 done. I do not have sleuthkit code inside an ide, so did not use breakpoints. Instead, I changed TskDbSqlite::findParObjId() to return the parent_meta_addr when it is not found and return 1 when it is found in the cache map.
>> 
>> Performing queries on the generated sqlite, there were 19.558 cache misses from an image with 3 ntfs partitions and 127.408 files. I confirmed that many parent_meta_addr missed from cache (now stored in tsk_objects.par_obj_id) are into tsk_files.meta_addr. The complete paths corresponding to these meta_addr are parents of those files whose processing have not found them in cache.
>> 
>> Other tests resulted in:
>> 182.256 cache misses from 433.321 files (ntfs)
>> 892.359 misses from 1.811.393 files (ntfs)
>> 169.819 misses from 3.177.917 files (hfs)
>> 
>> Luis Nassif
>> 
>> 
>> 
>> 2014-05-01 16:14 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
>> Forgot to mention: we are using sleuthkit 4.1.3
>> 
>> Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...> escreveu:
>> 
>> Hi Brian,
>> 
>> The 3 cases above were ntfs. I also tested with hfs and canceled loaddb after 1 day. The modified version finished after 8hours and added about 3 million entries. We will try to do the tests you have suggested.
>> 
>> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:
>> Hi Luis,
>> 
>> What kind of file system was it? I fixed a bug a little while ago in that code for HFS file systems that resulted in a lot of cache misses.
>> 
>> In theory, everything should be cached.  It sounds like a bug if you are getting so many misses.  The basic idea of this code is that everything in the DB gets assigned a unique object ID and we make associations between files and their parent folder's unique ID.
>> 
>> Since you seem to be comfortable with a debugger in the code, can you set a breakpoint for when the miss happens and:
>> 1) Determine the path of the file that was being added to the DB and the parent address that was trying to be found.
>> 2) Use the 'ffind' TSK tool to then map that parent address to a path.  Is it a subset of the path from #1?
>> 3) Open the DB in a SQLite tool and do something like this:
>> 
>> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
>> 
>> Is it in the DB?
>> 
>> Thanks!
>> 
>> brian
>> 
>> 
>> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
>> 
>>> Hi,
>>> 
>>> We have investigated a bit why the add image process is too slow in some cases. The add image process time seems to be quadratic with the number of files in the image.
>>> 
>>> We detected that the function TskDbSqlite::findParObjId(), in db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id mapping in the local cache for a lot of files, causing it to search for the mapping in the database (not sure if it is an non-indexed search?)
>>> 
>>> For testing purposes, we added a "return 1;" line right after the cache look up, disabling the database look up, and this resulted in great speed ups:
>>> 
>>> number of files / default load_db time / patched load_db time
>>> ~80.000 / 20min / 2min
>>> ~300.000 / 3h / 7min
>>> ~700.000 / 48h / 27min
>>> 
>>> We wonder if it is possible to store all par_meta_addr -> par_id mappings into local cache (better) or doing an improved (indexed?) search for the mapping in the database. We think that someone with more knowledge of load_db code could help a lot here.
>>> ------------------------------------------------------------------------------
>>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>>> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>>> unparalleled scalability from the best Selenium testing platform available.
>>> Simple to use. Nothing to install. Get started now for free."
>>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>>> sleuthkit-users mailing list
>>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> http://www.sleuthkit.org
>> 
>> 
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
>> unparalleled scalability from the best Selenium testing platform available.
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>> sleuthkit-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> http://www.sleuthkit.org
> 
> 
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Slow Add Image Process Cause

[Brian C. <ca...@sl...>]

Thanks for the tests.  I wonder if it has to do with an incorrect sequence number. NTFS increments the sequence number each time a file is re-allocated. Deleted orphan files could be getting misses.  I'll add some logging on my system and see what kind of misses I get.

brian

On May 1, 2014, at 8:39 PM, Luís Filipe Nassif <lfc...@gm...> wrote:

> Ok, tests 1 and 3 done. I do not have sleuthkit code inside an ide, so did not use breakpoints. Instead, I changed TskDbSqlite::findParObjId() to return the parent_meta_addr when it is not found and return 1 when it is found in the cache map.
> 
> Performing queries on the generated sqlite, there were 19.558 cache misses from an image with 3 ntfs partitions and 127.408 files. I confirmed that many parent_meta_addr missed from cache (now stored in tsk_objects.par_obj_id) are into tsk_files.meta_addr. The complete paths corresponding to these meta_addr are parents of those files whose processing have not found them in cache.
> 
> Other tests resulted in:
> 182.256 cache misses from 433.321 files (ntfs)
> 892.359 misses from 1.811.393 files (ntfs)
> 169.819 misses from 3.177.917 files (hfs)
> 
> Luis Nassif
> 
> 
> 
> 2014-05-01 16:14 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:
> Forgot to mention: we are using sleuthkit 4.1.3
> 
> Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...> escreveu:
> 
> Hi Brian,
> 
> The 3 cases above were ntfs. I also tested with hfs and canceled loaddb after 1 day. The modified version finished after 8hours and added about 3 million entries. We will try to do the tests you have suggested.
> 
> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:
> Hi Luis,
> 
> What kind of file system was it? I fixed a bug a little while ago in that code for HFS file systems that resulted in a lot of cache misses.
> 
> In theory, everything should be cached.  It sounds like a bug if you are getting so many misses.  The basic idea of this code is that everything in the DB gets assigned a unique object ID and we make associations between files and their parent folder's unique ID.
> 
> Since you seem to be comfortable with a debugger in the code, can you set a breakpoint for when the miss happens and:
> 1) Determine the path of the file that was being added to the DB and the parent address that was trying to be found.
> 2) Use the 'ffind' TSK tool to then map that parent address to a path.  Is it a subset of the path from #1?
> 3) Open the DB in a SQLite tool and do something like this:
> 
> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
> 
> Is it in the DB?
> 
> Thanks!
> 
> brian
> 
> 
> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...> wrote:
> 
> > Hi,
> >
> > We have investigated a bit why the add image process is too slow in some cases. The add image process time seems to be quadratic with the number of files in the image.
> >
> > We detected that the function TskDbSqlite::findParObjId(), in db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id mapping in the local cache for a lot of files, causing it to search for the mapping in the database (not sure if it is an non-indexed search?)
> >
> > For testing purposes, we added a "return 1;" line right after the cache look up, disabling the database look up, and this resulted in great speed ups:
> >
> > number of files / default load_db time / patched load_db time
> > ~80.000 / 20min / 2min
> > ~300.000 / 3h / 7min
> > ~700.000 / 48h / 27min
> >
> > We wonder if it is possible to store all par_meta_addr -> par_id mappings into local cache (better) or doing an improved (indexed?) search for the mapping in the database. We think that someone with more knowledge of load_db code could help a lot here.
> > ------------------------------------------------------------------------------
> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> > unparalleled scalability from the best Selenium testing platform available.
> > Simple to use. Nothing to install. Get started now for free."
> > http://p.sf.net/sfu/SauceLabs_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
> 
> 
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

Ok, tests 1 and 3 done. I do not have sleuthkit code inside an ide, so did
not use breakpoints. Instead, I changed TskDbSqlite::findParObjId() to
return the parent_meta_addr when it is not found and return 1 when it is
found in the cache map.

Performing queries on the generated sqlite, there were 19.558 cache misses
from an image with 3 ntfs partitions and 127.408 files. I confirmed that
many parent_meta_addr missed from cache (now stored in
tsk_objects.par_obj_id) are into tsk_files.meta_addr. The complete paths
corresponding to these meta_addr are parents of those files whose
processing have not found them in cache.

Other tests resulted in:
182.256 cache misses from 433.321 files (ntfs)
892.359 misses from 1.811.393 files (ntfs)
169.819 misses from 3.177.917 files (hfs)

Luis Nassif



2014-05-01 16:14 GMT-03:00 Luís Filipe Nassif <lfc...@gm...>:

> Forgot to mention: we are using sleuthkit 4.1.3
> Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...> escreveu:
>
> Hi Brian,
>>
>> The 3 cases above were ntfs. I also tested with hfs and canceled loaddb
>> after 1 day. The modified version finished after 8hours and added about 3
>> million entries. We will try to do the tests you have suggested.
>> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:
>>
>>> Hi Luis,
>>>
>>> What kind of file system was it? I fixed a bug a little while ago in
>>> that code for HFS file systems that resulted in a lot of cache misses.
>>>
>>> In theory, everything should be cached.  It sounds like a bug if you are
>>> getting so many misses.  The basic idea of this code is that everything in
>>> the DB gets assigned a unique object ID and we make associations between
>>> files and their parent folder's unique ID.
>>>
>>> Since you seem to be comfortable with a debugger in the code, can you
>>> set a breakpoint for when the miss happens and:
>>> 1) Determine the path of the file that was being added to the DB and the
>>> parent address that was trying to be found.
>>> 2) Use the 'ffind' TSK tool to then map that parent address to a path.
>>>  Is it a subset of the path from #1?
>>> 3) Open the DB in a SQLite tool and do something like this:
>>>
>>> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
>>>
>>> Is it in the DB?
>>>
>>> Thanks!
>>>
>>> brian
>>>
>>>
>>> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > We have investigated a bit why the add image process is too slow in
>>> some cases. The add image process time seems to be quadratic with the
>>> number of files in the image.
>>> >
>>> > We detected that the function TskDbSqlite::findParObjId(), in
>>> db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
>>> mapping in the local cache for a lot of files, causing it to search for the
>>> mapping in the database (not sure if it is an non-indexed search?)
>>> >
>>> > For testing purposes, we added a "return 1;" line right after the
>>> cache look up, disabling the database look up, and this resulted in great
>>> speed ups:
>>> >
>>> > number of files / default load_db time / patched load_db time
>>> > ~80.000 / 20min / 2min
>>> > ~300.000 / 3h / 7min
>>> > ~700.000 / 48h / 27min
>>> >
>>> > We wonder if it is possible to store all par_meta_addr -> par_id
>>> mappings into local cache (better) or doing an improved (indexed?) search
>>> for the mapping in the database. We think that someone with more knowledge
>>> of load_db code could help a lot here.
>>> >
>>> ------------------------------------------------------------------------------
>>> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>>> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>>> > unparalleled scalability from the best Selenium testing platform
>>> available.
>>> > Simple to use. Nothing to install. Get started now for free."
>>> >
>>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>>> > sleuthkit-users mailing list
>>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>>> > http://www.sleuthkit.org
>>>
>>>

Re: [sleuthkit-users] Unencrypted Data Detection

[Joel F. <Joe...@is...>]

How about calculating entropy (https://github.com/sleuthkit/c_EntropyModule).
Would that work for you?


On Thu, May 1, 2014 at 3:22 PM, Brandon Lashmet <bla...@gm...> wrote:

> Is it possible to use SleuthKit to detect unencrypted data? What would the
> process be?
>
> Thanks,
>
> -Brandon
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

[sleuthkit-users] Unencrypted Data Detection

[Brandon L. <bla...@gm...>]

Is it possible to use SleuthKit to detect unencrypted data? What would the
process be?

Thanks,

-Brandon

[sleuthkit-users] Installing Sleuthkit Patches

[Brandon L. <bla...@gm...>]

Can someone tell me how to install SleuthKit patches?

I want to install the one mentioned in the article
here<https://www.utica.edu/academic/institutes/ecii/publications/articles/A0B3DC9E-F145-4A89-36F7462B629759FE.pdf>.
 Search for "Appendix A: FRSS-Sleuthkit Patch."

It says it's for version 1.69, so will it still work with the latest
version of SleuthKit?

Thanks for the help.

-Brandon

Re: [sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

Forgot to mention: we are using sleuthkit 4.1.3
Em 01/05/2014 16:09, "Luís Filipe Nassif" <lfc...@gm...> escreveu:

> Hi Brian,
>
> The 3 cases above were ntfs. I also tested with hfs and canceled loaddb
> after 1 day. The modified version finished after 8hours and added about 3
> million entries. We will try to do the tests you have suggested.
> Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:
>
>> Hi Luis,
>>
>> What kind of file system was it? I fixed a bug a little while ago in that
>> code for HFS file systems that resulted in a lot of cache misses.
>>
>> In theory, everything should be cached.  It sounds like a bug if you are
>> getting so many misses.  The basic idea of this code is that everything in
>> the DB gets assigned a unique object ID and we make associations between
>> files and their parent folder's unique ID.
>>
>> Since you seem to be comfortable with a debugger in the code, can you set
>> a breakpoint for when the miss happens and:
>> 1) Determine the path of the file that was being added to the DB and the
>> parent address that was trying to be found.
>> 2) Use the 'ffind' TSK tool to then map that parent address to a path.
>>  Is it a subset of the path from #1?
>> 3) Open the DB in a SQLite tool and do something like this:
>>
>> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
>>
>> Is it in the DB?
>>
>> Thanks!
>>
>> brian
>>
>>
>> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...>
>> wrote:
>>
>> > Hi,
>> >
>> > We have investigated a bit why the add image process is too slow in
>> some cases. The add image process time seems to be quadratic with the
>> number of files in the image.
>> >
>> > We detected that the function TskDbSqlite::findParObjId(), in
>> db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
>> mapping in the local cache for a lot of files, causing it to search for the
>> mapping in the database (not sure if it is an non-indexed search?)
>> >
>> > For testing purposes, we added a "return 1;" line right after the cache
>> look up, disabling the database look up, and this resulted in great speed
>> ups:
>> >
>> > number of files / default load_db time / patched load_db time
>> > ~80.000 / 20min / 2min
>> > ~300.000 / 3h / 7min
>> > ~700.000 / 48h / 27min
>> >
>> > We wonder if it is possible to store all par_meta_addr -> par_id
>> mappings into local cache (better) or doing an improved (indexed?) search
>> for the mapping in the database. We think that someone with more knowledge
>> of load_db code could help a lot here.
>> >
>> ------------------------------------------------------------------------------
>> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
>> > unparalleled scalability from the best Selenium testing platform
>> available.
>> > Simple to use. Nothing to install. Get started now for free."
>> >
>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>> > sleuthkit-users mailing list
>> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
>> > http://www.sleuthkit.org
>>
>>

Re: [sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

Hi Brian,

The 3 cases above were ntfs. I also tested with hfs and canceled loaddb
after 1 day. The modified version finished after 8hours and added about 3
million entries. We will try to do the tests you have suggested.
Em 01/05/2014 15:48, "Brian Carrier" <ca...@sl...> escreveu:

> Hi Luis,
>
> What kind of file system was it? I fixed a bug a little while ago in that
> code for HFS file systems that resulted in a lot of cache misses.
>
> In theory, everything should be cached.  It sounds like a bug if you are
> getting so many misses.  The basic idea of this code is that everything in
> the DB gets assigned a unique object ID and we make associations between
> files and their parent folder's unique ID.
>
> Since you seem to be comfortable with a debugger in the code, can you set
> a breakpoint for when the miss happens and:
> 1) Determine the path of the file that was being added to the DB and the
> parent address that was trying to be found.
> 2) Use the 'ffind' TSK tool to then map that parent address to a path.  Is
> it a subset of the path from #1?
> 3) Open the DB in a SQLite tool and do something like this:
>
> SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE
>
> Is it in the DB?
>
> Thanks!
>
> brian
>
>
> On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...>
> wrote:
>
> > Hi,
> >
> > We have investigated a bit why the add image process is too slow in some
> cases. The add image process time seems to be quadratic with the number of
> files in the image.
> >
> > We detected that the function TskDbSqlite::findParObjId(), in
> db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
> mapping in the local cache for a lot of files, causing it to search for the
> mapping in the database (not sure if it is an non-indexed search?)
> >
> > For testing purposes, we added a "return 1;" line right after the cache
> look up, disabling the database look up, and this resulted in great speed
> ups:
> >
> > number of files / default load_db time / patched load_db time
> > ~80.000 / 20min / 2min
> > ~300.000 / 3h / 7min
> > ~700.000 / 48h / 27min
> >
> > We wonder if it is possible to store all par_meta_addr -> par_id
> mappings into local cache (better) or doing an improved (indexed?) search
> for the mapping in the database. We think that someone with more knowledge
> of load_db code could help a lot here.
> >
> ------------------------------------------------------------------------------
> > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> > Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> > unparalleled scalability from the best Selenium testing platform
> available.
> > Simple to use. Nothing to install. Get started now for free."
> >
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> > sleuthkit-users mailing list
> > https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> > http://www.sleuthkit.org
>
>

Re: [sleuthkit-users] Slow Add Image Process Cause

[Brian C. <ca...@sl...>]

Hi Luis,

What kind of file system was it? I fixed a bug a little while ago in that code for HFS file systems that resulted in a lot of cache misses. 

In theory, everything should be cached.  It sounds like a bug if you are getting so many misses.  The basic idea of this code is that everything in the DB gets assigned a unique object ID and we make associations between files and their parent folder's unique ID. 

Since you seem to be comfortable with a debugger in the code, can you set a breakpoint for when the miss happens and:
1) Determine the path of the file that was being added to the DB and the parent address that was trying to be found.
2) Use the 'ffind' TSK tool to then map that parent address to a path.  Is it a subset of the path from #1?
3) Open the DB in a SQLite tool and do something like this:

SELECT * from tsk_files where meta_addr == META_ADDR_FROM_ABOVE

Is it in the DB?

Thanks!

brian


On May 1, 2014, at 11:58 AM, Luís Filipe Nassif <lfc...@gm...> wrote:

> Hi,
> 
> We have investigated a bit why the add image process is too slow in some cases. The add image process time seems to be quadratic with the number of files in the image.
> 
> We detected that the function TskDbSqlite::findParObjId(), in db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id mapping in the local cache for a lot of files, causing it to search for the mapping in the database (not sure if it is an non-indexed search?)
> 
> For testing purposes, we added a "return 1;" line right after the cache look up, disabling the database look up, and this resulted in great speed ups:
> 
> number of files / default load_db time / patched load_db time
> ~80.000 / 20min / 2min
> ~300.000 / 3h / 7min
> ~700.000 / 48h / 27min
> 
> We wonder if it is possible to store all par_meta_addr -> par_id mappings into local cache (better) or doing an improved (indexed?) search for the mapping in the database. We think that someone with more knowledge of load_db code could help a lot here.
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

[sleuthkit-users] Slow Add Image Process Cause

[Luís F. N. <lfc...@gm...>]

Hi,

We have investigated a bit why the add image process is too slow in some
cases. The add image process time seems to be quadratic with the number of
files in the image.

We detected that the function TskDbSqlite::findParObjId(), in
db_sqlite.cpp, is not finding the parent_meta_addr -> parent_file_id
mapping in the local cache for a lot of files, causing it to search for the
mapping in the database (not sure if it is an non-indexed search?)

For testing purposes, we added a "return 1;" line right after the cache
look up, disabling the database look up, and this resulted in great speed
ups:

number of files / default load_db time / patched load_db time
~80.000 / 20min / 2min
~300.000 / 3h / 7min
~700.000 / 48h / 27min

We wonder if it is possible to store all par_meta_addr -> par_id mappings
into local cache (better) or doing an improved (indexed?) search for the
mapping in the database. We think that someone with more knowledge of
load_db code could help a lot here.

Re: [sleuthkit-users] BitCoin Forensic investigator

[Simson G. <si...@ac...>]

Given the significant interest in BitCoin, we will probably incorporate this into the bulk_extractor main line in version 1.5.


On Apr 30, 2014, at 8:26 AM, Joel Fernandez <Joe...@is...> wrote:

> A student in my Spring 2013 course wrote feature bulk_extractor feature extractors for bitcoin and other e-currency. Since bulk_extractor doesn't care about the file-system, it works ok RAM, mobile, ios, windows, etc. It works pretty well. 
> 
> Colt VanWInkle Project with open-source code http://cyfor.isis.poly.edu/43-spring_2013_digital_forensics_final_project_page.html
> Video Presentation: http://youtu.be/24rSn6S4cnM
> Synopsis: http://cyfor.isis.poly.edu/assets/uploads/pages/spring_2013_digital_forensics_final_project_page/colt/vanwinkle_final.pdf
> 
> Cheers.
> 
> 
> 
> 
>  
> 
> 
> On Wed, Apr 30, 2014 at 3:25 AM, Enkidu Mo Shiri <vol...@gm...> wrote:
> Hello there,
> IM working on a project to create investigator tool for mobile phones( androind,windows,ios) to find evidences of any activity of bitcoin wallet on phone system.including ram and harddisk. i want to know if anyone has done such project before or not. i did google it and searched in journals such as IEEE,but so far,i just ofund IEF which is not focusing on bitcoin and they have it just as a function. is there any tool for finding evidences of bitcoin wallet leaving on system in mobile phones? (harddisk and ram)
> thank you 
> 
> Ehsan Moshiri (Enkidu)
> Digital Forensic Student
> H/P:+96164953954 , +961124249769
> Linkedin: http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/
> Facebook: Enkidu Mo Shi Ri
> wechat: Enkidu-Moshiri
> Line: Enkidu.Moshiri
> 
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
> 
> 
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs_______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org

Re: [sleuthkit-users] BitCoin Forensic investigator

[Joel F. <Joe...@is...>]

A student in my Spring 2013 course wrote feature bulk_extractor feature
extractors for bitcoin and other e-currency. Since bulk_extractor doesn't
care about the file-system, it works ok RAM, mobile, ios, windows, etc. It
works pretty well.

Colt VanWInkle Project with open-source code
http://cyfor.isis.poly.edu/43-spring_2013_digital_forensics_final_project_page.html
Video Presentation: http://youtu.be/24rSn6S4cnM
Synopsis:
http://cyfor.isis.poly.edu/assets/uploads/pages/spring_2013_digital_forensics_final_project_page/colt/vanwinkle_final.pdf

Cheers.







On Wed, Apr 30, 2014 at 3:25 AM, Enkidu Mo Shiri <vol...@gm...>wrote:

> Hello there,
> IM working on a project to create investigator tool for mobile phones(
> androind,windows,ios) to find evidences of any activity of bitcoin wallet
> on phone system.including ram and harddisk. i want to know if anyone has
> done such project before or not. i did google it and searched in journals
> such as IEEE,but so far,i just ofund IEF which is not focusing on bitcoin
> and they have it just as a function. is there any tool for finding
> evidences of bitcoin wallet leaving on system in mobile phones? (harddisk
> and ram)
> thank you
>
> *Ehsan Moshiri (Enkidu)*
> *Digital Forensic Student*
> *H/P:+96164953954 , +961124249769*
>
> *Linkedin: http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/
> <http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/> *
> *Facebook: Enkidu Mo Shi Ri*
> *wechat: Enkidu-Moshiri*
> *Line: Enkidu.Moshiri*
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>
>

[sleuthkit-users] BitCoin Forensic investigator

[Enkidu Mo S. <vol...@gm...>]

Hello there,
IM working on a project to create investigator tool for mobile phones(
androind,windows,ios) to find evidences of any activity of bitcoin wallet
on phone system.including ram and harddisk. i want to know if anyone has
done such project before or not. i did google it and searched in journals
such as IEEE,but so far,i just ofund IEF which is not focusing on bitcoin
and they have it just as a function. is there any tool for finding
evidences of bitcoin wallet leaving on system in mobile phones? (harddisk
and ram)
thank you

*Ehsan Moshiri (Enkidu)*
*Digital Forensic Student*
*H/P:+96164953954 , +961124249769*

*Linkedin: http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/
<http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/>*
*Facebook: Enkidu Mo Shi Ri*
*wechat: Enkidu-Moshiri*
*Line: Enkidu.Moshiri*

[sleuthkit-users] Detect overwritten files in FAT32

[Andreas E. <and...@ne...>]

Hello, I have been trying to detect files that have been overwritten in a FAT32 image by looking at the properties on TSK_FS_META.
I tried to get a hint by opening the image in Autopsy but it seems that the GUI only displays them as "Deleted" and there is no way of detecting if a file has been overwritten.
If I open the same image in EnCase it correctly shows the files as "Is Overwritten".

Is there any other flag or value I can look at in order to detect if a file has been deleted and overwritten?

Regards
Andreas Eriksson

Andreas Eriksson
Project Manager R&D

[NetClean]

NetClean Technologies Sweden AB
F?rsta L?nggatan 30 - SE-413 27 G?teborg - Sweden

Phone: +46 31 719 08 00 - Fax: +46 31 13 89 50
Direct: +46 31 719 08 16 - Mobile: +46 739 07 41 79

and...@ne...
<mailto:and...@ne...>www.netclean.com<http://www.netclean.com>

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

Re: [sleuthkit-users] Configure Java/JNI

[Derrick K. <dk...@gm...>]

Hello.

If you are compiling Sleuthkit to use standalone on the command line the
Java/JNI bindings won't matter.  They are primarily used for Autopsy 3 so
if you plan on compiling Autopsy 3 against TSK later on then you'll want
them.

My suspicion is that you have not installed Ant on your system and that's
why the Java/JNI features are not included.  Can you confirm if you have
Ant installed?

Derrick


On Sat, Apr 26, 2014 at 3:41 PM, Fsheathiii <fsh...@ne...> wrote:

> I am building Sleuth Kit on a Suse Linux system.   I see that Java/JNI
> is not supported.  I have determined that there is a Sleuth kit Java
> Package for this.  How do I get this feature enabled? Does it matter
>
> heath@suse64:~/Autopsy/sleuthkit-4.1.3> sh configure
> .
> checking for javac... javac
> checking if javac works... yes
> checking for javac... /usr/bin/javac
> checking symlink for /usr/bin/javac... /etc/alternatives/javac
> checking symlink for /etc/alternatives/javac...
> /usr/lib64/jvm/java-1.7.0-openjdk/bin/javac
> .
> .
> checking for java... java
> .
> .
>
> Features:
>    Java/JNI support:                      no
>
> Thank You
>
> Frank S. Heath
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> sleuthkit-users mailing list
> https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
> http://www.sleuthkit.org
>

[sleuthkit-users] Configure Java/JNI

[Fsheathiii <fsh...@ne...>]

I am building Sleuth Kit on a Suse Linux system.   I see that Java/JNI 
is not supported.  I have determined that there is a Sleuth kit Java 
Package for this.  How do I get this feature enabled? Does it matter

heath@suse64:~/Autopsy/sleuthkit-4.1.3> sh configure
.
checking for javac... javac
checking if javac works... yes
checking for javac... /usr/bin/javac
checking symlink for /usr/bin/javac... /etc/alternatives/javac
checking symlink for /etc/alternatives/javac... 
/usr/lib64/jvm/java-1.7.0-openjdk/bin/javac
.
.
checking for java... java
.
.

Features:
   Java/JNI support:                      no

Thank You

Frank S. Heath

[sleuthkit-users] Encryption Detection

[Brandon L. <bla...@gm...>]

Hi all,

I'm wondering if it's possible to use SleuthKit as an encryption detection
program.  Specifically, can it be used to scan a large directory of files
and report the ones that are unencrypted?

There is a patch referenced on page 12 of this
article<https://www.utica.edu/academic/institutes/ecii/publications/articles/A0B3DC9E-F145-4A89-36F7462B629759FE.pdf>,
but I'm not sure if it can be used for the purposes mentioned.

Thanks for your help,

-Brandon

[sleuthkit-users] Open Source Digital Forensics Conference - Call For Presentations

[Brian C. <ca...@sl...>]

The 5th Annual Open Source Digital Forensics Conference (OSDFCon) will be held on November 5, 2014 in Herndon, VA.  All users and developers are invited to submit a presentation or workshop topic by June 1st.  This is a unique opportunity to present your work and experiences to over 400 people.

The conference will be attended by both digital forensic investigators and developers.  This event is a great opportunity to make investigators aware of your tools, get feedback from users, meet fellow developers and users, and help direct the future of open source digital forensics software.

To receive updates about the conference, sign up for e-mail updates (http://www.osdfcon.org) or watch #osdfcon on twitter.



TOPICS

We are looking for 35-minute talks on a variety of topics about using open source tools, including:

 * New tools and analysis techniques
 * New features to mature tools
 * Open, plug-in analysis framework designs and experiences
 * Automated analysis
 * Hard drive analysis and triage
 * Memory and network forensics
 * Mobile device forensics
 * Analyzing application-level artifacts
 * Cyber incident response
 * User experiences
 * Case studies


We also have openings for half-day workshops on the day before the conference (November 4, 2014).  The workshops should teach people how to use or develop open source tools by providing hands-on guidance.



SUBMISSION INSTRUCTIONS

Topics can be submitted using an online form:

    http://www.basistech.com/osdfcon/cfp/

Submissions are due June 1, 2014. Our plan this year is to do an initial pass of the submissions and then use crowd sourcing to choose the final set of topics.

Re: [sleuthkit-users] mactime -i issues

[Stefan K. <sk...@bf...>]

Brian,

> I added a check though to give an error if you supply multiple '-i' values to make it more obvious that you can give only one.

Great. Thanks for the swift reply!

Cheers,

	Stefan.

-- 
Stefan Kelm                   <sk...@bf...>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstrasse 100             Tel: +49-721-96201-1
D-76133 Karlsruhe             Fax: +49-721-96201-99