sleuthkit-developers — A list to discuss development of new features of The Sleuth Kit and Autopsy

[sleuthkit-developers] Good vs. Bad Hashes

[Brian C. <ca...@sl...>]

Is anyone interested in looking into the best way to manage hashes? The 
definition of "good" versus "bad" is relative to the current 
investigation and I don't know the best way to handle this in The 
Sleuth Kit and Autopsy.  There could be a single database with 
categories of hashes and you choose which are good and which are bad 
for that investigation (similar to the new Forensic Hash Database that 
was announced and NSRL).  Or, you could import tens of hash databases 
and identify them as bad or good (like hashkeeper).

I think hashkeepr is LE-only, so I would rather focus on using NSRL and 
custom hashes made by md5sum.  If anyone is interested in working on a 
workable solution to this, let me know.

brian

[sleuthkit-developers] Beta of Autopsy v2.00

[Brian C. <ca...@sl...>]

Happy New Year everyone.  Over the holidays, I was able to perform the 
architecture redesign of Autopsy.

The current design is 3 years old and has one main file that is 10.000+ 
lines of code.  The new design is more modular and has 17 files and 
will be easier to add new features to.  The new design will be released 
as version 2.0 after I add some new incident response features.

The external look of the program is the same, so you shouldn't see a 
difference.  If you would like to use the new design before the full 
version is released, let me know (off list).  There maybe a few URLs 
and windows that didn't get a complete upgrade and I would like to find 
those and get any design feedback before 2.0 is released.

thanks,
brian

[sleuthkit-developers] icat -s

[Brian C. <ca...@ce...>]

Does anyone use the '-s' flag with icat?  It is currently setup to 
allow you to specify the type and id value for an NTFS inode, but you 
can also specify the type and id using the normal 123-128-1 inode 
address.  I am adding a flag to allow 'icat' to also show the slack 
space of the file and '-s' seems to be the most logical.  Autopsy 
doesn't use '-s' with icat and I no longer know why I added that flag.

So, if no one uses it, then I will replace it with the slack space 
option.

brian

Re: [sleuthkit-developers] Splitting and progress

[Brian C. <ca...@sl...>]

> I'm on the verge of a new release of the Indexed search 

>  functionality. I was wondering how the progress on the Autopsy 

>  restructuring was going. If you plan to release a new version 

>  soon, I will hold my release to go with it.. Otherwise I will release 

>  a version for Sleuthkit 1.65 and Autopsy 1.74.



Not anytime soon.



> Has anybody else already tested/used these tools? I would like to 

> receive some feedback if possible...



Have you run the tests that I developed for FAT keyword searching?

I released these a couple of weeks back on the CFTT@yahoogroups

list.



    http://www.cerias.purdue.edu/homes/carrier/forensics/tests/test2



> I have just conducted a forensic investigation on a 100 Gb disk... 

>  And therefore got ample oppertunity to test the new indexing 

>  features..

> 

> A few numbers:

> Indexing of the disk took only 9 hours and resulted in 4.7 Billion 

>  indexed points....



Do you know how that time compares with FTK?



> All words letter and number combinations of length 3 to 8 were 

>  indexed.

> Combining of the index files into a single index took 4.5 hours and 

>  resulted in a 17 Gb file.

> Searches for a specific word resulted in 75000 hits and only took 7 

> seconds to perform.



Very cool.  One of the current limitations of the Autopsy searching

is that it does not reconstruct files before searching.  So, if a file

has a string that crosses fragmented sectors, it will not be found.

Changing the search method to look at each file (using 'fls' and 'icat')

would be VERY slow (it would be comparable to running 'sorter').

I would hope that the indexing could make the search for each

file much faster.



thanks,

brian

[sleuthkit-developers] Splitting and progress

[Paul B. <ba...@fo...>]

Hi Brian end everybody else,

The list has been a bit empty lately....

I'm on the verge of a new release of the Indexed search functionality. I =
was wondering how the progress on the Autopsy restructuring was going. =
If you plan to release a new version soon, I will hold my release to go =
with it.. Otherwise I will release a version for Sleuthkit 1.65 and =
Autopsy 1.74.

Has anybody else already tested/used these tools? I would like to =
receive some feedback if possible...

I have just conducted a forensic investigation on a 100 Gb disk... And =
therefore got ample oppertunity to test the new indexing features..

A few numbers:
Indexing of the disk took only 9 hours and resulted in 4.7 Billion =
indexed points....
All words letter and number combinations of length 3 to 8 were indexed.
Combining of the index files into a single index took 4.5 hours and =
resulted in a 17 Gb file.
Searches for a specific word resulted in 75000 hits and only took 7 =
seconds to perform.


--
Paul Bakker

[sleuthkit-developers] Perl Guru Needed - I/O

[Brian C. <ca...@sl...>]

Any Perl Gurus on this list?  

I'm trying to fix the problem that was mentioned on the users list 
recently.  With Perl 5.8.0 the output of some commands is not
shown.  Hit refresh and its there.  

I was using backticks for the commands and today I changed to 
using 'open' with pipes (which I had been meaning to do for a
while).  That doesn't help.  I set the pipe to autoflush and that didn't
help.  If I sleep for a second before reading the pipe then the data
is always there, but it slows the whole process down.  If I do a 
'wait' on the output then for large amounts of output the system 
hangs because the process won't continue until its buffer is read.

Any other ideas?  I don't understand why this only happens with
5.8 and not 5.6.

brian

RE: [sleuthkit-developers] New Features / Changes

[Paul B. <ba...@fo...>]

> > > - Redesign Autopsy so that it is easier for people to add=20
> > >   functions.
> > This would help a lot.... It is probably closely tied to the Hooks=20
> >  feature I'm proposing below. You should be able to hook a function=20
> >  into a page. Shall I assist you with this feature?
>=20
> The hooks feature, which I agree would be nice, doesn't need to be
> tied to the re-architecture.  The hooks would probably be an=20
> installation wide configuration.  A config file can have the regular
> expression to match against 'file' and the path of the program to
> execute.=20

Yes I agree.. For the hooks to applications...
But I was also referring to function hooks..

Thus in the search tab I just have to add somthing like:
add_hook(INDEXED_SEARCH_BOX);
Which is a function defined in another file thus not needing to add =
something in the code of the current search screen.

> What is the output of the libPST?  Are there tools pre-defined or is
> it just a library that you must write the code for?

As far as I know: both...

So that can always be made to work.

> The registry tools that I know of, are being developed in the same
> model as The Sleuth Kit.  So, there is a regls and a regcat. =20
> They work
> with 2K and XP (I think).  These tools would also fall under the
> 'Application Analysis' mode.    Actually, I guess the hooks design=20
> would also fall under the Appliation Analysis mode.  Instead of=20
> opening up just the 'Cell' window, it would open the application mode
> that had the HTML cell as one of the tabs. =20

OK great.. We'll see about that then.. And otherwise we can always =
switch to the other one instead.
=20
Signing off... Stepping in a plane in 3 hours....

Paul Bakker..

RE: [sleuthkit-developers] New Features / Changes

[Brian C. <ca...@sl...>]

On 15 Aug 2003 00:39 PDT you wrote:

> Hi Brian and everyone else,
> 
> I want to welcome everybody as well...
> I'm pleased to see so many people have joined already.
> 
> Now to business...
> 
> On you proposed features:
> 
> > - Redesign Autopsy so that it is easier for people to add 
> >   functions.
> This would help a lot.... It is probably closely tied to the Hooks 
>  feature I'm proposing below. You should be able to hook a function 
>  into a page. Shall I assist you with this feature?

The hooks feature, which I agree would be nice, doesn't need to be
tied to the re-architecture.  The hooks would probably be an 
installation wide configuration.  A config file can have the regular
expression to match against 'file' and the path of the program to
execute. 

> 
> > - Add a sector offset value to all Sleuth Kit tools so that they can be 
> >   used on a disk image instead of just file system images
> 
> I was about to propose the same.... Though not only offset is 
>  required, also the size should be given. (Tools that process the 
>  partition as raw data (Like Indexed searching) should not read too 
>  much)

I was actually referring to just the file system tools (even though I
said all tools).  So, if your partition begins in sector 63, then you can
run the following:

  # fls -f ntfs -o 63 disk.img

> As to the new (and future) features:
> 
> - Hooks. I would like to see the possilibity to add a hook to a "file 
>  type" so it is possible to open a .gif in an external OR internal 
>  viewer. Internal viewers would be the integrated Foundstone tools 
>  and possibly ol2mbox.

Yea, that should be easy to do.  

> - Integrate LibPST into Autopsy. The ol2mbox library (Better known 
>  as libPST) is almost newly released and is much improved. LibPST 
>  know supports not only mail messages but also appointments, tasks 
>  and contacts from Microsoft Outlook PST files. This is for me a much 
>  wanted functionality for Forensic research under *NIX....

I agree that this is needed, but am unsure how it should be 
integrated into autopsy.  I do not want to make a new tab for it,
because that does not scale very well.  I am almost wondering if 
there should be an 'Application Analysis' mode.  The current tabs are
all 'File System Analysis' modes (well besides file type sorting).

What is the output of the libPST?  Are there tools pre-defined or is
it just a library that you must write the code for?



> - RAW partition. Ability to at least view a raw part of data. When 
>  imaging a disk that is only partitioned for 10%, I would like the 
>  possibility to use Autopsy to at least browse and search the other 
>  90% for possible data.

I have had that on the TODO list since v1.00, but have never done
it.  I was originally doing it for swap space, but partitions are good 
too.

> - Spliced imagefiles. Ability for the tools to work with spliced 
>  imagefiles. Sometimes it is not workable to have one 80 Gb image, 
>  but you have spliced it into 10 Gb parts. I would like to see the tools 
>  to be able to work with that, though I know this to be hard an d 
>  perhaps a very future feature.

Yea, it is has not been on the top of my list.

> I would like to point out the kregedit/regedit projects (From Samba >  developers) which can be used to graphically show windows registry 
>  files. I already had contact with them and they plan to move the 
>  registry functionality into a seperate library, thus providing maybe 
>  an alternative to the Registry Viewing Tools (Which I have not seen 
>  yet and don't know how far they are...)

The registry tools that I know of, are being developed in the same
model as The Sleuth Kit.  So, there is a regls and a regcat.  They work
with 2K and XP (I think).  These tools would also fall under the
'Application Analysis' mode.    Actually, I guess the hooks design 
would also fall under the Appliation Analysis mode.  Instead of 
opening up just the 'Cell' window, it would open the application mode
that had the HTML cell as one of the tabs.  

 
> As to the Autopsy interface:
> I think some things should be moved around to new pages. Things 
>  like creating a DLS image of you image should not be in the 
>  keyword searching tab but in a special tab, just like maybe the 
>  generating of strings files and the creating of indexes. They clutter 
>  the keyword search interface and require unwanted behaviour if 
>  some other tab (Like for instance the File Recovery tab (Foremost 
>  inclusion) requires a DLS of your image. You have to go to Keyword 
>  Searching to make this.

Good point.  I think the proper place for it is the 'Image Details'
view though.  There is no need for another tab.  A link can exist from
the needed tabs to the Image Details window to make the needed
files.

My short-term goals are for the sector offset into the file system
tools and incorporate disk images into Autopsy.  The re-design
of Autopsy is also short-term.  My initial plan is to have a file
for each 'tab'.  So, there is a search file, a 'file' file, a 'meta' file,
etc and a couple of general files.  I do not have much experience
with very large Perl programs though and have not done any
research on how others have done it so I'm not sure of the 
best way to do it (which is why I've always found better things
to work on).   The raw data will also be a short-term goal for me.

brian

[sleuthkit-developers] No comments from me

[Paul B. <ba...@fo...>]

Hi everybody...

I will not be responding to comments on my mail and on other mails for =
the next 10 days, because of a little holiday...

I will read everthing when I come back...

Paul Bakker

RE: [sleuthkit-developers] New Features / Changes

[Paul B. <ba...@fo...>]

Hi Brian and everyone else,

I want to welcome everybody as well...
I'm pleased to see so many people have joined already.

Now to business...

On you proposed features:

> - Redesign Autopsy so that it is easier for people to add=20
>   functions.
This would help a lot.... It is probably closely tied to the Hooks =
feature I'm proposing below. You should be able to hook a function into =
a page. Shall I assist you with this feature?

> - Add a sector offset value to all Sleuth Kit tools so that they can =
be=20
>   used on a disk image instead of just file system images

I was about to propose the same.... Though not only offset is required, =
also the size should be given. (Tools that process the partition as raw =
data (Like Indexed searching) should not read too much)


As to the new (and future) features:

- Hooks. I would like to see the possilibity to add a hook to a "file =
type" so it is possible to open a .gif in an external OR internal =
viewer. Internal viewers would be the integrated Foundstone tools and =
possibly ol2mbox.
- Integrate LibPST into Autopsy. The ol2mbox library (Better known as =
libPST) is almost newly released and is much improved. LibPST know =
supports not only mail messages but also appointments, tasks and =
contacts from Microsoft Outlook PST files. This is for me a much wanted =
functionality for Forensic research under *NIX....
- RAW partition. Ability to at least view a raw part of data. When =
imaging a disk that is only partitioned for 10%, I would like the =
possibility to use Autopsy to at least browse and search the other 90% =
for possible data.
- Spliced imagefiles. Ability for the tools to work with spliced =
imagefiles. Sometimes it is not workable to have one 80 Gb image, but =
you have spliced it into 10 Gb parts. I would like to see the tools to =
be able to work with that, though I know this to be hard an d perhaps a =
very future feature.

I would like to point out the kregedit/regedit projects (From Samba =
developers) which can be used to graphically show windows registry =
files. I already had contact with them and they plan to move the =
registry functionality into a seperate library, thus providing maybe an =
alternative to the Registry Viewing Tools (Which I have not seen yet and =
don't know how far they are...)

As to the Autopsy interface:
I think some things should be moved around to new pages. Things like =
creating a DLS image of you image should not be in the keyword searching =
tab but in a special tab, just like maybe the generating of strings =
files and the creating of indexes. They clutter the keyword search =
interface and require unwanted behaviour if some other tab (Like for =
instance the File Recovery tab (Foremost inclusion) requires a DLS of =
your image. You have to go to Keyword Searching to make this.


--
Paul Bakker

-----Oorspronkelijk bericht-----
Van: Brian Carrier [mailto:ca...@sl...]
Verzonden: donderdag 14 augustus 2003 17:37
Aan: sle...@so...
Onderwerp: [sleuthkit-developers] New Features / Changes




Welcome everyone.

At this point, there are around 15 people on this list so it is time to=20
start discussing what to develop.

I would first like to say that I welcome any help, espeically on the
interface.  I enjoy the low-level disk and file system tools and would
welcome anyone that would like to offer a better, and still free =
interface.  My primary focus right now is on my Purdue research,
so any help with this work is appreciated. =20

The following are the projects that I know of:
- Indexed Search (Fox-IT)
- Integrating foremost (Fox-IT)
- New Interface (openforensics.org)
- Registry Viewing Tools (not yet public)

The smaller features that I want to see done are the following:
- Redesign Autopsy so that it is easier for people to add=20
functions. =20
- Add a sector offset value to all Sleuth Kit tools so that they can be=20
used on a disk image instead of just file system images
- Add 'mmls' and offset values to Autopsy so that disk images can=20
be imported
- Integrate the new Foundstone tools for viewing the trash bin,
history file etc.
- Improve timeline tools and add some scripts to parse logs into the
body file.

I think the redesign is the most important to make the Fox-IT work
easier.

thanks for your interest.
brian

[sleuthkit-developers] New Features / Changes

[Brian C. <ca...@sl...>]

Welcome everyone.

At this point, there are around 15 people on this list so it is time to 
start discussing what to develop.

I would first like to say that I welcome any help, espeically on the
interface.  I enjoy the low-level disk and file system tools and would
welcome anyone that would like to offer a better, and still free interface.  My primary focus right now is on my Purdue research,
so any help with this work is appreciated.  

The following are the projects that I know of:
- Indexed Search (Fox-IT)
- Integrating foremost (Fox-IT)
- New Interface (openforensics.org)
- Registry Viewing Tools (not yet public)

The smaller features that I want to see done are the following:
- Redesign Autopsy so that it is easier for people to add 
functions.  
- Add a sector offset value to all Sleuth Kit tools so that they can be 
used on a disk image instead of just file system images
- Add 'mmls' and offset values to Autopsy so that disk images can 
be imported
- Integrate the new Foundstone tools for viewing the trash bin,
history file etc.
- Improve timeline tools and add some scripts to parse logs into the
body file.

I think the redesign is the most important to make the Fox-IT work
easier.

thanks for your interest.
brian

Re: [sleuthkit-developers] What to do...

[Brian C. <ca...@sl...>]

> I think it would be a good idea to collect a list of features/changes that
> are needed/wanted for inclusion in Sleuthkit....

I am not ignoring this thread.  I am just waiting a day until people who
want to sign up have done so.  

I'll send a list of what I know of in the morning.

brian

Re: [sleuthkit-developers] What to do...

[Jon W. <jw...@op...>]

Paul,

    What I'd like to do is get you and Brian familiar with our tool, Rex.
It's an EnCase-like tool written in Java, with a C back end. It primarily
runs on Linux (the C backend is Linux specific right now). The nice thing is
that it's relatively easy to tie all the sluethkit functionality into the
code (we even have an interface for an indexed search, although we have a
license for a proprietary indexing engine).

    I'd like to get your impressions of the tool and see if you think it's
worth your time to assist us in working on it. Please don't misunderstand
me, I am very impressed with all the work Brian has done on Autopsy in
particular, but if we want this tool (I'm speaking here of our combined
efforts) to have extremely wide adoption I think going with a familiar
interface is a large bonus.

    You can download the tool by going to
http://opensource.veritect.com/files/publicc/projects/rex

    You'll need a Java 1.4.2 enviornment for this to run, but it has tested
out on WIndows, Linux, and OSX. THis does not include the back end (indexed
search, etc.) but should give you an idea of what we have accomplished.

    Please give it a once-over and let me know what you think. My
development team has been looking through sleuthkit, and I'd like to start
figuring out what the development priorities will be for this project so we
can start trying to move to a common goal.

    Anyhow, those are my thoughts, I'm just rambling here off the top of my
head. Please take a look and let me know what you all think. My goal has
always been to make available a free (as in beer) and free (as in speech)
enterprise class forensics toolkit for the community. I think our combined
efforts have a real solid chance of pulling it off. I look forward to seeing
what you all think.

    -Jon

----- Original Message ----- 
From: "Paul Bakker" <ba...@fo...>
To: <sle...@li...>
Sent: Wednesday, August 13, 2003 7:27 AM
Subject: [sleuthkit-developers] What to do...


Hi all...

I think it would be a good idea to collect a list of features/changes that
are needed/wanted for inclusion in Sleuthkit....

That way we can split up these features/changes between the developers..
Thus no work is performed double... It is coordinated and we can first
discuss their functionality/implementation...

What do you all think?

Paul Bakker


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
sleuthkit-developers mailing list
sle...@li...
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers

[sleuthkit-developers] random thoughts (was re: What to do...)

[Jon W. <jw...@op...>]

Paul,

    What I'd like to do is get you and Brian familiar with our tool, =
Rex.
It's an EnCase-like tool written in Java, with a C back end. It =
primarily
runs on Linux (the C backend is Linux specific right now). The nice =
thing is
that it's relatively easy to tie all the sluethkit functionality into =
the
code (we even have an interface for an indexed search, although we have =
a
license for a proprietary indexing engine).

    I'd like to get your impressions of the tool and see if you think =
it's
worth your time to assist us in working on it. Please don't =
misunderstand
me, I am very impressed with all the work Brian has done on Autopsy in
particular, but if we want this tool (I'm speaking here of our combined
efforts) to have extremely wide adoption I think going with a familiar
interface is a large bonus.

    You can download the tool by going to
http://opensource.veritect.com/files/publicc/projects/rex

    You'll need a Java 1.4.2 enviornment for this to run, but it has =
tested
out on WIndows, Linux, and OSX. THis does not include the back end =
(indexed
search, etc.) but should give you an idea of what we have accomplished.

    Please give it a once-over and let me know what you think. My
development team has been looking through sleuthkit, and I'd like to =
start
figuring out what the development priorities will be for this project so =
we
can start trying to move to a common goal.

    Anyhow, those are my thoughts, I'm just rambling here off the top of =
my
head. Please take a look and let me know what you all think. My goal has
always been to make available a free (as in beer) and free (as in =
speech)
enterprise class forensics toolkit for the community. I think our =
combined
efforts have a real solid chance of pulling it off. I look forward to =
seeing
what you all think.

    -Jon

----- Original Message -----=20
From: "Paul Bakker" <ba...@fo...>
To: <sle...@li...>
Sent: Wednesday, August 13, 2003 7:27 AM
Subject: [sleuthkit-developers] What to do...


Hi all...

I think it would be a good idea to collect a list of features/changes =
that
are needed/wanted for inclusion in Sleuthkit....

That way we can split up these features/changes between the developers..
Thus no work is performed double... It is coordinated and we can first
discuss their functionality/implementation...

What do you all think?

Paul Bakker


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/=
01
_______________________________________________
sleuthkit-developers mailing list
sle...@li...
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers

[sleuthkit-developers] What to do...

[Paul B. <ba...@fo...>]

Hi all...

I think it would be a good idea to collect a list of features/changes =
that are needed/wanted for inclusion in Sleuthkit....

That way we can split up these features/changes between the developers..
Thus no work is performed double... It is coordinated and we can first =
discuss their functionality/implementation...

What do you all think?

Paul Bakker