[sleuthkit-users] Testers for indexed searching in Autopsy and Sleuthkit

[Paul B. <ba...@fo...>]

=20
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I work at a company doing Forensic IT investigations in the Netherlands =
called Fox-IT (http://www.fox-it.com). We are working on an all-Linux =
environment for Forensic research.

As the main Forensic tool we would like to use Autopsy/Sleuthkit. As it =
is missing some features in comparison to (commercial) Windows products, =
we've decided to contribute and add some new features to Autopsy and =
Sleuthkit. We're doing this in cooperation with Brian Carrier of @stake.

One of the major missing features is indexed searching. Indexed =
searching greatly speeds up searches for words during investigations.

We created a first implementation for indexed searching in Autopsy and =
Sleuthkit. This e-mail is to inform the users of this new addition and =
to request testers as the code is still in beta. After the addition has =
been successfully tested it will be submitted for integration in Autopsy =
and Sleuthkit.

Indexed searching requires the creation of two additional files (And =
thus will require additional diskspace). The total size of these files =
is comparable to the size of the strings file generated from an image. =
For the creation however, twice the space of the strings file is =
required.

It has been tested on a Debian Linux system and on a number of forensic =
images. The speedup for searching is very great (Searches on a 5 Gb =
image file for a single word in less than 1 second (Resulting in 11866 =
hits), compared to 168 seconds using the regular grepping on the strings =
file).

For the indexed search two files are required: a "mangled strings" file =
and an "index" file.
The creation of the "mangled strings" file requires the strings file for =
the image. The process is split in two parts (but are combined within =
Autopsy) and takes about 68 minutes to complete for a 3.5 Gb strings =
file, resulting in a 4.0 Gb "mangled strings" file. During the proces =
about 8.5 Gb of temporary space is required!
The creation of the "index" file requires the "mangled strings" file and =
takes about 5 minutes to complete for the aforementioned 4.0 Gb file. =
The resulting "index" file is only 5 Mb in size.

Features:
 - Tools for Indexed searching in sleuthkit.
 - Creation of necessary files integrated into Autopsy interface.
 - Indexed Search field (At the bottom of the "Keyword search" page).
 - Case insensitive searching.

There are still some limitations:
 - Only the ASCII character set is recognized for indexing.
   This is because the meaning of Unicode characters depends on the
   context. This makes it very hard to index these. If somebody knows
   how this could be integrated in the utilities, I will gladly add
   the functionality.
 - Only able to search for single words.
   Option for combining multiple searches will be added in a later
   version (In addition to the option to recall search results).
 - Only start of words can be searched.
   e.g. the original word is "baseball". A search for "base" will
   match, a search for "ball" will not. In the future I will expand
   the indexing functionality. This will require a lot of additional
   diskspace (So this option comes at a price).
 - No regex searches possible. (It is almost impossible to combine
   indexed searching with regex.)

The available patches are for Autopsy 1.71 and Sleuthkit 1.61. They add =
a first (beta) version of indexed searching to Autopsy.

It is still in beta and therefore I would greatly appreciate it if =
people would test the indexed searching on other machines and images and =
send their problems, feedback and feature requests to me.

All feedback is appreciated! My goal is to add useful features (like =
indexed searching) to Autopsy and Sleuthkit. This requires feedback! ;-)

Please send an e-mail to me if you'd like to test the patches.


- --
Paul Bakker

Fox-IT Experts in IT Security!
Haagweg 137=20
2281 AG RIJSWIJK=20
T 070 336 9999=20
F 070 336 9990=20
I www.fox-it.com=20
E ba...@fo...
57A6 C5EA 55E4 CC1C A967 B13C F8C0 C0FB 8135 E225

Disclaimer: This email may contain confidential information. If this =
message is not addressed to you, you may not retain or use the =
information in it for any purpose. If you have received it in error, =
please notify the sender and delete this message. We try to screen out =
viruses but take no responsibility if this email contains a virus.

-----BEGIN PGP SIGNATURE-----
Version: PGP 7.1.1

iQA/AwUBPre/s/jAwPuBNeIlEQIQjACePe0LIhqbp9VoWEwHw7tXKPslGV8AnRy9
/SuxnfOWHs5a5j0YkQ8ZLKIT
=3DZikc
-----END PGP SIGNATURE-----