sleuthkit-users — List to discuss Autopsy and The Sleuth Kit.

Re: [sleuthkit-users] cdrom

[Brian C. <ca...@sl...>]

On 15 Jun 2003 03:11 PDT you wrote:
> Is it possible to forensic cdrom ? There isn't an option for iso in the 
> panel "add a new image" -> "file system type". What is the good way to 
> forensic a cdrom ?

The Sleuth Kit does not have support for an ISO image.  

You can likely mount the image in loopback with Linux though and use standard tools on it.  If you want to make a timeline, then you can use the 'mac-robber' tool from the sleuthkit.org website to collect time information. Although, I'm not sure if CDs just have their MAC times
set to when the ISO was created or not (obviously the A-time will not 
be updated).

There is also likely no valuable information in "unallocated" space
either, so  mounting in loopback will probably do just fine.

brian

[sleuthkit-users] cdrom

[Philippe J. <phi...@wa...>]

Hi,

Is it possible to forensic cdrom ? There isn't an option for iso in the 
panel "add a new image" -> "file system type". What is the good way to 
forensic a cdrom ?

thant you

Philippe

Re: [sleuthkit-users] dls linux-ext3 error

[Brian C. <ca...@sl...>]

On 14 Jun 2003 19:11 PDT you wrote:

> Hi:
> 
> I am trying to recover some deleted files on partition hda6.
> I created an image using this command:
> 
> dls -v -f linux-ext3 /dev/hda6 > /mnt/usb/output
> 
> But when I tried to stat the image I get this error:
> 
> debian:# fsstat -f linux-ext3 /mnt/usb/output
> fsstat: Error: /mnt/usb/output is not an EXT2FS file system
> 
> Well of course it's not an ext2 system. It's a ext3 system. Am I 
> going to have to redo this image as an ext2 system in order to 
> import it? Perhaps I did something wrong when using dls?

I'll fix the error message.  EXT2FS and EXT3FS are almost identical
and therefore use the same code in the Sleuth Kit.  I'll make that
message reflect the specific FS though.

The message is appearing because the result is no longer a file
system.  'dls' goes through the allocation bitmap and finds the
ones that are not allocated and prints them.  So, there is no 
longer any structure to the data, it is just raw data.  Therefore,
all you can do with it is search it and use data carving tools.

> Also the Image size doesn't appear to match the size of 'available' space on the target partition.
> 
> partion sizes (kb):
> /dev/hda6             13179944   3945620   8564820  32% /mnt/hda6
> 
> image size(b):
> -rw-r--r--    1 root     root     9455947776 Jun 13 18:29 output

Actually, they do work out.  I'm assuming the full disk is 13179944k
and 3945620k is the used space.  If you subtract them, you 
actually get 9234324k (not 8564820k).  If you divide the size
of the dls output by 1024 you get:

9455947776 / 1024 = 9234324k

I have no clue where df got the 8564820k value came from.  

brian

[sleuthkit-users] dls linux-ext3 error

[Gary M. W. <wit...@so...>]

Hi:

I am trying to recover some deleted files on partition hda6.
I created an image using this command:

dls -v -f linux-ext3 /dev/hda6 > /mnt/usb/output

But when I tried to stat the image I get this error:

debian:# fsstat -f linux-ext3 /mnt/usb/output
fsstat: Error: /mnt/usb/output is not an EXT2FS file system

Well of course it's not an ext2 system. It's a ext3 system. Am I going to have to redo this image as an ext2 system in order to import it? Perhaps I did something wrong when using dls?

Also the Image size doesn't appear to match the size of 'available' space on the target partition.

partion sizes (kb):
/dev/hda6             13179944   3945620   8564820  32% /mnt/hda6

image size(b):
-rw-r--r--    1 root     root     9455947776 Jun 13 18:29 output

I would be very thankful for any help.

Re: [sleuthkit-users] Road Machine, Imaging, and Processor Speed?

[<for...@kr...>]

On l=F8rdag, jun 14, 2003, at 16:50 Europe/Copenhagen, Rich Thompson=20
wrote:

> Hey,
Hi there
>
> I need to put together a portable imaging kit.
> Obviously I will use a laptop (w/ Linux), but question
> is on CPU speed.  I realize CPU speed is very
> important for my analysis station (searching etc...),
> but how important is a faster processor to making a dd
> image for take back to the lab?
why laptop? - see below

>
> Budget is limited, but I don't want to shoot myself in
> the foot either.  I use a 3.06ghz P4 w/ 1gb ram as my
> analysis machine and it hauls butt on everything
> (although, I do want to build a dual cpu machine - I
> hear they really crank on searching!)  What is
> everyone else using for their road machines?
>
> Does anyone have any preferences or experience with
> low, mid, and max speed laptops?  Or, for that matter,
> any opinions on using an Apple laptop to make dd
> images to take back?
I personally decided to buy a small EPIA machine
for use in forensics - which can use ordinary 3.5" harddisk!

They cost about one third of a laptop and I assume that
the 3.5" disks are faster than most laptop drives anyway.

The machine is based on a small Epia M 10000 Nehemiah
1GHz processor, 512MB memory and 80GB disk and a
Morex Cubid 2688r=A0
63.5mm(H) x 273mm(D) x 295mm(B)

This machine can also handle being transported better than
laptops - can stand more abuse, compared to a laptop IMHO.

The plans are for bringing it to the customer, and do initial
analysis on it.

I am sorry for not having any hard facts, but maybe I should run
the honeynet challenge through the machine.

Best regards

--
Henrik Lund Kramsh=F8j, cand.scient, CISSP
e-mail: hl...@se..., tlf: 2026 6000
www.security6.net - IPv6, sikkerhed, netv=E6rk og UNIX

[sleuthkit-users] RE: Road Machine, Imaging, and Processor Speed?

[Eagle I. S. <in...@ea...>]

I have been toying with the same idea and would love feedback also.

I'm a newbie to Linux, and so far have had horrendous speed issues with
dd. (over 90 minutes to image a 6 gig drive on the same IDE chain). Now
granted
I used a basic command of the form:

dindang:~/test #  dd if=/dev/hda1 of=/root/test/hda1

There may some parameters I don't know that could have helped.

The machine was a 2.0G P4 with 1Gig of Ram, running Suse 8.0.

I'm on my way to buy and Image Masster solo to do imaging, but like Rich,
I'd love to be able to use a laptop for this kind of field work, or even a
custom built
PC, but I haven't had faith in running DD since my 6gig drive experience.

An Image Masster Solo is super-fast. It'll image 40 gigs in less than 15
minutes.

I've seen it and used a loaner unit many many times, and I've been impressed
every time. It has no regard for bad HD or sectors, it also scours the image
drive to DOD specifications before making the copy.
It also has a feature to add a text file with drive info to the image drive
if you
are imaging multiple drives. But, alas, it costs in the region of $1500.

Niall
BSEE, B. Math. LPI
Eagle Investigative Services, Inc.
Atlanta, GA.



-----Original Message-----
From: sle...@li...
[mailto:sle...@li...]On Behalf Of Rich
Thompson
Sent: Saturday, June 14, 2003 10:51 AM
To: lin...@ya...;
sle...@li...
Subject: [sleuthkit-users] Road Machine, Imaging, and Processor Speed?


Hey,

I need to put together a portable imaging kit.
Obviously I will use a laptop (w/ Linux), but question
is on CPU speed.  I realize CPU speed is very
important for my analysis station (searching etc...),
but how important is a faster processor to making a dd
image for take back to the lab?

Budget is limited, but I don't want to shoot myself in
the foot either.  I use a 3.06ghz P4 w/ 1gb ram as my
analysis machine and it hauls butt on everything
(although, I do want to build a dual cpu machine - I
hear they really crank on searching!)  What is
everyone else using for their road machines?

Does anyone have any preferences or experience with
low, mid, and max speed laptops?  Or, for that matter,
any opinions on using an Apple laptop to make dd
images to take back?

I really want to get the best machine possible, stay
within my budget (which I don't have the $$ figure
yet), and be able to get other peripherals.  I will be
using firefly's from digital intel for connecting
suspect and target drives, and will max out ram on
whatever I get.

So, what are your thoughts?

Thanks in advance,
Rich

__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com


-------------------------------------------------------
This SF.NET email is sponsored by: eBay
Great deals on office technology -- on eBay now! Click here:
http://adfarm.mediaplex.com/ad/ck/711-11697-6916-5
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org

[sleuthkit-users] Road Machine, Imaging, and Processor Speed?

[Rich T. <te...@ya...>]

Hey,

I need to put together a portable imaging kit. 
Obviously I will use a laptop (w/ Linux), but question
is on CPU speed.  I realize CPU speed is very
important for my analysis station (searching etc...),
but how important is a faster processor to making a dd
image for take back to the lab?

Budget is limited, but I don't want to shoot myself in
the foot either.  I use a 3.06ghz P4 w/ 1gb ram as my
analysis machine and it hauls butt on everything
(although, I do want to build a dual cpu machine - I
hear they really crank on searching!)  What is
everyone else using for their road machines?

Does anyone have any preferences or experience with
low, mid, and max speed laptops?  Or, for that matter,
any opinions on using an Apple laptop to make dd
images to take back?

I really want to get the best machine possible, stay
within my budget (which I don't have the $$ figure
yet), and be able to get other peripherals.  I will be
using firefly's from digital intel for connecting
suspect and target drives, and will max out ram on
whatever I get.

So, what are your thoughts?

Thanks in advance,
Rich

__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com

RE: [sleuthkit-users] opensourceforensics.org

[Altheide, C. B. <Alt...@nv...>]

> -----Original Message-----
> From: Brian Carrier [mailto:ca...@sl...] 
> 
> > Knoppix is a very useful boot cd for forensic work 
> especially if you 
> > like to do manual poking around, as by defult it mounts the local 
> > disks read only so that the file's atime settings won't be modified 
> > accidentally.
> 
> There has actually been a thread on the linux_forensics@yahoo 
> list about this since someone noticed that the hash on an 
> EXT3FS file system changed after mounting it read-only with 
> knoppix.  They have been doing more tests and will be 
> publishing a final report.
> 

I've actually done some independent testing to see if I could reproduce
those results, but my findings indicate that knoppix does *not* write to the
journal when mounting a drive read-only.  Granted - I may have been doing
something different than Ernie, so I'm anxiously awaiting his report.

Cory Altheide
Computer Forensics Specialist
NCI Information Systems, Inc.
NNSA Cyber Forensics Center
alt...@nv...

Re: [sleuthkit-users] opensourceforensics.org

[Brian C. <ca...@sl...>]

On 12 Jun 2003 07:13 PDT you wrote:
> > Note that there are no bootable Linux CDs on the
> > site yet.  That is because all of the ones that I know
> > of do not include source code. They use an open
> > source OS, but only the ISO is available from the
> > website.
> 
> You've clearly not seen Knoppix.
> http://www.knopper.net/knoppix/index-en.html
> 
> All the sources are on the CD and if not there are available for download
> from http://www.knopper.net/download/knoppix/

I have never checked the sources on the CD, but the ones on the 
website do not seem complete.  For example, based on the file
names, I can't find the kernel source or version of 'binuntils'.
Of course, I could be wrong since I haven't opened up every tar 
file there.  

This brings up the point of basic documentation though.  Providing
a list of packages and libraries and no docs or scripts on how to 
put it together makes verification difficult.

> Knoppix is a very useful boot cd for forensic work especially if you like to
> do manual poking around, as by defult it mounts the local disks read only so
> that the file's atime settings won't be modified accidentally.

There has actually been a thread on the linux_forensics@yahoo
list about this since someone noticed that the hash on an EXT3FS
file system changed after mounting it read-only with knoppix.  They
have been doing more tests and will be publishing a final report.

> Also of interest, and knoppix based is a project Morphix
> http://morphix.sourceforge.net/modules/news/

I hadn't seen that one yet.  Thanks.

brian

Re: [sleuthkit-users] opensourceforensics.org

[Enda C. <en...@co...>]

> Note that there are no bootable Linux CDs on the
> site yet.  That is because all of the ones that I know
> of do not include source code. They use an open
> source OS, but only the ISO is available from the
> website.

You've clearly not seen Knoppix.
http://www.knopper.net/knoppix/index-en.html

All the sources are on the CD and if not there are available for download
from http://www.knopper.net/download/knoppix/

Knoppix is a very useful boot cd for forensic work especially if you like to
do manual poking around, as by defult it mounts the local disks read only so
that the file's atime settings won't be modified accidentally.

Also of interest, and knoppix based is a project Morphix
http://morphix.sourceforge.net/modules/news/

Lets you easily customise your boot cd.

 -Enda.

[sleuthkit-users] Autopsy 1.73 Release

[Brian C. <ca...@sl...>]

In record time, a new version of Autopsy is available.  The new mactime 
flag format ('-i day' instead of just '-i') was not incorporated into the 
distributed version of autopsy.  So, when a timeline was created an error was shown.  v1.73 fixes this.  Thanks to Cathy Buckman for pointing 
this out!

MD5: ea6d78cc494aa7255ef05ccb2006f0e8

http://www.sleuthkit.org/autopsy/index.php

thanks,
brian

RE: [sleuthkit-users] The Sleuth Kit 1.62 and Autopsy 1.72 Release

[Eagle I. S. <in...@ea...>]

Brian, 

>>- The results of keyword searches are saved to a file and can be quickly
>>  recalled. 

Awesome work. This is really a neat feature. Good Job!!

Niall.

-----Original Message-----
From: sle...@li...
[mailto:sle...@li...]On Behalf Of Brian
Carrier
Sent: Tuesday, June 10, 2003 2:32 AM
To: sle...@li...;
sle...@li...
Subject: [sleuthkit-users] The Sleuth Kit 1.62 and Autopsy 1.72 Release




The Sleuth Kit v1.62 and Autopsy v1.72 are now available.

Overview:
The Sleuth Kit has a few bug fixes and a few updates.  Autopsy
also has a few bug fixes and two new features.  

brian


THE SLEUTH KIT 1.62
MD5:
  sleuthkit-1.62.tar.gz = 98947fb65b41aa5ba600422bd8390062

Updates:
- Added the '-d' flag to 'mactime' to output the timeline in comma
  delimited format so that it can be imported into spread sheets
  for report generation or graphing.
- 'mactime' can create summary index files in a daily or hourly
  basis.  These are useful with the -d flag to import the summary
  files into a spread sheet and graph a histogram of activity.

Bug Fixes:
- In 'fsstat', the last group in an FFS file system could have 
  reported an incorrect last fragment.
- The last fragments in an FFS file system can be read when there
  are not enough fragments for the block.
- The 'file' output is sanitized in 'sorter' to reduce UTF-8 messages.
- 'sorter' now accepts linux-ext3 as a file system type.

http://www.sleuthkit.org/sleuthkit/index.php
http://sleuthkit.sourceforge.net/sleuthkit/index.php

AUTOPSY 1.72
MD5:
  autopsy-1.72.tar.gz = f8a74270ced5c302c04b5f17f4643827

New Features / Updates: 
- The new Event Sequencer mode allows one to create time-based
  events for file activity and other logs.   This allows one to
  easily sort a sequence of events during the investigation.
- The results of keyword searches are saved to a file and can be quickly
  recalled. 

Bug Fixes:
- calc_md5() would error if it was called more than once (Paul Bakker)
- Added 'LANG=C LC_ALL=C' to sorter and mactime to reduce the UTF-8
  warning messages (debugging help from Daniel Schwartzer).
- The timeline view now allows multiple users for a UID (reported
  by Cathy Buckman).


http://www.sleuthkit.org/autopsy/index.php
http://sleuthkit.sourceforge.net/autopsy/index.php



-------------------------------------------------------
This SF.net email is sponsored by:  Etnus, makers of TotalView, The best
thread debugger on the planet. Designed with thread debugging features
you've never dreamed of, try TotalView 6 free at www.etnus.com.
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org

[sleuthkit-users] The Sleuth Kit 1.62 and Autopsy 1.72 Release

[Brian C. <ca...@sl...>]

The Sleuth Kit v1.62 and Autopsy v1.72 are now available.

Overview:
The Sleuth Kit has a few bug fixes and a few updates.  Autopsy
also has a few bug fixes and two new features.  

brian


THE SLEUTH KIT 1.62
MD5:
  sleuthkit-1.62.tar.gz = 98947fb65b41aa5ba600422bd8390062

Updates:
- Added the '-d' flag to 'mactime' to output the timeline in comma
  delimited format so that it can be imported into spread sheets
  for report generation or graphing.
- 'mactime' can create summary index files in a daily or hourly
  basis.  These are useful with the -d flag to import the summary
  files into a spread sheet and graph a histogram of activity.

Bug Fixes:
- In 'fsstat', the last group in an FFS file system could have 
  reported an incorrect last fragment.
- The last fragments in an FFS file system can be read when there
  are not enough fragments for the block.
- The 'file' output is sanitized in 'sorter' to reduce UTF-8 messages.
- 'sorter' now accepts linux-ext3 as a file system type.

http://www.sleuthkit.org/sleuthkit/index.php
http://sleuthkit.sourceforge.net/sleuthkit/index.php

AUTOPSY 1.72
MD5:
  autopsy-1.72.tar.gz = f8a74270ced5c302c04b5f17f4643827

New Features / Updates: 
- The new Event Sequencer mode allows one to create time-based
  events for file activity and other logs.   This allows one to
  easily sort a sequence of events during the investigation.
- The results of keyword searches are saved to a file and can be quickly
  recalled. 

Bug Fixes:
- calc_md5() would error if it was called more than once (Paul Bakker)
- Added 'LANG=C LC_ALL=C' to sorter and mactime to reduce the UTF-8
  warning messages (debugging help from Daniel Schwartzer).
- The timeline view now allows multiple users for a UID (reported
  by Cathy Buckman).


http://www.sleuthkit.org/autopsy/index.php
http://sleuthkit.sourceforge.net/autopsy/index.php

[sleuthkit-users] opensourceforensics.org

[Brian C. <ca...@sl...>]

Hey all,

I setup a new website that is a reference for finding open source forensics tools and papers.  The goal is to have a central site to easily find new tools.  I'm SURE that I have missed a few of them, so if there are any that you use that are not on there, then let me know.  

    http://www.opensourceforensics.org

Note that there are no bootable Linux CDs on the site yet.  That is because all of the ones that I know of do not include source code. They use an open source OS, but only the ISO is available from the website.  
  
brian

Re: [sleuthkit-users] Best way to multi-keyword search with Autopsy Regex?

[Brian C. <ca...@sl...>]

On 05 Jun 2003 03:05 PDT you wrote:

> I'm looking for the best way to regex search with Autopsy for two 
> disjoint words.
> 
> In other words, I am looking for the appearance of two names in a given 
> sector, i.e. Bob and Mike

You can't restrict yourself to one sector.  Autopsy will first run strings on the image and then 'grep' the strings output.  So, at best you can only search for the two words within the same string.  

> What would be the best way to do this?
...
 
> [[Mike|mike](.*)[Bob|bob]|[Bob|bob](.*)[Mike|mike]]

You don't want to use the '[' because that tells grep to use any char in between the next ']'.  You could maybe use the following:
 ((([Mm]ike)(.*)([Bb]ob))|(([Bb]ob)(.*)([Mm]ike)))

Typically (.*) will gobble up as much as possibe though, so it maynot be the best choice.  In Perl, (.*?) would be used to minimize how much it uses.  I don't know of the equivalent in grep. 

You can't use this in Autopsy, but i would use the following from the command line on the strings file:

grep -E '([Mm]ike)' img.dd.str | grep -E '([Bb]ob)'

you'll have to manually calculate which block the hit is in, but it will give you a quick overview of whether the words exist.

I've included a 'grep cheat sheet' in the next version of Autopsy and am looking for feedback on the most important things to have documented there.  Send me anything people want to  see.

good luck

brian

Re: [sleuthkit-users] NTFS ADS

[Brian C. <ca...@sl...>]

On 05 Jun 2003 02:54 PDT you wrote:

> Hi all,
> are NTFS ADS implemented in the ntfs driver from task? How can I detect
> those NTFS streams in Linux? Thanks!

Yes, all NTFS attributes can be seen with The Sleuth Kit.  An ADS is simply a second $Data attribute for a file or directory and you can view the contents of any file attribute (i.e. the FILE_NAME or STANDARD_INFORMATION attribute).  

An ADS in The Sleuth Kit has the format of 'file_name:ads_name'.  So, if you are using the command line, you can use the following to see just the ADS:

fls -f ntfs -rp img.dd | grep ":"

Autopsy does not have any way to extract just the ADS, but they are shown in the parent directory.  

brian

[sleuthkit-users] Best way to multi-keyword search with Autopsy Regex?

[Matthew M. S. <mm...@ta...>]

To all-


I'm looking for the best way to regex search with Autopsy for two 
disjoint words.

In other words, I am looking for the appearance of two names in a given 
sector, i.e. Bob and Mike

What would be the best way to do this?

I've tried

[Bob|bob](.*)[Mike|mike]

granted, if the names come in reverse I'm not going to get anything, so 
perhaps..

[[Mike|mike](.*)[Bob|bob]|[Bob|bob](.*)[Mike|mike]]


Or am I completely going the wrong direction here?


Thanks!

M

[sleuthkit-users] NTFS ADS

[David B. <to...@so...>]

Hi all,
are NTFS ADS implemented in the ntfs driver from task? How can I detect
those NTFS streams in Linux? Thanks!

Re: [sleuthkit-users] error running timeline in solaris 9 with s

[Brian C. <ca...@dm...>]

Oops.  The regular expression in Autopsy was not expecting to have multiple users for the same UID (the root/johnny string).  Replace base/autopsyfunc.pm.base with the one attached and re-make autopsy (you can keep the same config file though to save the questions).  

thanks,
brian


On 04 Jun 2003 12:08 PDT you wrote:

> Hi. I have seen issues on this list for mactimes with Redhat 8, but have not
> seen anyone
> write about any errors with the timeline function in Solaris 9.
> 
> The timeline will run, and some output will be generated, but most of the
> output looks like:
> Error parsing string: Mon Jul 9 2002 16:46:02 16020 .a. -/-rw-r--r--
> root/johnny bon 113477 /usr/sbin/ftpd
> 
> Any ideas on how I can fix this? I am running perl 5.6.1, which is shipped
> with solaris 9.
> Thanks.
> Cathy
> buc...@dc...

[sleuthkit-users] error running timeline in solaris 9 with sleuthkit 1.61 and autop sy 1.71

[Buckman, C. <Buc...@dc...>]

Hi. I have seen issues on this list for mactimes with Redhat 8, but have not
seen anyone
write about any errors with the timeline function in Solaris 9.

The timeline will run, and some output will be generated, but most of the
output looks like:
Error parsing string: Mon Jul 9 2002 16:46:02 16020 .a. -/-rw-r--r--
root/johnny bon 113477 /usr/sbin/ftpd

Any ideas on how I can fix this? I am running perl 5.6.1, which is shipped
with solaris 9.
Thanks.
Cathy
buc...@dc...

Re: [sleuthkit-users] Future of indexing in Autopsy and Sleuthkit

[Matthew M. S. <mm...@ta...>]

Paul Bakker wrote:

> 
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Hello,
>
>As some people may already know, I am in the process of adding an Indexed Search feature to Autopsy and Sleuthkit, which are Open Source filesystem forensic tools.
>
>I have some issues that concern these additions and I would like to get community members' opinions on some of these. So anyone who is using Autopsy/Sleuthkit or just wants to give his/her opinion: Feel free to give your opinion and let me know if I should or should not implement these features/issues.
>
>Issue 1:
>I think it is advisable to limit the indexed character range to only alphanumeric characters instead of the current limitation of all printable ASCII characters. The consequences are the following:
> - POSITIVE: The size of the used index files is smaller (Now it's the size of the strings file of an image) Which is quite huge if you have just copied a 80 Gb partition.
> - NEGATIVE: Indexed Searching on other characters will not be possible anymore.
> - POSITIVE: It will be easier to search for substrings of words, which is not yet possible at the moment. (It is possible in both versions, but will take a huge extra space if used on the original charachter range)
> - POSITIVE: Searching will be even quicker.
>  
>
Paul, is it just me, or do I read that as alphanumeric only? I often 
need to search for instances of email addresses, and while it is not 
always mandatory, having access to the @ symbol sure does speed the 
process up.

>Issue 2:
>Human readability of the files. A speedup in the indexed searching process and a redeuction of the size of the used files can be accomplished by changing the format of the index files. The consequence is that these cannot be read by a human anymore (No more text-format file). The consequences are the following:
> - POSITIVE: Speed of searches is increased
> - POSITIVE: Size of used files is reduces
> - NEGATIVE: Files cannot be checked anymore with the human eye.
>
>For the moment this are the issues. Maybe more will come..
>
>  
>
Not an issue in my opinion, in fact I agree with another post that 
mentioned making the file layout open, someone here will write a tool to 
read it.



-----------------------------------------------------------------
This list is provided by the SecurityFocus ARIS analyzer service.
For more information on this free incident handling, management 
and tracking system please see: http://aris.securityfocus.com

RE: [sleuthkit-users] Future of indexing in Autopsy and Sleuthkit

[Brian C. <ca...@sl...>]

Paul Bakker <ba...@fo...> said:

> > >Issue 2:
> > >Human readability of the files. A speedup in the indexed 
> > >searching process and a redeuction of the size of the used 
> >
> > Not an issue in my opinion, in fact I agree with another post that 
> > mentioned making the file layout open, someone here will 
> > write a tool to 
> > read it.
> 
> I will do both. I will document the file format and provide a tool to
> convert it to human readable format.

Perfect.  One of the goals of Autopsy is that all of its data and configuration files are 
open so that any tool can utilize them and one is not restricted to Autopsy if (s)he 
starts with it.  Maybe we can eventully do some Sleuth Kit Informer articles on the 
format and design ...

I would actually say to keep it in text for the initial versions so that people can verify 
it, feel comfortable with it, and debug any issues.  It can be optimized later.

thanks,
brian

RE: [sleuthkit-users] Future of indexing in Autopsy and Sleuthkit

[Paul B. <ba...@fo...>]

Hi Matthew,

Thanks for your response.

> Paul, is it just me, or do I read that as alphanumeric only? I often=20
> need to search for instances of email addresses, and while it is not=20
> always mandatory, having access to the @ symbol sure does speed the=20
> process up.
I can understand your problem.... Please try to understand mine (As I =
see it). The problem with indexed searching is that you have to have a =
limited set of characters to search for. Otherwise it's not possible to =
generate an index file. The size of the index file grows exponentially =
with the size of the character set.

Therefore it might be possible to add some other characters like the =
diacrtitic ASCII characters and maybe an @ (BUt then other people want =
to have other characters too). Based on this it will probably be =
configurable in the final version.

Unicode is for me a NoNo.... Beacause of the sheer size of the set of =
characters contained therein.
If anyone can suggest a fix/solution I would greatly appreciate that!

I'm still thinking about a better solution.

You should remember though that there will always be Standard searching =
with regexp and all... Indexed searching is just to generate a speedup =
for the most commonly used search strings (Which in my opinion are the =
Alphanumeric and diacritic ASCII characters. PLEASE DEBATE WITH ME ON =
THIS!!!!!!!)

> >Issue 2:
> >Human readability of the files. A speedup in the indexed=20
> >searching process and a redeuction of the size of the used=20
>
> Not an issue in my opinion, in fact I agree with another post that=20
> mentioned making the file layout open, someone here will=20
> write a tool to=20
> read it.

I will do both. I will document the file format and provide a tool to
convert it to human readable format.

--
Paul Bakker

Fox-IT Experts in IT Security!
Haagweg 137=20
2281 AG RIJSWIJK=20
T 070 336 9999=20
F 070 336 9990=20
I www.fox-it.com=20
E ba...@fo...
57A6 C5EA 55E4 CC1C A967 B13C F8C0 C0FB 8135 E225

Disclaimer: This email may contain confidential information. If this =
message is not addressed to you, you may not retain or use the =
information in it for any purpose. If you have received it in error, =
please notify the sender and delete this message. We try to screen out =
viruses but take no responsibility if this email contains a virus.

[sleuthkit-users] RE: Future of indexing in Autopsy and Sleuthkit

[Paul B. <ba...@fo...>]

Hi Simson,

Thanks for the response

> If you limit to printable ASCII characters, there will be=20
> problems for=20
> people outside the US (or people working with data outside=20
> the US). You=20
> need to be able to handle roman characters with accents. These are=20
> normally represented with high-bits. If the user searches for an e,=20
> they probably want to match on =E8 and =E9 and possibly other e's as =
well.
>=20
> Then you have the issue of Arabic, Hebrew, and 16-bit characters.
>=20
> At a minimum, I think that you should transparently handle codepages=20
> and coerce them into 7-bit ASCII. But ideally you should handle=20
> UNICODE, UTF-8, UTF-16, etc. Or do something for Arabic.

OK.. The problem with indexed searching is that you have to have a =
limited
set of characters to search for. Otherwise it's not possible to generate
an index file. The size of the index file grows exponentially with the =
size
of the character set.

That said I will possibly add the diacritic ASCII characters, but =
Unicode contains
way to much characters. Therefore Unicode poses a problem....

If anyone can suggest a fix/solution I would greatly appreciate that!

I'm still thinking about a better solution.


--
Paul Bakker

Fox-IT Experts in IT Security!
Haagweg 137=20
2281 AG RIJSWIJK=20
T 070 336 9999=20
F 070 336 9990=20
I www.fox-it.com=20
E ba...@fo...
57A6 C5EA 55E4 CC1C A967 B13C F8C0 C0FB 8135 E225

Disclaimer: This email may contain confidential information. If this =
message is not addressed to you, you may not retain or use the =
information in it for any purpose. If you have received it in error, =
please notify the sender and delete this message. We try to screen out =
viruses but take no responsibility if this email contains a virus.

[sleuthkit-users] Re: Future of indexing in Autopsy and Sleuthkit

[Matt B. <MB...@st...>]

If you are feeling ambitious, why not give the option to the user. Take
the suggestions you receive from this list to determine the default
behavior of the application, and then give the user the option of
changing that behavior if desired. In my opinion, one of the largest
benefits to using open source software is its flexibility.


Matt Bergen
Lead Information Security Officer
Wyoming Department of Employment

>>> "Simson L. Garfinkel" <si...@lc...> 05/22/03 09:27AM >>>
Paul,

Here are some issues you may not have considered:
>
> Issue 1:
> I think it is advisable to limit the indexed character range to only

> alphanumeric characters instead of the current limitation of all=20
> printable ASCII characters.

If you limit to printable ASCII characters, there will be problems for

people outside the US (or people working with data outside the US). You

need to be able to handle roman characters with accents. These are=20
normally represented with high-bits. If the user searches for an e,=20
they probably want to match on =E8 and =E9 and possibly other e's as well.

Then you have the issue of Arabic, Hebrew, and 16-bit characters.

At a minimum, I think that you should transparently handle codepages=20
and coerce them into 7-bit ASCII. But ideally you should handle=20
UNICODE, UTF-8, UTF-16, etc. Or do something for Arabic.
>
> Issue 2:
> Human readability of the files. A speedup in the indexed searching=20
> process and a redeuction of the size of the used files can be=20
> accomplished by changing the format of the index files. The=20
> consequence is that these cannot be read by a human anymore (No more

> text-format file). The consequences are the following:
>  - POSITIVE: Speed of searches is increased
>  - POSITIVE: Size of used files is reduces
>  - NEGATIVE: Files cannot be checked anymore with the human eye.

I do not think that this is important. The index files should be in=20
binary; create a tool to browse or view them.


-----------------------------------------------------------------
This list is provided by the SecurityFocus ARIS analyzer service.
For more information on this free incident handling, management
and tracking system please see: http://aris.securityfocus.com=20