sleuthkit-developers Mailing List for The Sleuth Kit (Page 39)
Brought to you by:
carrier
Menu
▾
▴
sleuthkit-developers — A list to discuss development of new features of The Sleuth Kit and Autopsy
You can subscribe to this list here.
| 2003 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(10) |
Sep
(2) |
Oct
|
Nov
(1) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2004 |
Jan
(22) |
Feb
(39) |
Mar
(8) |
Apr
(17) |
May
(10) |
Jun
(2) |
Jul
(6) |
Aug
(4) |
Sep
(1) |
Oct
(3) |
Nov
|
Dec
|
| 2005 |
Jan
(2) |
Feb
(6) |
Mar
(2) |
Apr
(2) |
May
(13) |
Jun
(2) |
Jul
|
Aug
|
Sep
(5) |
Oct
|
Nov
(2) |
Dec
|
| 2006 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
(2) |
Jun
(9) |
Jul
(4) |
Aug
(2) |
Sep
|
Oct
(1) |
Nov
(9) |
Dec
(4) |
| 2007 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
(3) |
May
|
Jun
|
Jul
(6) |
Aug
|
Sep
(4) |
Oct
|
Nov
|
Dec
(2) |
| 2008 |
Jan
(4) |
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
(9) |
Jul
(14) |
Aug
|
Sep
(5) |
Oct
(10) |
Nov
(4) |
Dec
(7) |
| 2009 |
Jan
(7) |
Feb
(10) |
Mar
(10) |
Apr
(19) |
May
(16) |
Jun
(3) |
Jul
(9) |
Aug
(5) |
Sep
(5) |
Oct
(16) |
Nov
(35) |
Dec
(30) |
| 2010 |
Jan
(4) |
Feb
(24) |
Mar
(25) |
Apr
(31) |
May
(11) |
Jun
(9) |
Jul
(11) |
Aug
(31) |
Sep
(11) |
Oct
(10) |
Nov
(15) |
Dec
(3) |
| 2011 |
Jan
(8) |
Feb
(17) |
Mar
(14) |
Apr
(2) |
May
(4) |
Jun
(4) |
Jul
(3) |
Aug
(7) |
Sep
(18) |
Oct
(8) |
Nov
(16) |
Dec
(1) |
| 2012 |
Jan
(9) |
Feb
(2) |
Mar
(3) |
Apr
(13) |
May
(10) |
Jun
(7) |
Jul
(1) |
Aug
(5) |
Sep
|
Oct
(3) |
Nov
(19) |
Dec
(3) |
| 2013 |
Jan
(16) |
Feb
(3) |
Mar
(2) |
Apr
(4) |
May
|
Jun
(3) |
Jul
(2) |
Aug
(17) |
Sep
(6) |
Oct
(1) |
Nov
|
Dec
(4) |
| 2014 |
Jan
(2) |
Feb
|
Mar
(3) |
Apr
(7) |
May
(6) |
Jun
(1) |
Jul
(18) |
Aug
|
Sep
(3) |
Oct
(1) |
Nov
(26) |
Dec
(7) |
| 2015 |
Jan
(5) |
Feb
(1) |
Mar
(2) |
Apr
|
May
(1) |
Jun
(1) |
Jul
(5) |
Aug
(7) |
Sep
(4) |
Oct
(1) |
Nov
(1) |
Dec
|
| 2016 |
Jan
(3) |
Feb
|
Mar
(1) |
Apr
|
May
(1) |
Jun
(13) |
Jul
(23) |
Aug
(2) |
Sep
(11) |
Oct
|
Nov
(1) |
Dec
|
| 2017 |
Jan
(4) |
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(2) |
Dec
|
| 2018 |
Jan
|
Feb
|
Mar
(2) |
Apr
|
May
(1) |
Jun
(3) |
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
(2) |
Dec
|
| 2019 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2020 |
Jan
(4) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
(5) |
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
| 2024 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(1) |
|
From: Brian C. <ca...@sl...> - 2004-04-04 15:49:52
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 2, 2004, at 6:00 PM, Linux Tard wrote: > But what I > like more so is ability to study volume for supported > types to recover deleted files without me manually > doing so. File recovery in the data carving (i.e. foremost) sense or recovery using file system structures (i.e. norton undelete-like)? brian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFAcC6lOK1gLsdFTIsRAiRuAKCDcJPqupUWDoFCc9Ord74bM1bnzACfePRb EpB4jQx4Y6nNlBimgpg//Uk= =z9OX -----END PGP SIGNATURE----- |
|
From: Linux T. <lin...@ya...> - 2004-04-02 23:00:56
|
I saved my nickels and finally purchased SMART and glad to do so. It does support filesystems in that if your Linux system supports a FS TYPE SMART support it and allow you to mount it and access it. But what I like more so is ability to study volume for supported types to recover deleted files without me manually doing so. Yes, not free but very worth price, IMO. -lt --- Márcio Carneiro <ma...@di...> wrote: > On Fri, 2 Apr 2004 09:43:32 -0500, Brian Carrier > <ca...@sl...> escreveu: > > > > Great. If you send me the link, I'll add it to the > links page. > > Oh, sorry, for now it's in portuguese. But, it's > nothing big... > > > I know of people who have talked about > implementing Reiser, but nothing > > has been done as far as I know. There is someone > working on JFS. I > > don't think I've heard of anyone doing XFS. I > don't have plans to add > > these in the near future. I'm actually more > inclined to add HFS+ > > before any of the Linux-based ones. > > Mmmm, HFS+ is a good one! > > I saw that SMART should read Reiser, XFS, and others > (http://www.securitywizardry.com/fortoolkits.htm): > > "SMART can acquire digital evidence from a wide > variety of workstations, servers and digital > devices. SMART authenticates the data it acquires > using any or all of the CRC32, MD5SUM and SHA1 > algorithms. SMART also provides for the compression > of data using standard Gzip or BZ2 compression, as > well as a seekable compression format. SMART > "understands" many file systems, including VFAT, > NTFS, ext2, ext3, Reiser, HFS, HFS+, XFS, JFS, > ISO9660, BeFS and many more. SMART can recover > deleted files from these file systems and interpret > file system meta-data such as date and time stamps, > file attributes, etc. SMART enables complex searches > to be conducted quickly and easily. Full GREP > syntax, intelligent rules based options and fully > automated recovery are possible without scripting or > programming." > I got a SMART demo once, but had no time to test > it... :-/ And, it's commercial... :-( > > Abraços, > > Márcio. > __________________________________ Do you Yahoo!? Yahoo! Small Business $15K Web Design Giveaway http://promotions.yahoo.com/design_giveaway/ |
|
From: Brian C. <ca...@sl...> - 2004-04-02 21:18:05
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > I saw that SMART should read Reiser, XFS, and others > (http://www.securitywizardry.com/fortoolkits.htm): According to that page, TASK was written by Vern Paxson! I didn't realize that. > SMART "understands" many file systems, including VFAT, NTFS, ext2, > ext3, Reiser, HFS, HFS+, XFS, JFS, ISO9660, BeFS and many more. SMART uses a combination of the local Linux kernel file system support and custom code. It mounts images in loopback using the standard Linux file systems and then has custom modules for the deleted files and such. brian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFAbdiHOK1gLsdFTIsRAisJAJ9fFyC6MrUZydZYUu/YJ3a001JREwCbBtDG 4acWufJFQuOBbFjxNrJZZEY= =zfXb -----END PGP SIGNATURE----- |
|
From: knut.eckstein <knu...@he...> - 2004-04-02 19:22:59
|
SGksDQogDQo+IEJ1dCwgSSBoYXZlIGEgaXNzdWUgYWJvdXQgb3RoZXIgZmlsZXN5c3RlbSB0aGFu IGV4dHsyLDN9LiANCj4gRG9lcyBhbnlib2R5IGtub3dzIHdoYXQgYXJlIHRoZSBhdmFpbGFibGUg dG9vbHMgZm9yIA0KPiBmb3JlbnNpYyBhbmFseXNpcyBpbiBSZWlzZXIgYW5kIFhGUyBmaWxlc3lz dGVtcz8gDQo+IEFyZSB0aGVyZSBwbGFucyB0byBhZGQgdGhpcyB0byBTbGV1dGggS2l0Pw0KDQpF bkNhc2UgKHd3dy5lbmNhc2UuY29tKSBzdXBwb3J0cyBSZWlzZXJGUywgYnV0IHRoYXQncw0KY29t bWVyY2lhbCB0b28sIGFyb3VuZCAkMzAwMCBBRkFJSy4NCg0KQmVzdCByZWdhcmRzLA0KDQpLbnV0 DQoNCg0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tDQpUaGlzIFNGLk5ldCBlbWFpbCBpcyBzcG9uc29yZWQgYnk6IElCTSBMaW51eCBUdXRv cmlhbHMNCkZyZWUgTGludXggdHV0b3JpYWwgcHJlc2VudGVkIGJ5IERhbmllbCBSb2JiaW5zLCBQ cmVzaWRlbnQgYW5kIENFTyBvZg0KR2VuVG9vIHRlY2hub2xvZ2llcy4gTGVhcm4gZXZlcnl0aGlu ZyBmcm9tIGZ1bmRhbWVudGFscyB0byBzeXN0ZW0NCmFkbWluaXN0cmF0aW9uLmh0dHA6Ly9hZHMu b3Nkbi5jb20vP2FkX2lkPTE0NzAmYWxsb2NfaWQ9MzYzOCZvcD1jbGljaw0KX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCnNsZXV0aGtpdC1kZXZlbG9wZXJz IG1haWxpbmcgbGlzdA0Kc2xldXRoa2l0LWRldmVsb3BlcnNAbGlzdHMuc291cmNlZm9yZ2UubmV0 DQpodHRwczovL2xpc3RzLnNvdXJjZWZvcmdlLm5ldC9saXN0cy9saXN0aW5mby9zbGV1dGhraXQt ZGV2ZWxvcGVycw0KDQo= |
|
From: Márcio C. <ma...@di...> - 2004-04-02 17:44:18
|
On Fri, 2 Apr 2004 09:43:32 -0500, Brian Carrier <ca...@sl...> escreveu: > > Great. If you send me the link, I'll add it to the links page. Oh, sorry, for now it's in portuguese. But, it's nothing big... > I know of people who have talked about implementing Reiser, but nothing > has been done as far as I know. There is someone working on JFS. I > don't think I've heard of anyone doing XFS. I don't have plans to add > these in the near future. I'm actually more inclined to add HFS+ > before any of the Linux-based ones. Mmmm, HFS+ is a good one! I saw that SMART should read Reiser, XFS, and others (http://www.securitywizardry.com/fortoolkits.htm): "SMART can acquire digital evidence from a wide variety of workstations, servers and digital devices. SMART authenticates the data it acquires using any or all of the CRC32, MD5SUM and SHA1 algorithms. SMART also provides for the compression of data using standard Gzip or BZ2 compression, as well as a seekable compression format. SMART "understands" many file systems, including VFAT, NTFS, ext2, ext3, Reiser, HFS, HFS+, XFS, JFS, ISO9660, BeFS and many more. SMART can recover deleted files from these file systems and interpret file system meta-data such as date and time stamps, file attributes, etc. SMART enables complex searches to be conducted quickly and easily. Full GREP syntax, intelligent rules based options and fully automated recovery are possible without scripting or programming." I got a SMART demo once, but had no time to test it... :-/ And, it's commercial... :-( Abraços, Márcio. |
|
From: Paul B. <ba...@fo...> - 2004-04-02 15:26:12
|
> I know of people who have talked about implementing Reiser,=20 > but nothing=20 > has been done as far as I know. There is someone working on JFS. I=20 > don't think I've heard of anyone doing XFS. I don't have=20 > plans to add=20 > these in the near future. I'm actually more inclined to add HFS+=20 > before any of the Linux-based ones. That would be very much appreciated on my side ;-).. I just had two cases with iMacs... And HFS+.... had to do a lot by hand again... Paul Bakker=20 |
|
From: Brian C. <ca...@sl...> - 2004-04-02 14:43:49
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Apr 2, 2004, at 8:09 AM, MXrcio Carneiro wrote:
> Hi!
>
> I'm preparing a course about Linux Forensics (mainly about file
> system). Of course Sleuth and Autopsy is the main tool (I want to show
> FLAG, but I'm having a hard time to install it - maybe should look for
> the Live CD?).
Great. If you send me the link, I'll add it to the links page.
> But, I have a issue about other filesystem than ext{2,3}. Does anybody
> knows what are the available tools for forensic analysis in Reiser and
> XFS filesystems? Are there plans to add this to Sleuth Kit?
I know of people who have talked about implementing Reiser, but nothing
has been done as far as I know. There is someone working on JFS. I
don't think I've heard of anyone doing XFS. I don't have plans to add
these in the near future. I'm actually more inclined to add HFS+
before any of the Linux-based ones.
brian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
iD8DBQFAbXwaOK1gLsdFTIsRAtviAJ9FW/T/oH23Z5w+eiSshfGvf4GvDACeOpzf
tautD44QmPwQaEW5Crz7wqw=
=6akc
-----END PGP SIGNATURE-----
|
|
From: Márcio C. <ma...@di...> - 2004-04-02 13:19:27
|
Hi!
I'm preparing a course about Linux Forensics (mainly about file system). Of course Sleuth and Autopsy is the main tool (I want to show FLAG, but I'm having a hard time to install it - maybe should look for the Live CD?).
But, I have a issue about other filesystem than ext{2,3}. Does anybody knows what are the available tools for forensic analysis in Reiser and XFS filesystems? Are there plans to add this to Sleuth Kit?
Thanks in advance.
Márcio.
|
|
From: Brian C. <ca...@sl...> - 2004-03-11 03:41:17
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ok, bug fixed. Stupid error using an unsigned variable. The same problem existed in the FFS code and is now fixed. The new versions can be found here: sleuthkit.sf.net/sleuthkit/ffs.c sleuthkit.sf.net/sleuthkit/ext2fs.c One of the things that I forgot on the list of goals for v2 (and Knut Eckstein reminded me of) is that I want to go through the code and get rid of all of the uses of size_t and addr_t type variables and move to int32_t and u_int32_t because the sizes and signs for the different platforms is too confusing. brian On Mar 10, 2004, at 4:06 PM, Epsilon wrote: > --- Brian Carrier <ca...@sl...> wrote: >> >> On Feb 18, 2004, at 1:58 PM, Epsilon wrote: >> >>> I'm getting a very large (>500 MB) file when using the -s option >> with >>> icat when I should be getting a file that's around 40 KB. I'm >> using >>> sleuthkit-1.67. Anyone else seeing this? >> >> Wow. What file system type? Can you send the output of running >> 'istat' on it? > > OK, I've been meaning to respond to this for a while. I'm now using > sleuthkit-1.68 under Fedora Core 1 with latest patches applied. I'm > using the honeypot.hda5.dd image from here: > > http://honeynet.org/misc/files/challenge-images.tar > > And here's the exact command I'm running: > > $ ./icat -s -f linux-ext2 honeypot.hda5.dd 1604 > inode-1604-all.out > > After about 5 seconds I ^C it and run icat w/o the -s: > > $ ./icat -f linux-ext2 honeypot.hda5.dd 1604 > inode-1604-data.out > > Look at the results: > > $ ls -l *.out > -rw-r--r-- 1 ep users 141107200 Mar 10 16:01 inode-1604-all.out > -rw-r--r-- 1 ep users 119671 Mar 10 16:01 inode-1604-data.out > > I'm expecting to see inode-1604-all.out to be 122880 bytes in size > (4096 * 30 clusters). Is this a wrong assumption? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFAT9ucOK1gLsdFTIsRAqn2AJ0U0L/JA/AxZ+dl2Vl5n6uRjLXDSwCePJx4 qyTQvjU7ZF2QRhEwkF0qzVA= =mGQ8 -----END PGP SIGNATURE----- |
|
From: Epsilon <ep...@ya...> - 2004-03-10 21:24:24
|
--- Brian Carrier <ca...@sl...> wrote: > > On Feb 18, 2004, at 1:58 PM, Epsilon wrote: > > > I'm getting a very large (>500 MB) file when using the -s option > with > > icat when I should be getting a file that's around 40 KB. I'm > using > > sleuthkit-1.67. Anyone else seeing this? > > Wow. What file system type? Can you send the output of running > 'istat' on it? OK, I've been meaning to respond to this for a while. I'm now using sleuthkit-1.68 under Fedora Core 1 with latest patches applied. I'm using the honeypot.hda5.dd image from here: http://honeynet.org/misc/files/challenge-images.tar And here's the exact command I'm running: $ ./icat -s -f linux-ext2 honeypot.hda5.dd 1604 > inode-1604-all.out After about 5 seconds I ^C it and run icat w/o the -s: $ ./icat -f linux-ext2 honeypot.hda5.dd 1604 > inode-1604-data.out Look at the results: $ ls -l *.out -rw-r--r-- 1 ep users 141107200 Mar 10 16:01 inode-1604-all.out -rw-r--r-- 1 ep users 119671 Mar 10 16:01 inode-1604-data.out I'm expecting to see inode-1604-all.out to be 122880 bytes in size (4096 * 30 clusters). Is this a wrong assumption? TIA, ep __________________________________ Do you Yahoo!? Yahoo! Search - Find what youre looking for faster http://search.yahoo.com |
|
From: Brian C. <ca...@sl...> - 2004-03-08 15:11:19
|
FYI: perl2exe seems to work well with Autopsy, although I had to use the eval version, the full version is $150, and it expires in 30 days. brian On Mar 6, 2004, at 6:10 PM, Brian Carrier wrote: > Anyone have experience with "safely" running Perl from a CD? My goal > is to have the equivalent to a statically compiled version of Perl on > a CD for the incident response features of Autopsy. |
|
From: Brian C. <ca...@sl...> - 2004-03-06 23:25:37
|
Anyone have experience with "safely" running Perl from a CD? My goal is to have the equivalent to a statically compiled version of Perl on a CD for the incident response features of Autopsy. I'm assuming I can either copy the needed Perl libraries to the CD along with the perl executable, or require people to compile a static version of Perl (which I haven't played with yet). The former seems the more flexible since not all people have the source for Perl. Has anyone looked into this type of thing before? brian |
|
From: Brian C. <ca...@sl...> - 2004-03-02 16:48:06
|
On Mar 2, 2004, at 8:38 AM, Paul Bakker wrote: > I wrote a small patch for Sleuthkit 1.67 that makes it possible to use > Raw fs in a generic manner (Similar to fatfs, extfs, etc) This is a really good idea. I can't believe I didn't do this earlier. It is much better than the hacks that I had for adding support for swap and raw into the 'dcat' tool. This is a real easy patch to work in and I'll put this and a swap one into the next release. thanks, brian |
|
From: Brian C. <ca...@sl...> - 2004-03-02 16:45:06
|
On Mar 2, 2004, at 5:26 AM, Michael Cohen wrote: > Hi Brian, > I tried to compile the sleuthkit on an opteron today with negative > outcomes. The first problem is the _llseek function defined in > myseek.c is > not compatible for the opteron since OFF_T is already 64bits, a simple > lseek > can be substituted for myseek(). I think OFF_T is always 64-bits on new Linux kernels, even if you have a 32-bit processor. I didn't realize that the seek function was fixed though (I hadn't even thought of this problem until you mentioned it actually). See if there is a compile time variable that is set for 64-bit hardware and use that in fs_tools.h to set the USE_MYLSEEK, HAVE_LLSEEK, and LSEEK variables. > ( Yeah i know the comment just above the code said: > /* > * This is LINUX, live on the bleeding edge and watch your software > break > * with the next release... > */ > Looks like this is the time when things break) This is all legacy TCT stuff and I've actually never looked into the details of the Linux seek system calls. > Secondly, the code seems to be peppered with things like: > fs_dent->path = (char *)(int)begin + 1; > where begin is char *. > I am not sure what the purpose of the double cast here is but the > compiler > spits out lots of warnings, and the program crashes at random places > (possibly due to this kind of casts). Interesting. The reason is that we want the address contents of 'begin' and some compilers will complain if you do not first mask it to an int before adding 1 to it. What warnings do you get? It could be because the address is actually a 64-bit value, but the int is still only a 32-bit value so it is truncated.... I guess the code could be written as (char *)( (int)begin + 1); I'm not sure if that would remove your warnings, but there still could be the 32 vs 64-bit address problems. > Are there plans to make the code 64 bit safe? and what is the best way > to > do so? I have no clue. I have never looked into this before. I did have plans to go through the TSK code before v2 is released and get rid of the use of OFF_T and ADDR_T values and replace them with u_int32 or int64_t variables so that the code is more portable across platforms that have different sizes for off_t or addr_t. Maybe other changes will also need to occur. I don't have access to such a system, so I can't do much for testing or debugging :( brian |
|
From: Paul B. <ba...@fo...> - 2004-03-02 13:51:54
|
Brian, I wrote a small patch for Sleuthkit 1.67 that makes it possible to use = Raw fs in a generic manner (Similar to fatfs, extfs, etc) It makes it possible to use fs_open() to open raw images... Most = functions will return error if called on it, but this way external tools (Like searchtools) do not need to make exceptions to use raw file = systems... Maybe something similar should be done for swap (Perhaps it = can also use rawfs)... It is small and first version, but it works for me (what I need at the = moment is only access to an FS_INFO object with the size inside and thus = the possiblility to call fs_read_random).. Could you integrate this patch in Sleuthkit 1.68 and expand it where = necessary? Paul Bakker |
|
From: Michael C. <mic...@ne...> - 2004-03-02 10:39:22
|
Hi Brian, I tried to compile the sleuthkit on an opteron today with negative outcomes. The first problem is the _llseek function defined in myseek.c is not compatible for the opteron since OFF_T is already 64bits, a simple lseek can be substituted for myseek(). ( Yeah i know the comment just above the code said: /* * This is LINUX, live on the bleeding edge and watch your software break * with the next release... */ Looks like this is the time when things break) Secondly, the code seems to be peppered with things like: fs_dent->path = (char *)(int)begin + 1; where begin is char *. I am not sure what the purpose of the double cast here is but the compiler spits out lots of warnings, and the program crashes at random places (possibly due to this kind of casts). Are there plans to make the code 64 bit safe? and what is the best way to do so? Michael. |
|
From: Brian C. <ca...@sl...> - 2004-02-25 19:02:37
|
> The major change now is that I implemented exception handling > throughout all > the libraries used by the io subsystem. I did not touch the rest of > fstools > for the moment, just until i get the general opinion from the list > about this > move, but in the long term I believe that all code should be converted > to use > exceptions, rather than calling error(). I agree. The error() functionality is a TCT legacy and I have no issues with adding exceptions. My time has been extremely limited for working on the tools, but here are my goals for the next few months. - I have most of Autopsy changed for v2 and would like to get that out in the next couple of weeks. - Start planning what v2 for The Sleuth Kit will look like. So far we have the following potential additions: - New image format support and corresponding library - indexing support - Re-examine output data of tools (such as ils) - Add exception handling Anything else I have forgotten about? brian |
|
From: Michael C. <mic...@ne...> - 2004-02-25 14:08:45
|
Hi Brian,
> I quickly looked the changes over. Is there a need to pass FS_INFO to
> the read functions? I didn't see you using them and it would be nice
> if we could avoid doing that (so that we don't have to restrict
> ourselves to file system code).
Good point, this is now fixed. read_blocks moved into the fs struct and the
io_info struct is now free from all references to fs_info.
The major change now is that I implemented exception handling throughout all
the libraries used by the io subsystem. I did not touch the rest of fstools
for the moment, just until i get the general opinion from the list about this
move, but in the long term I believe that all code should be converted to use
exceptions, rather than calling error(). Exceptions make the code cleaner and
easier to follow, and also make it possible to use this code as part of
bigger pieces of code (e.g. as a python or perl module). I really need the io
subsystem to be available as a python module for flag so I need good
exception support in the iosubsystem code.
The swig interface has also been tuned to support exceptions in an intelligent
manner and pass C exceptions to python exceptions seamlessly. Here is an
example python session:
Python 2.3.3 (#2, Jan 13 2004, 00:47:05)
[GCC 3.3.3 20040110 (prerelease) (Debian)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import iosubsys
>>> io=iosubsys.io_open("ewf")
>>> iosubsys.parse_options(io,"off=32")
Traceback (most recent call last):
File "<stdin>", line 1, in ?
RuntimeError: option off not recognised
>>> iosubsys.parse_options(io,"/storage/tmp/1.e01")
0
>>> iosubsys.read_random(io,100,0)
(100, '.... data chopped for briefness... ')
>>> iosubsys.read_random(io,100,10000000)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
IOError: Attempting to seek past the end of the file!
>>>
Here we see a python session using the iosubsys module, with an encase file of
a floppy disk. When we try to set an option that is not supported (off=32),
we immediately get a python runtime exception. This exception can be caught
and an intelligent course of action can be taken by the python environment.
Similarly we see an IOError exception raised when trying to read past the end
of file. Just out of interest, this is the C code that actually raises this
exception: (line 377 in libevf.c):
if(chunk>offsets->max_chunk) {
RAISE(E_IOERROR,NULL,"Attempting to seek past the end of the file!");
};
All the underlying libraries support exceptions. At this stage I did not
change any of the fstools themselves to handle the exceptions, so when a
similar error occurs using the fls tool for example, its unhandled and causes
the program to exit; pretty much the same behaviour as previously:
bash$ ../bin/fls -f fat -i ewf /etc/passwd
Unhandled Exception(IO Error): File format not recognised as EWF
Again the exception was raised from deep inside the library, but since it was
not caught by anything it caused an unhandled exception termination.
Thoughts anyone?
Michael
|
|
From: Paul B. <ba...@fo...> - 2004-02-24 08:49:07
|
Hi t f, > I see much fuss over indexing capabilities, but not a lot of=20 > discussion as=20 > to the purpose of said indexes. One might assume they are to=20 > be searched,=20 > but, why would one search on base64-encoded, zipped, cab'd, pdf'd, or=20 > otherwise "unreadable" data? I suppose it would be helpful=20 > for identifying=20 > certain malicious files as they lay on the drive, but I don't=20 > see too many=20 > other applications. You're absolutely right... And this will happen in the "Searchtools" implementation. But this requires "interpreters" as I said.. There has to be a generic way to interpret the data. The best way would be if this is integrated into Sleutkit. If this will not happen, release 4 of the code (The future release, not the next one) will use external interpreters to read through data and index it. > My point is this: Shouldn't there be some work toward=20 > preprocessing this=20 > data, THEN indexing the intelligible bits?? Just a=20 > suggestion. Take it for=20 > what it's worth. See above... You are right, BUT! I would still want to have an index tool show me ALL occurrences of that string I am searching for on the disk. Not only Raw (Thus uninterpreted), but also fragmented versions and versions in files that have to be interpreted (Also multiple levels deep, e.g. a PDF inside a ZIP file).... The goal of Searchtools is to be capable of supporting all these strings and thus give the maximum information! You can always use the command-line options (Or Autopsy integrated HTML page) to only index or search interpreted strings. Almost all options and behavioural aspects of the tools can be tweaked. > I have seen a great deal of excellent discussion here over=20 > the past few=20 > months. Sleuthkit is becoming a very impressive tool. I=20 > look forward to=20 > seeing where it goes over the next few months. Thanks to all=20 > for the great=20 > work! You're welcome! ;-).. I hope this answers your questions.. Paul Bakker |
|
From: t f <dor...@ho...> - 2004-02-24 00:54:45
|
Greetings, I see much fuss over indexing capabilities, but not a lot of discussion as to the purpose of said indexes. One might assume they are to be searched, but, why would one search on base64-encoded, zipped, cab'd, pdf'd, or otherwise "unreadable" data? I suppose it would be helpful for identifying certain malicious files as they lay on the drive, but I don't see too many other applications. My point is this: Shouldn't there be some work toward preprocessing this data, THEN indexing the intelligible bits?? Just a suggestion. Take it for what it's worth. I have seen a great deal of excellent discussion here over the past few months. Sleuthkit is becoming a very impressive tool. I look forward to seeing where it goes over the next few months. Thanks to all for the great work! -dorkus _________________________________________________________________ Watch high-quality video with fast playback at MSN Video. Free! http://click.atdmt.com/AVE/go/onm00200365ave/direct/01/ |
|
From: Paul B. <ba...@fo...> - 2004-02-23 15:10:31
|
Hi Michael, (Again! ;-) We should start to call on the phone! ;-)) <Snip about seek performance> > I dont find any difference really???? I thought the only=20 > overhead in a seek is=20 > a system call because the kernel would already have that in=20 > the disk cache=20 > and doesnt really need to seek the physical disk at all.=20 > Maybe its an OS=20 > thing? Im using kernel 2.4.21. You're right.. It is not as bad as it used to be... (Partly because of the way my code now works)... I will implement everything using fs_read_random ;-).. Problem solved! hehe... > Cool, so when do you actually do the reading of the blocks?=20 > Or do you just use=20 > the file_walk and inode_walk to find out if a string is in a=20 > file or out of=20 > the file without reading any blocks? See below!... All actual reading is done if I find a read fragment (Fragment is two non-sequential blocks). > > The raw mode (The real mode) will use the 64kb blocks in a > > "walking buffer" kind of way... Every time a new block is loaded, > > the last xx (25) bytes of the old block will be prepended and also > > indexed... That way no data will ever get missed... >=20 > Thats great. I also noticed (I only have the current version=20 > which is on the=20 > web site - without all the bells and wistles) that the buffer is user=20 > settable, so if seeking proves to be too much of a problem,=20 > users can just=20 > set the rolling buffer to be really large. Indeed ;-))... The default will be larger if that proves to be better!... > Cool that sounds very promising. I am looking forward to=20 > seeing the next=20 > release, in the meantime I shall play with the current=20 > release. Are there=20 > many large changes in the new release? The largest change is the support for fragmented strings (Strings located on two non sequential blocks). Furthermore the internal format has changed so almost twice the amount of "data" can be stored in memory... On disk only a small 15% increase has been booked (Storage-wise), but becuase more can be stored in memory, less redundant parts are stored, so the total profit on disk storage can be around = 33%.. Storage has also changed in the way that you only have to specify a directory and the rest will be handled (So no more specifying a config = file, and a index file (Or multiple index files))... Version support to recognize older Index files and not blindly use them. A few handy tools for checking index files and such.... Owh.. And I'm currently busy changing my code to use the fstools = fs_read_random() function and not use libc's fread() ;-) Paul Bakker |
|
From: Paul B. <ba...@fo...> - 2004-02-23 15:02:43
|
> Actually, I'm not even sure why fs_read_random doesn't use the=20 > fs->read_pos value before it does the seek. My thoughts exactly ;-) |
|
From: Brian C. <ca...@sl...> - 2004-02-23 14:31:30
|
On Feb 23, 2004, at 3:03 AM, Paul Bakker wrote: >> I quickly looked the changes over. Is there a need to pass >> FS_INFO to >> the read functions? I didn't see you using them and it would be nice >> if we could avoid doing that (so that we don't have to restrict >> ourselves to file system code). > > Well I do see advantages.... I already wanted to ask this... > > The problem with the current code is that it is not possible to > "read_random" an image efficiently because it cannot check the current > offset in the image.. This results in unnecessary seeks.. And seeks are > very expensive if they come in millions.... We can easily fix that though with some data in an IMG_INFO struct, which is a more appropriate location than the FS_INFO structure. For the code that uses split or RAID images, we will need multiple copies of the current offset data. Actually, I'm not even sure why fs_read_random doesn't use the fs->read_pos value before it does the seek. brian |
|
From: Michael C. <mic...@ne...> - 2004-02-23 14:20:21
|
> Hi Michael,
Hi Paul,
> That's what I do.. (Only with a 64kb buffer)... But the current
> read_random, will do a seek every time.. And that is not good...
> (Time/Seekwise)...
Paul, Im not sure if I understand you right here - you claim that sequential
reads are slower than seek and then reads to the same position? I wasnt sure
about this and so I tested it (reading 100 byte lots out of 1gb so there is
no disk cache):
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <fcntl.h>
int main() {
int i;
int fd;
char buf[100];
fd=open("/var/tmp/honeypot.hda5.dd",O_RDONLY);
for(i=0;i<1000000000;i+=100) {
lseek(fd,i,SEEK_SET);
if(read(fd,buf,100)<100) {
printf("Read failed");
};
};
};
First I ran with the seek in there (the seek doesnt really do anything - its
only seeking to the same spot it would be in anyway - which i think is what
you mean in this situation):
$ time ./test
real 0m27.228s
user 0m2.060s
sys 0m9.250s
And without the seek (just commented out):
$ time ./test
real 0m27.000s
user 0m1.340s
sys 0m8.770s
I dont find any difference really???? I thought the only overhead in a seek is
a system call because the kernel would already have that in the disk cache
and doesnt really need to seek the physical disk at all. Maybe its an OS
thing? Im using kernel 2.4.21.
> The size is adaptable and will not change the concept.... Only the
> implementation of the current functions do not allow me to read the data
> of an image without them performing any seeks..
>
> > So for example suppose you have a file thats 30 blocks big
> > (~120kb). While the
> > file_walk might call the callback 30 times, (once for each
> > block), In the end
> > David's print_blocks function will print a single entry for
> > 30 consecutive
> > blocks.
>
> OK this might be handy, but will only change how many times the
> callback is called.. No data in the blocks is read on calling
> the callback (Because of the FS_FLAG_AONLY flag).
Cool, so when do you actually do the reading of the blocks? Or do you just use
the file_walk and inode_walk to find out if a string is in a file or out of
the file without reading any blocks?
> See above.. Icat does read every block, but the FS_FLAG_AONLY flag
> makes it NOT read the data in the block, so no seeks will happen
> there.
> > The other problem I can think about (again, I havent seen
> > your code yet so im
> > sorry if im talking crap here), is that if you only do the
> > indexing on each
> > buffer returned by the file_walk callback, then wouldnt you
> > be missing words
> > that happen to be cut by the block boundary? i.e. half a word
> > will be on one
> > block and the other half on the next block? This problem will
> > be alleviated
> > to some extent by indexing larger buffers than blocksizes.
>
> You're talking crap! ;-)) Joking..
> No Michael.. This will not happen.. The raw_fragment mode only
> indexes sdtrings in the fragmented part (Thus 10 bytes before and
> 10 bytes after the fragmented part) (Or 25... Depends.. hehe)
Cool, thats great!!!
> The raw mode (The real mode) will use the 64kb blocks in a
> "walking buffer" kind of way... Every time a new block is loaded,
> the last xx (25) bytes of the old block will be prepended and also
> indexed... That way no data will ever get missed...
Thats great. I also noticed (I only have the current version which is on the
web site - without all the bells and wistles) that the buffer is user
settable, so if seeking proves to be too much of a problem, users can just
set the rolling buffer to be really large.
> > That said indexing is a tough job and it does take a long time... its
> > inescapable. Im interested in your indexing implementation,
> > because the
> > database implementation requires a copy of all the strings to
> > live in the db,
> > which basically blows the db up to about 1/3-1/2 the size of the
> > (uncompressed) image. This is not really practical.
>
> Well that could be called small.. It all depends on the text elements
> inside the image file.. If an image contains only text files, a index
> can even become larger that the image itself.
> For the size part, I think that my fileformat will result in smaller
> size files than a database, as the format is optimized for containing
> index trees.. I have reviewed the formats inside (As did my brother)
> and we have sqeezed the format even smaller in the upcoming release..
Cool that sounds very promising. I am looking forward to seeing the next
release, in the meantime I shall play with the current release. Are there
many large changes in the new release?
Michael.
|
|
From: Paul B. <ba...@fo...> - 2004-02-23 13:26:45
|
Hi Michael, > > I currently support two modes (As will the new release)... > > > > The first is raw mode, meaning that all data on the disk is indexed > > as it is!.. That means that the whole disk is walked sequentially in > > (currently) 64k blocks... This can be enlarged if that=20 > would increase > > performance of the underlying subsystem.. > It is possible for you then to just malloc a buffer (say 1mb)=20 > and fill it=20 > sequentially, and then just index the buffer? Or will that complicate=20 > matters? That's what I do.. (Only with a 64kb buffer)... But the current = read_random, will do a seek every time.. And that is not good... (Time/Seekwise)... The size is adaptable and will not change the concept.... Only the implementation of the current functions do not allow me to read the data of an image without them performing any seeks.. > So for example suppose you have a file thats 30 blocks big=20 > (~120kb). While the=20 > file_walk might call the callback 30 times, (once for each=20 > block), In the end=20 > David's print_blocks function will print a single entry for=20 > 30 consecutive=20 > blocks. OK this might be handy, but will only change how many times the callback is called.. No data in the blocks is read on calling the callback (Because of the FS_FLAG_AONLY flag). > From experinece, most files are not really fragmented and at=20 > most I have seen=20 > large files fragmented into 2-3 parts. Thats an average of=20 > 2-3 reads for=20 > large files, and a single read for small files - not too=20 > expensive at all.=20 > (contrast this with reading the block on each callback you=20 > will need to read=20 > every 4kb in every file, or upto several hundred times for=20 > each large file). See above (This does not happen...).... I only read the last 10 bytes from the first block before the fragment and first 10 bytes from the block after the fragment. =20 > I just ran icat under gdb again to confirm what im saying=20 > here. This is the=20 > result: > (gdb) b icat_action > (gdb) r -f linux-ext2 /var/tmp/honeypot.hda1.dd 16 > /tmp/vmlinuz >=20 > Breakpoint 1, icat_action (fs=3D0x8064e18, addr=3D2080, > buf=3D0x8065c48 ... , size=3D1024, flags=3D1028, ptr=3D0x805d934 = "") >=20 > The size is the size that icat writes in every call to=20 > icat_action, and it=20 > seems to be alway 1024 here (block size). So the icat_action=20 > callback is=20 > called for every single block. See above.. Icat does read every block, but the FS_FLAG_AONLY flag makes it NOT read the data in the block, so no seeks will happen there. > The other problem I can think about (again, I havent seen=20 > your code yet so im=20 > sorry if im talking crap here), is that if you only do the=20 > indexing on each=20 > buffer returned by the file_walk callback, then wouldnt you=20 > be missing words=20 > that happen to be cut by the block boundary? i.e. half a word=20 > will be on one=20 > block and the other half on the next block? This problem will=20 > be alleviated=20 > to some extent by indexing larger buffers than blocksizes. You're talking crap! ;-)) Joking..=20 No Michael.. This will not happen.. The raw_fragment mode only indexes sdtrings in the fragmented part (Thus 10 bytes before and 10 bytes after the fragmented part) (Or 25... Depends.. hehe) The raw mode (The real mode) will use the 64kb blocks in a "walking buffer" kind of way... Every time a new block is loaded, the last xx (25) bytes of the old block will be prepended and also indexed... That way no data will ever get missed... <Snipped a part about the flag implementation> > That said indexing is a tough job and it does take a long time... its=20 > inescapable. Im interested in your indexing implementation,=20 > because the=20 > database implementation requires a copy of all the strings to=20 > live in the db,=20 > which basically blows the db up to about 1/3-1/2 the size of the=20 > (uncompressed) image. This is not really practical. Well that could be called small.. It all depends on the text elements inside the image file.. If an image contains only text files, a index can even become larger that the image itself. For the size part, I think that my fileformat will result in smaller size files than a database, as the format is optimized for containing index trees.. I have reviewed the formats inside (As did my brother) and we have sqeezed the format even smaller in the upcoming release.. <Snipped the rest> Paul Bakker |
1 message has been excluded from this view by a project administrator.
