elftoolchain-developers

[Elftoolchain-developers] A safe way to re-build a string table ..

[Jay V. <jay...@th...>]

Hi all,

I am building an application which utilizes elfutils to add NOTE records to ELF files .. 
the purpose of this application is to provide a form of 'internal inventory control' over 
executables being used internally in our environment, and for that purpose a NOTE record 
is added to ELF files containing an MD5 sum of various parts of the ELF files themselves.

All has been working fine with this utility program so far, but I have run into some 
problems with the handling of the string table - namely, where I was once realloc()'ing 
the string table data buffer and appending my new ".note.MD5" string to the list, and this 
isn't a very safe way of doing things - segfaults galore.  Its not stable, alas, which is 
why I ended up implementing things this way only to discover on larger string tables that 
there was a problem, then tracing it back (valgrind to the rescue) to the fact that libelf 
doesn't like it when you realloc() its jiggly bits^W^Wstruct fields ..

So my next approach has been to, as I'm iterating through the original ELF file, make a 
new string table out of the headers for each section, and upon completion (calculation of 
the to-be-inserted md5 string, construction of a proper NOTE record, attachment of the 
NOTE to the ELF file, etc.) I then burn the old string table and attach the new one.

However, I'm really not sure if this is the safe way to do things, or if there is any 
other way that I could utilize - its a bit 'dodgy' to me at first blush, because what if 
there are cases out there in our ELF collection where I somehow mess up the string table, 
or what about the unspoken cases where a string table is expected to be something that 
libebl doesn't reproduce - I've already noted that some string tables created through 
libebl don't exactly 100% match the original ones (such as the ones in my redhat machines 
libc) due to internal optimization of the string table itself (finding overlaps and not 
inserting duplicate string values and so on..)

My question to the wise developers on this list is whether or not there is another way to 
safely add an entry to an existing string table without doing too much 
create-new/destroy-old processing?  I've added the use of libebl to my project, and I'm 
under way with completing my implementation as described above, but I'd really like some 
dialog with other ELF hackers on what they think about this method?

BTW, I plan to release my "ELFNoter" app as GPL when its working/debugged .. I wonder if 
there is interest in including a working app to attach ELF Notes in the elfutils package? 
I'd be happy were that the case, to provide this app for review.

-- 
;
--
Jay Vaughan
jay...@th...

Re: [Elftoolchain-developers] A safe way to re-build a string table ..

[Kai W. <kai...@gm...>]

Hi Jay, 

First of all I think you are posting to the wrong mailling list...

Second, for your problem: If this .note.MD5 (for example) section does
not need to be presented to the runtime loader or used during runtime,
you don't need to roll your own tool and you can just use objcopy(1)
to handle it:

1. Calculate the checksum or other stuff of the ELF, construct the Elf Notes
   and then write all the bits to a temporary file, say '/tmp/foo'.
2. Execute objcopy to add a .note.MD5 section to the ELF file:
   objcopy --add-section '.note.MD5=/tmp/foo' your_executable
  (objcopy will handle all the string table, section table headers stuff etc)

Best,
Kai

On Tue, Feb 23, 2010 at 03:51:48PM +0100, Jay Vaughan wrote:
> Hi all,
> 
> I am building an application which utilizes elfutils to add NOTE records to ELF files .. 
> the purpose of this application is to provide a form of 'internal inventory control' over 
> executables being used internally in our environment, and for that purpose a NOTE record 
> is added to ELF files containing an MD5 sum of various parts of the ELF files themselves.
> 
> All has been working fine with this utility program so far, but I have run into some 
> problems with the handling of the string table - namely, where I was once realloc()'ing 
> the string table data buffer and appending my new ".note.MD5" string to the list, and this 
> isn't a very safe way of doing things - segfaults galore.  Its not stable, alas, which is 
> why I ended up implementing things this way only to discover on larger string tables that 
> there was a problem, then tracing it back (valgrind to the rescue) to the fact that libelf 
> doesn't like it when you realloc() its jiggly bits^W^Wstruct fields ..
> 
> So my next approach has been to, as I'm iterating through the original ELF file, make a 
> new string table out of the headers for each section, and upon completion (calculation of 
> the to-be-inserted md5 string, construction of a proper NOTE record, attachment of the 
> NOTE to the ELF file, etc.) I then burn the old string table and attach the new one.
> 
> However, I'm really not sure if this is the safe way to do things, or if there is any 
> other way that I could utilize - its a bit 'dodgy' to me at first blush, because what if 
> there are cases out there in our ELF collection where I somehow mess up the string table, 
> or what about the unspoken cases where a string table is expected to be something that 
> libebl doesn't reproduce - I've already noted that some string tables created through 
> libebl don't exactly 100% match the original ones (such as the ones in my redhat machines 
> libc) due to internal optimization of the string table itself (finding overlaps and not 
> inserting duplicate string values and so on..)
> 
> My question to the wise developers on this list is whether or not there is another way to 
> safely add an entry to an existing string table without doing too much 
> create-new/destroy-old processing?  I've added the use of libebl to my project, and I'm 
> under way with completing my implementation as described above, but I'd really like some 
> dialog with other ELF hackers on what they think about this method?
> 
> BTW, I plan to release my "ELFNoter" app as GPL when its working/debugged .. I wonder if 
> there is interest in including a working app to attach ELF Notes in the elfutils package? 
> I'd be happy were that the case, to provide this app for review.
> 
> -- 
> ;
> --
> Jay Vaughan
> jay...@th...
> 
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Elftoolchain-developers mailing list
> Elf...@li...
> https://lists.sourceforge.net/lists/listinfo/elftoolchain-developers

Re: [Elftoolchain-developers] A safe way to re-build a string table ..

[Jay V. <jay...@th...>]

Hi Kai - thanks for responding ..

> First of all I think you are posting to the wrong mailling list...
>

Ah, I perhaps should be on elfutils-developers?  I don't mind either way, I just need some
ELF expert(s) who are willing to talk about this stuff .. ;)  Been winging it on my own on
this development lately, so its just good to know people who know ELF.  Please feel free
to tell me to sod off if this is not topical for this list, though. :)

> Second, for your problem: If this .note.MD5 (for example) section does
> not need to be presented to the runtime loader or used during runtime,
> you don't need to roll your own tool and you can just use objcopy(1)
> to handle it:

It does need to be readable during runtime and the loader in our internal system is going
to be expected to use this information along with other online services and dynamic loader
paths, so I'd really like to not use objcopy if it means it will produce a more 'standard
ELF Note' to write an app to do it.

> 
> 1. Calculate the checksum or other stuff of the ELF, construct the Elf Notes
>    and then write all the bits to a temporary file, say '/tmp/foo'.
> 2. Execute objcopy to add a .note.MD5 section to the ELF file:
>    objcopy --add-section '.note.MD5=/tmp/foo' your_executable
>   (objcopy will handle all the string table, section table headers stuff etc)

One thing this method doesn't do is properly update the name/namesz and type fields, I
have found (correct me if I'm wrong) - so for my purposes it appears that the objcopy
technique will not produce an *entirely* valid (to-spec) NOTE record.  What I am doing is
using a struct like this in my elfnoter application:

#define INTW1X_ELF_SECTION      ".note.MD5"
#define ELF_TEXT_NAME           ".text"

#define INTW1X_MD5_NOTE_NAME    "w1x_md5"
#define INTW1X_MD5_TYPE         1

#define MD5_DESC "deadbeefd00df00ddeadbeefd00df00d"
#define MD5_BUF_LEN 33

const struct intw1x_elf_note_md5 local_elf_note_md5
     __attribute__ ((section(INTW1X_ELF_SECTION))) =
{
     .elf_note = {
     	.n_namesz = sizeof(INTW1X_MD5_NOTE_NAME),
	.n_descsz = MD5_BUF_LEN,
	.n_type = INTW1X_MD5_TYPE
     },
     .name = INTW1X_MD5_NOTE_NAME,
     .desc = MD5_DESC
};

.. then, locating this ELF record internally within my elfnoter app programmatically, and
'injecting' it into the target executable after the md5 string is updated, so that I
produce an entirely valid (all fields correct) record in the destination binary.  At least
the thinking is that inlining the definition of this record means that the final object is
going to be as standard as gcc is going to make it and thus all I have to do is modify the
string and glom the object onto my target ELF binary.  This is working, its just a little
delicate when it comes to updating the string table, alas.

If there is a way to get objcopy to do this work, I'd love to know .. perhaps it would be
worthwhile investigating this further and adding code to objcopy for NOTE objects, but
then again I don't know enough about this to say for sure and I appreciate your feedback.

-- 
;
--
Jay Vaughan
jay...@in...

Re: [Elftoolchain-developers] A safe way to re-build a string table ..

[Kai W. <kai...@gm...>]

On Tue, Feb 23, 2010 at 04:46:04PM +0100, Jay Vaughan wrote:
> Hi Kai - thanks for responding ..
> 
> > First of all I think you are posting to the wrong mailling list...
> >
> 
> Ah, I perhaps should be on elfutils-developers?  I don't mind either way, I just need some
> ELF expert(s) who are willing to talk about this stuff .. ;)  Been winging it on my own on
> this development lately, so its just good to know people who know ELF.  Please feel free
> to tell me to sod off if this is not topical for this list, though. :)

I suggest you post this to elfutils mailing list as well, may be you
can get some answers from there...

> > Second, for your problem: If this .note.MD5 (for example) section does
> > not need to be presented to the runtime loader or used during runtime,
> > you don't need to roll your own tool and you can just use objcopy(1)
> > to handle it:
> 
> It does need to be readable during runtime and the loader in our internal system is going
> to be expected to use this information along with other online services and dynamic loader
> paths, so I'd really like to not use objcopy if it means it will produce a more 'standard
> ELF Note' to write an app to do it.

Then in that case, objcopy won't work. objcopy can only add
"un-loadable" ELF sections.

> > 
> > 1. Calculate the checksum or other stuff of the ELF, construct the Elf Notes
> >    and then write all the bits to a temporary file, say '/tmp/foo'.
> > 2. Execute objcopy to add a .note.MD5 section to the ELF file:
> >    objcopy --add-section '.note.MD5=/tmp/foo' your_executable
> >   (objcopy will handle all the string table, section table headers stuff etc)
> 
> One thing this method doesn't do is properly update the name/namesz and type fields, I
> have found (correct me if I'm wrong) - so for my purposes it appears that the objcopy
> technique will not produce an *entirely* valid (to-spec) NOTE record.  What I am doing is
> using a struct like this in my elfnoter application:
> 
> #define INTW1X_ELF_SECTION      ".note.MD5"
> #define ELF_TEXT_NAME           ".text"
> 
> #define INTW1X_MD5_NOTE_NAME    "w1x_md5"
> #define INTW1X_MD5_TYPE         1
> 
> #define MD5_DESC "deadbeefd00df00ddeadbeefd00df00d"
> #define MD5_BUF_LEN 33
> 
> const struct intw1x_elf_note_md5 local_elf_note_md5
>      __attribute__ ((section(INTW1X_ELF_SECTION))) =
> {
>      .elf_note = {
>      	.n_namesz = sizeof(INTW1X_MD5_NOTE_NAME),
> 	.n_descsz = MD5_BUF_LEN,
> 	.n_type = INTW1X_MD5_TYPE
>      },
>      .name = INTW1X_MD5_NOTE_NAME,
>      .desc = MD5_DESC
> };
> 
> .. then, locating this ELF record internally within my elfnoter app programmatically, and
> 'injecting' it into the target executable after the md5 string is updated, so that I
> produce an entirely valid (all fields correct) record in the destination binary.  At least
> the thinking is that inlining the definition of this record means that the final object is
> going to be as standard as gcc is going to make it and thus all I have to do is modify the
> string and glom the object onto my target ELF binary.  This is working, its just a little
> delicate when it comes to updating the string table, alas.

You approach here looks fine to me. I don't know about elfutils libelf
and libebl, I just have general suggestion here:

1. string table (.shstrtab) create-new/destroy-old process is neccessary
   and safer.
2. make sure you also updated section header table, but I guess libebl
   has already done this for you?
3. since you're adding a "loadable" section, you need to update some entry
   in the program header (Elf_Phdr) to include the .note.MD5 section
   you added. Also you probably should insert .note.MD5 section at the end
   of the loadable sections but before ".bss" to avoid relocating ".text" and
   ".data" etc. I suppose you've already handled this since your tool is working.

> If there is a way to get objcopy to do this work, I'd love to know .. perhaps it would be
> worthwhile investigating this further and adding code to objcopy for NOTE objects, but
> then again I don't know enough about this to say for sure and I appreciate your feedback.

Binutils is lot messier than elfutils, so building your own tool using
elfutils is probably much easier than extending objcopy I guess.

Kai