John Levon wrote:
> On Tue, Oct 30, 2007 at 09:31:37PM +0000, Gisle Dankel wrote:
>> Hi Daniel, Maynard and others,
> Did you guys have a look at this?
>> For simple programs, it works more or less out of the box by linking our
>> DBT against libopagent.so. For each portion of generated code that I'm
> That's encouraging.
>> interested in profiling, I call the op_write_native_code() to dump it to
>> a file that is converted to an ELF file by opjitconv (run from
>> opcontrol). By encoding the file name and address range of the
>> translated program in the symbols in this file, I can can get all the
>> way back to source if I want to. Example of a symbol is:
> Eww eww eww. How can we do this without a hack like this?
>> This approach works fine for anonymous regions*.
>> One major missing piece is handling *named* JIT regions (these are not
>> in an ELF format and contain no symbol information). The DBT stores
>> translated code in file backed caches that survive a single run. The
>> files associated with the caches are deleted from the file system as
>> soon as they're created, but still exist in memory and managed by a daemon.
>> Example of such a cache file as it appears in opreport is:
> Hmm. This is a pretty obscure case - why are your translations
It's one of the methods we use for sharing translations between programs
(method depends on the OS). The above file could contain the translated
code for libc for instance, which a daemon hands out to translator
instances needing libc translations by passing a file handle through a
socket. The translator instances then map the translations into their
address spaces. In this case, the files are deleted from the file system
immediately so that they're only kept in memory (or in swap) and
disappear when the last program using it is killed.
>> To allow oprofile to work with these files I've done the following
>> changes (hacks):
>> Note that for this to work the original non-ELF version must have been
>> (as it conveniently has in this case) unlinked from the file system.
> Isn't it simpler for you to just not delete the translation file?
Bolting on an ELF header to the translation files would be difficult...
The contents change frequently, so the header would have to b updated as
well to reflect that. Also, the translation file contains all sorts of
stuff - the generated ELF file contains maybe 50% of the translation
file. The dump file that is used to generate the ELF file is created
when a translation file is mapped into a translator process by
extracting the relevant bits and generating some useful symbols, and
then amended as necessary.
This approach is nice and simple and has a minimal impact on the translator.
>> 4. It appears that for some reason the ELF files generated with
>> opjitconv must be treated differently from other binaries. Mapping
>> samples to symbols in .jo files is done using sym.vma() instead of
>> sym.filepos(). Perhaps someone can explain why this is?
> Daniel? This sounds like a nasty difference? Is it because the section's
> base VMA is not correct? Can we make it be so?
>> So... a few of questions then:
>> 1. Embedding lots of information in the symbols is probably not the way
>> forward. An alternative might be ta add a new function to libopagent
>> that allows additional information of any sort to be embedded in the ELF
>> file somewhere. Any thoughts on this?
> We already have:
> 48 /* idea how to post additional information for a piece of code.
> 49 we use the code address as reference
> 50 int op_write_loader_name(const void* code_addr, char const *
> 51 */
> I presume we'd need a more general version of this to associate 'record
> types' with a particular code region. We'd like some well-defined ones
> (loader, source-lines arrays), and some 'private' ones such as
Yes, just a custom blob would do.
>> 2. The approach used for the named JIT regions won't work if the files
>> still exist on the file system. A better way (in general) might be to
>> change the op_bfd constructor to recognize a named JIT file by name
>> (e.g. ending in .JIT) and redirect to a generated ELF file instead.
>> Other ideas?
> What else processes your ELF files? Can you modify the ELF header? I'm
> up for such an indirection in op_bfd if it can be done cleanly.
Not sure I understand. I've only used oprofile (opreport and opannotate)
and objdump to process the ELF files so far.
The ELF files are generated using opjitconv.
>> 3. It's common for the DBTs to throw away generated code and create new
>> code in its place (reusing memory). The current JIT extension deals with
>> this by
>> a) Recording the times at which JIT code was created and destroyed
>> (op_write_native_code() and op_unlink_native_code())
>> b) Detecting overlapping regions in opjitconv and keeping only one
>> version per address range (the one that existed for the longest amount
>> of time). This is at least what it looks like, please correct me if I'm
>> Since samples are not timestamped (correct?), it is therefore possible
>> to end up with a large number of samples being allocated to the wrong
>> version of generated code, especially if the different versions existed
>> for a similar amount of time. For example, a program that does an execve
>> will reuse the pid for a new executable, which will (probably) cause all
>> the generated code getting replaced with different generated code. There
>> are other situations as well where we throw away some or all generated
>> code and start again.
>> This is quite an annoying problem. One way around this would be for the
>> DBT to ensure that memory for JIT code is never reused when oprofile
>> support is enabled. Unfortunately this could quite easily change the
>> behavior of the DBT substantially, so I'd rather not go down that route.
>> Any thoughts on this?
> Currently that technique is the only one available really. Some previous
> designs attempted to deal with this better but was pretty nasty.
.... It seems like a difficult problem, especially with multiple CPUs.
Have you considered an approach where a JIT can can increment a global
counter (infrequently) that is used to tag samples (e.g. through the
The events that are really of interest are the ones where a large region
of code is thrown out and replaced with new generated code. These events
are not at all that common. I suspect that the same is true for most JVMs...
Maybrook House, 40 Blackfriars Street, Manchester M3 2EG
Phone: +44 (0)161 836 2343 Fax: +44 (0)161 836 2399
* Eliminate the hardware|software dependency
This E-mail and any attachments hereto are strictly confidential and intended solely for the addressee. If you are not the intended addressee please notify the sender by return and delete the message. You must not disclose, forward or copy this E-mail or attachments to any third party without the prior consent of the sender.