This email contains a first draft of a proposal to capture JITted code
based on earlier patches and email exchanges. I would greatly
appreciate feedback on this proposal.
The Oprofile daemon has been modified to aggregate anonymous samples in
version 0.9. The daemon aggregates anonymous samples into contiguous
memory ranges for each module. The opreport treats one contiguous
memory range as a module with a name similar to "anon (tgid:13340
Providing anonymous sample data is better than no data at all, but it
does not provide any information to map the samples to the internals of
the virtual machine. The goal of this proposal is to present a
mechanism to profile generated JIT code using the current Oprofile
The mechanism must meet the following requirements:
*It must be agnostic of the virtual machine. The mechanism must work
with various JVMs and Mono.
*The mechanism must require only minimum changes to the current Oprofile
The detailed proposal is as follows:
1. Proposed agents have the following responsibilities=20
Managing shared memories: Agents are responsible for creating shared
memories and directories saving the native code by calling the
libopagent API. Details are hidden from agents.
Writing out JIT generated native code: The second responsibly of an
agent is to dump the native code by calling the API provided by the
2. As described in the previous section, libopagent provides two main
functionalities to agents:
Managing shared memories: Shared memory is the chosen IPC mechanism for
its fast speed and flexibility. By using shared memory, agents can be
started before the daemon is running. Each shared memory requires a
unique key to identify itself. A unique key can be generated by calling
the ftok() system call. Quoting from the man page of ftok, ftok will
"convert a pathname and a project identifier to a System V IPC key".
The path name libopagent used to generate a unique key of each agent
will be the directory to which the agent will write the ELF file holding
the ELF representation of the generated JIT code. The path is described
in the next sub-section. =20
The shared memory will contain a list of each generated JIT code region
with the starting address, size, and corresponding path of the ELF
format binary file of the dumped generated JIT code. The implementation
of the data structure to be contained in the shared memory must satisfy
the following requirements:
a) Handle regenerated JIT code.
b) The data structure must support constant time look up an entry.
c) The data structure must allow the daemon check whether the data
structure has been modified since the last time the daemon read the data
d) The data structure must allow the daemon find the changed
entries in constant time.
Circular buffer implemented on top of the shared memory satisfies all of
the four requirements above. Read and write pointer indicate the valid
range of memory in the circular buffer. In order to pass the read and
write pointer between the daemon and the agent, a header is placed at
the beginning of the shared memory to store the read and write pointer,
and also the number of valid entry currently in the buffer. The daemon
and the agent must update the header when entries in the buffer is read
and written. A semaphore is used to synchronize reading from and
writing to the buffer.
A portion of JIT code can be regenerated to either the same address or a
different address. If a portion of JIT code is regenerated to the same
address, libopagent will take the following sequence of actions:
a) Find the original entry of the address in the shared memory
b) Dump the new JIT code into a new ELF file.
c) Change the corresponding name of the ELF file in the entry.
d) Change the data structure indicating that it has been modified.
For example, if function foo() was originally generated at address
0x1234, and it is regenerated at the same address, the agents calls
libopagent to update the share memory entry and to dump the newly
generated code. The libopagent will modify the data structure to
reflect its modified status. =20
The name of the binary ELF format file will be in the form of
%address%-%hex number%. The %hex number% is a two digit hex number
denoting the number of times this particular address has been reused
(that is, JIT code has been generated to it). It would start with 00
and increment up to FF, allowing the address to be reused up to 256
During agent initialization and termination, an agent must call the
appropriate libopagent function to create and destroy the shared memory
specific to the particular agent.
Writing Java native code into ELF format binary file using BFD.
The ideal path to save the native code in a new directory is under
Oprofile lib directory. By default, it is /var/lib/oprofile/. The full
path of the new directory will, by default, be /var/lib/oprofile/agent.
It will be referred to as the agent directory. =20
For each instance of an agent, libopagent will create a subdirectory
with tgid as its name under the agent directory. The unique path will
be passed to ftok() to generate unique keys for each agent. The ELF
format binary file will be stored in the tgid directory with the name
<function name>-virtual load address. The ELF files will be used by the
3. The daemon will enumerate directories in the agent directory upon
initialization. It will delete any stale directories and shared memory.
The daemon will create a mapping of JIT code for each tgid in the agent
directory. During processing, if a sample is not a kernel sample and is
an anonymous sample, the daemon will take the following sequence of
*Check if the sample is from a tgid that already has a JIT code mapping.
*If the tgid has no JIT code mapping, check if the tgid exists in the
agent directory. If it does, the daemon would create a new mapping for
this tgid and use the mapping to resolve the sample. To avoid
repeatedly checking the file system, we can store the time of the
previous file system check and give up after a certain time duration is
*The sample will be resolved using the ELF file stored in the agent
directory. The daemon will resolve the instruction pointer (RIP) into
an offset of the ELF file similar to a normal binary file.
*If the process generating the anonymous sample does not have an agent,
the anonymous sample will be processed by the current algorithm in
4. Potential problems:
*Saving to the agent directory would require root privileges. Agents
without root privilege would need to write the generated JIT code to a
temporary directory such as /tmp/oprofile/agent or the home directory
such as /home/<user>/.oprofile/agent. The daemon would have to check
both root and non-root directories. If we choose to save in the
temporary directory, the ELF file might be lost during reboot and the
user will not be able to retrieve assembly level data.
*This proposal is treating each JIT generated region as one module. I
chose this approach to avoid the thorny issues of when to write out the
ELF file, ELF file structural changes, synchronization between all
agents and the daemon, etc. By choosing this approach, it would crowd
the module view and system view if no modification is done to the
reporting tool. This approach slightly violates the Oprofile philosophy
that no modification to the reporting tools is needed when the agent
*This approach assumes that each range of the JIT code is equivalent to
a function. It seems to be the case for both Java and Mono, but may not
be true in the future.