From: Yeh, J. <jas...@am...> - 2006-05-19 19:00:25
|
Hi, This email contains a first draft of a proposal to capture JITted code based on earlier patches and email exchanges. I would greatly appreciate feedback on this proposal. The proposal: The Oprofile daemon has been modified to aggregate anonymous samples in version 0.9. The daemon aggregates anonymous samples into contiguous memory ranges for each module. The opreport treats one contiguous memory range as a module with a name similar to "anon (tgid:13340 range:0xb2694000-0xb2724000)". =20 Providing anonymous sample data is better than no data at all, but it does not provide any information to map the samples to the internals of the virtual machine. The goal of this proposal is to present a mechanism to profile generated JIT code using the current Oprofile infrastructure. The mechanism must meet the following requirements: *It must be agnostic of the virtual machine. The mechanism must work with various JVMs and Mono. *The mechanism must require only minimum changes to the current Oprofile infrastructure. The detailed proposal is as follows: 1. Proposed agents have the following responsibilities=20 Managing shared memories: Agents are responsible for creating shared memories and directories saving the native code by calling the libopagent API. Details are hidden from agents. Writing out JIT generated native code: The second responsibly of an agent is to dump the native code by calling the API provided by the libopagent API. 2. As described in the previous section, libopagent provides two main functionalities to agents: Managing shared memories: Shared memory is the chosen IPC mechanism for its fast speed and flexibility. By using shared memory, agents can be started before the daemon is running. Each shared memory requires a unique key to identify itself. A unique key can be generated by calling the ftok() system call. Quoting from the man page of ftok, ftok will "convert a pathname and a project identifier to a System V IPC key". The path name libopagent used to generate a unique key of each agent will be the directory to which the agent will write the ELF file holding the ELF representation of the generated JIT code. The path is described in the next sub-section. =20 The shared memory will contain a list of each generated JIT code region with the starting address, size, and corresponding path of the ELF format binary file of the dumped generated JIT code. The implementation of the data structure to be contained in the shared memory must satisfy the following requirements: a) Handle regenerated JIT code. b) The data structure must support constant time look up an entry. c) The data structure must allow the daemon check whether the data structure has been modified since the last time the daemon read the data structure. d) The data structure must allow the daemon find the changed entries in constant time. Circular buffer implemented on top of the shared memory satisfies all of the four requirements above. Read and write pointer indicate the valid range of memory in the circular buffer. In order to pass the read and write pointer between the daemon and the agent, a header is placed at the beginning of the shared memory to store the read and write pointer, and also the number of valid entry currently in the buffer. The daemon and the agent must update the header when entries in the buffer is read and written. A semaphore is used to synchronize reading from and writing to the buffer. A portion of JIT code can be regenerated to either the same address or a different address. If a portion of JIT code is regenerated to the same address, libopagent will take the following sequence of actions: a) Find the original entry of the address in the shared memory region. b) Dump the new JIT code into a new ELF file. c) Change the corresponding name of the ELF file in the entry. d) Change the data structure indicating that it has been modified. For example, if function foo() was originally generated at address 0x1234, and it is regenerated at the same address, the agents calls libopagent to update the share memory entry and to dump the newly generated code. The libopagent will modify the data structure to reflect its modified status. =20 The name of the binary ELF format file will be in the form of %address%-%hex number%. The %hex number% is a two digit hex number denoting the number of times this particular address has been reused (that is, JIT code has been generated to it). It would start with 00 and increment up to FF, allowing the address to be reused up to 256 times. During agent initialization and termination, an agent must call the appropriate libopagent function to create and destroy the shared memory specific to the particular agent. Writing Java native code into ELF format binary file using BFD. The ideal path to save the native code in a new directory is under Oprofile lib directory. By default, it is /var/lib/oprofile/. The full path of the new directory will, by default, be /var/lib/oprofile/agent. It will be referred to as the agent directory. =20 For each instance of an agent, libopagent will create a subdirectory with tgid as its name under the agent directory. The unique path will be passed to ftok() to generate unique keys for each agent. The ELF format binary file will be stored in the tgid directory with the name <function name>-virtual load address. The ELF files will be used by the reporting tools. 3. The daemon will enumerate directories in the agent directory upon initialization. It will delete any stale directories and shared memory. The daemon will create a mapping of JIT code for each tgid in the agent directory. During processing, if a sample is not a kernel sample and is an anonymous sample, the daemon will take the following sequence of actions: *Check if the sample is from a tgid that already has a JIT code mapping. *If the tgid has no JIT code mapping, check if the tgid exists in the agent directory. If it does, the daemon would create a new mapping for this tgid and use the mapping to resolve the sample. To avoid repeatedly checking the file system, we can store the time of the previous file system check and give up after a certain time duration is reached. *The sample will be resolved using the ELF file stored in the agent directory. The daemon will resolve the instruction pointer (RIP) into an offset of the ELF file similar to a normal binary file. *If the process generating the anonymous sample does not have an agent, the anonymous sample will be processed by the current algorithm in Oprofile. =20 4. Potential problems: *Saving to the agent directory would require root privileges. Agents without root privilege would need to write the generated JIT code to a temporary directory such as /tmp/oprofile/agent or the home directory such as /home/<user>/.oprofile/agent. The daemon would have to check both root and non-root directories. If we choose to save in the temporary directory, the ELF file might be lost during reboot and the user will not be able to retrieve assembly level data. *This proposal is treating each JIT generated region as one module. I chose this approach to avoid the thorny issues of when to write out the ELF file, ELF file structural changes, synchronization between all agents and the daemon, etc. By choosing this approach, it would crowd the module view and system view if no modification is done to the reporting tool. This approach slightly violates the Oprofile philosophy that no modification to the reporting tools is needed when the agent changes. *This approach assumes that each range of the JIT code is equivalent to a function. It seems to be the case for both Java and Mono, but may not be true in the future. Regards, Jason |
From: Jason Y. <jas...@am...> - 2006-08-24 20:36:28
Attachments:
jvmti-example.tar.gz
|
hi, I decided to start a new thread instead of adding to the previous thread cluttered with somewhat unrelated information. A simple jvmti example is attached with the mail. In the example, JVMTI_EVENT_COMPILED_METHOD_LOAD and JVMTI_EVENT_DYNAMIC_CODE_GENERATED are hooked to callbacks which will use libopagent API to write the JIT code into a ELF file in /var/lib/oprofile/jit/<pid>/. The callbacks would also use the API to store JIT info in shared memory to be read by the daemon. The daemon also uses libopagent API to read the shared memory retrieving JIT info. The JIT info is then used to process each sample from the event buffer. The JIT processing takes place before the anonymous sample processing. In the current patch, all ELF files are stored in /var/lib/oprofile/jit/<pid>, and this path will be the path reported by opreport. Since there is no difference between the ELF representation of the JIT code and any other executables in the system, opreport and opannotate do not need any patch. There are still some items on my TODO list, namely: 1. I have not verify whether oparchive works with the current patch. 2. opcontrol --reset currently would wipe out everything in /var/lib/oprofile/jit/ directory. This might not be the action user expects. 3. More testing. I have been using primarily jbb to test the patch. I have no idea how well or badly the patch work under different kind of workload. If anyone has any suggestion on the tests to run, please let me know. 4. Shared memory and semaphore are sometimes undeleted. Jason |
From: John L. <le...@mo...> - 2006-08-27 00:13:57
|
On Thu, Aug 24, 2006 at 03:36:07PM -0500, Jason Yeh wrote: > I decided to start a new thread instead of adding to the previous thread > cluttered with somewhat unrelated information. A simple jvmti example > is attached with the mail. Do we have code for review for all the oprofile side stuff? john |
From: John L. <le...@mo...> - 2006-05-21 19:58:44
|
On Fri, May 19, 2006 at 02:00:05PM -0500, Yeh, Jason wrote: > (that is, JIT code has been generated to it). It would start with 00 > and increment up to FF, allowing the address to be reused up to 256 > times. Why the restriction? > 3. The daemon will enumerate directories in the agent directory upon > initialization. It will delete any stale directories and shared memory. How would you determine staleness? What happens after a reboot, when tgid's can be reused? > *Saving to the agent directory would require root privileges. Agents > without root privilege would need to write the generated JIT code to a > temporary directory such as /tmp/oprofile/agent or the home directory > such as /home/<user>/.oprofile/agent. The daemon would have to check > both root and non-root directories. If we choose to save in the > temporary directory, the ELF file might be lost during reboot and the > user will not be able to retrieve assembly level data. Why would it be lost? How would the post-profiling tools even /find/ the stuff if it goes in home directories? It would have to be a global directory at least. > *This proposal is treating each JIT generated region as one module. I > chose this approach to avoid the thorny issues of when to write out the > ELF file, ELF file structural changes, synchronization between all > agents and the daemon, etc. By choosing this approach, it would crowd > the module view and system view if no modification is done to the > reporting tool. This approach slightly violates the Oprofile philosophy > that no modification to the reporting tools is needed when the agent > changes. The idea of using a separate file for each function is interesting, but I do have concerns. First off, we can easily expect a *lot* of such files. Will this be reasonably performant? Secondly, how would the post-profiling tools gather all of this data into a reasonably human profile. How do you plan to do this? In particular, what would "application name" and "image name" be in this context? regards, john |