From: Jimmy D. <jd...@us...> - 2004-09-24 20:49:19
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Proposal 1 changes to daemon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Assumptions ----------- 1) JIT must not re-use memory when re-jitting code. JIT developers can add an option to guarantee no re-use of memory. 2) Anonymous mapping data must be maintained by tgid in generated ELF files. Changes to daemon ----------------- 1) JIT data logger This function is invoked by the JIT whenever something is jitted. The JIT provides, for instance, tgid:pid:VMA:length:name[:other data]. The data format needs to be agreed upon - the above is just an example. The data logger then builds an ELF file, in the /tmp/oprofile directory, named anon.tgid. 2) Require an "Anonymous Sample Resolver" which would be invoked before adding a sample to the SFILE. The resolver would accept a VMA/tgid pair and return an offset into the ELF file maintained by the logger. 3) Ability to remove closed anonymous ELF files. Changes to opreport ------------------- 1) Add option --anon-samples This option allows including anonymous samples "bucket" in reports and does symbol resolution when required. Default is to not include anonymous samples in reports. JIT data logger --------------- 1) Implemented as a shared library. 2) Receives tgid and JIT mapping data from the JIT and maintains a ELF file containing each mapping as a symbol. Typically the symbol name will be some sort of method or function name (ie. name in the sample data). A suggested location for the ELF file is /tmp/oprofile/anon.tgid. Anonymous sample resolver --------------------------- 1) Receives a tgid and a VMA from the daemon and returns an offset into the JIT mapping (ELF file and some sort of cookie identifying the file. The data retuned is saved in the SFILE. Flow ---- * Closed JIT mapping files are erased. (at opcontrol --start) * JIT makes calls to the "JIT Data Logger" to write mapping data to ELF file. * Daemon/profiling session can start before or after the JIT starts. * As profiling session progresses the daemon is logging all anonymous samples for tgid in /var/lib/oprofile/samples/current/.../tmp/oprofile/anon.tgid/SFILE * User requests a report: opreport --anon-samples -l ... * From this point on processing continues as it does today. Advantages ---------- 1) OProfile does not have to understand the JITs, other than providing one interface for the JITs to log their mapping data. 2) Design should work with any JIT (java, C#) and with multiple, simultaneous JITs. 3) Works whether the JIT is started before or after the daemon. 4) Location of JIT mapping data is up to the logger (but must coordinate with daemon) Disadvantages ------------- 1) Increased overhead for profiled scenario: - JIT mapping ELF file must be built as scenario runs, adding trace overhead. 2) Requires new "JIT Data Logger" code. - There are complexity issues since JITs, if they support it, may want to provide mapping data which allows mapping to a source file/line, mapping to generated bytecode (in java), and whatever other JIT-specific mappings they may want to generate. - You may choose to limit what mapping data is allowed via the JIT-to-Logger interface. 3) Daemon will have to look up every single anonymous sample before writing it to the SFILE (to convert from a VMA to an offset). 4) Hole for re-using memory during re-jitting is not closed, but can be closed if the JIT does not re-use memory during re-jitting. |
From: John L. <le...@mo...> - 2004-09-27 20:38:11
|
On Fri, Sep 24, 2004 at 03:49:09PM -0500, Jimmy DeWitt wrote: > 1) JIT must not re-use memory when re-jitting code. JIT > developers can add an option to guarantee no re-use of memory. Right. Or we could provide a way to say "reset OProfile". Something we can think about later. > 2) Anonymous mapping data must be maintained by tgid in generated ELF > files. tgid and pid, surely? I'd like to be able to separate out my Java profiles by thread, right? > Changes to daemon > ----------------- > 1) JIT data logger > This function is invoked by the JIT whenever something is jitted. The > JIT provides, > for instance, tgid:pid:VMA:length:name[:other data]. The data format > needs to > be agreed upon - the above is just an example. > The data logger then builds an ELF file, in the /tmp/oprofile > directory, named > anon.tgid. I assume you mean /var/lib/oprofile/current/tmp/oprofile or similar here. Actually it would be much better to have naming like: /var/lib/oprofile/current/{anon}/{jit}/MyJavaProgram.tgid/ - this is the ELF or whatever file /var/lib/oprofile/current/{anon}/MyJavaProgram/CPU_CLK_UNHALTED.tgid.pid.... - normal oprofiled sample file that uses the above {bin} file This sort of thing is closer to the naming we already have, and it *also* covers anonymous samples from other places that don't have oprofile support (which has been an RFE for a while). With Java support, we can rename the execname "/path/to/jvm" with "MyJAvaProgram"; other sources might still end up as "/path/to/pythonvm", but they would show up in the summary reports. > 2) Require an "Anonymous Sample Resolver" which would be invoked before > adding a sample to the SFILE. The resolver would accept a VMA/tgid > pair > and return an offset into the ELF file maintained by the logger. OK > 1) Add option --anon-samples > This option allows including anonymous samples "bucket" in reports > and does symbol resolution when required. > Default is to not include anonymous samples in reports. Why is this the default? > JIT data logger > --------------- > 1) Implemented as a shared library. > 2) Receives tgid and JIT mapping data from the JIT and maintains a ELF > file > containing each mapping as a symbol. Typically the symbol name will > be > some sort of method or function name (ie. name in the sample data). Note that there is still *no* requirement that the file be an ELF format one. If it makes sense to use a different format then we should do that, then subclass op_bfd. This is something that can be decided based on how the implementation looks. > Anonymous sample resolver > --------------------------- > 1) Receives a tgid and a VMA from the daemon and returns an offset into > the > JIT mapping (ELF file and some sort of cookie identifying the file. > The data retuned is saved in the SFILE. This is essentially the same code as the "JIT data logger" right? i.e. they both have the info on the JIT mappings. > * Closed JIT mapping files are erased. (at opcontrol --start) How would this work? The post-prof tools need both the binary and the sample file in order to produce results. > 3) Daemon will have to look up every single anonymous sample before > writing it to the SFILE (to convert from a VMA to an offset). Shouldn't be a big deal, we do it for all non-anon samples already. Anyway, this looks reasonable to me. regards, john |
From: Frank L. <le...@us...> - 2004-09-28 16:22:17
|
opr...@li... wrote on 09/27/2004 03:37:14 PM: > On Fri, Sep 24, 2004 at 03:49:09PM -0500, Jimmy DeWitt wrote: > > > 1) JIT must not re-use memory when re-jitting code. JIT > > developers can add an option to guarantee no re-use of memory. > > Right. Or we could provide a way to say "reset OProfile". Something we > can think about later. Could we have more detail here? It's not clear what you are describing. > > > 2) Anonymous mapping data must be maintained by tgid in generated ELF > > files. > > tgid and pid, surely? I'd like to be able to separate out my Java > profiles by thread, right? Right > This sort of thing is closer to the naming we already have, and it > *also* covers anonymous samples from other places that don't have > oprofile support (which has been an RFE for a while). With Java support, > we can rename the execname "/path/to/jvm" with "MyJAvaProgram"; other > sources might still end up as "/path/to/pythonvm", but they would show > up in the summary reports. Sounds good, we will need to modify Will's patch. > > > 1) Add option --anon-samples > > This option allows including anonymous samples "bucket" in reports > > and does symbol resolution when required. > > Default is to not include anonymous samples in reports. > > Why is this the default? It doesn't matter to us. > Note that there is still *no* requirement that the file be an ELF format > one. If it makes sense to use a different format then we should do that, > then subclass op_bfd. This is something that can be decided based on how > the implementation looks. We would like to start with ELF. It might make it easier to develop if we do not need to make changes to the reporting tools. We can change it as we go if it is necessary. > > This is essentially the same code as the "JIT data logger" right? i.e. > they both have the info on the JIT mappings. yes > > > * Closed JIT mapping files are erased. (at opcontrol --start) > > How would this work? The post-prof tools need both the binary and the > sample file in order to produce results. Right, the sample and the fake ELF files need to be removed at the same time. When we do opcontrol --reset the fake ELF files that are not currently "opened" by the jit logger need to be removed. On archive we will need to save the fake ELF files. |
From: William C. <wc...@nc...> - 2004-09-28 17:03:44
|
Jimmy DeWitt wrote: > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Proposal 1 changes to daemon > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Assumptions > ----------- > 1) JIT must not re-use memory when re-jitting code. JIT > developers can add an option to guarantee no re-use of memory. > 2) Anonymous mapping data must be maintained by tgid in generated ELF > files. > > Changes to daemon > ----------------- > 1) JIT data logger > This function is invoked by the JIT whenever something is jitted. > The JIT provides, > for instance, tgid:pid:VMA:length:name[:other data]. The data > format needs to > be agreed upon - the above is just an example. > The data logger then builds an ELF file, in the /tmp/oprofile > directory, named > anon.tgid. So the instrumentation in JVMPI in JVM is going to need to relay that data to daemon. The profiled java and oprofile daemon may start and stop at different times: 1) oprofile daemon started followed by java program started 2) java program already running with JIT'ed code then oprofile daemon started. 3) oprofile daemon shutdown while java program still running 4) java program shutdown followed by oprofile daemon cases 1 and 4 would be the normal expectation. However, cases 2 and 3 also need to be handled. Also there the oprofile daemon would need to be able to handle multiple messages from multiple JVMs. Would this be some kind of socket that the JVM would send things to and they may or may not be read? Reading farther down this appears to be a file. > 2) Require an "Anonymous Sample Resolver" which would be invoked before > adding a sample to the SFILE. The resolver would accept a VMA/tgid > pair > and return an offset into the ELF file maintained by the logger. > 3) Ability to remove closed anonymous ELF files. Is the ability to remove the anonymous ELF files (and sample files) to remove clutter? Is root only going to be able to do this? Or would there be some way for users to remove the files they own? Right now all the sample files have the root ownership. Would it make sense for files for particular pid/tgid to have the same ownership as the user running them? > Changes to opreport > ------------------- > 1) Add option --anon-samples > This option allows including anonymous samples "bucket" in reports > and does symbol resolution when required. > Default is to not include anonymous samples in reports. Currently different pid/tgid samples are listed as separate columns in opreport and this leads to very wide opreports output. In some cases it might be useful to have data for different pids for the same executable next to each other, but for the generic "opreport" it makes the output difficult to read. The way that the sample files are stored are likely to trigger this undesirable behavior. > JIT data logger > --------------- > 1) Implemented as a shared library. > 2) Receives tgid and JIT mapping data from the JIT and maintains a ELF > file > containing each mapping as a symbol. Typically the symbol name will be > some sort of method or function name (ie. name in the sample data). > A suggested location for the ELF file is /tmp/oprofile/anon.tgid. One reason for the ELF file is to allow people developing the JVM to look at the performance of the code generated. Application developers are not probably not too interested in that, and will want the data mapped back to the Java methods and ideally, be able to use opannotate to map it back to the source code, but I don't know whether the JITs in the JVMs provide enough information for that level of detail of debugging information. > Anonymous sample resolver > --------------------------- > 1) Receives a tgid and a VMA from the daemon and returns an offset > into the > JIT mapping (ELF file and some sort of cookie identifying the file. > The data retuned is saved in the SFILE. > > > Flow > ---- > * Closed JIT mapping files are erased. (at opcontrol --start) Ah, so the JVM instrumentation is writing the JIT mapping files and these files are being used to communicate the information between the JVM and the oprofile daemon? Does the JVM guarantee that the JIT and related profiling hook is run before the JIT'ed code is run? Would like to avoid the situation where the JVMPI doesn't get around to writing out the data until after the oprofile daemon is processing a sample. > * JIT makes calls to the "JIT Data Logger" to write mapping data to > ELF file. > * Daemon/profiling session can start before or after the JIT starts. > * As profiling session progresses the daemon is logging all anonymous > samples for tgid in > /var/lib/oprofile/samples/current/.../tmp/oprofile/anon.tgid/SFILE > * User requests a report: opreport --anon-samples -l ... > * From this point on processing continues as it does today. > > > Advantages > ---------- > 1) OProfile does not have to understand the JITs, other than providing > one interface for the JITs to log their mapping data. > 2) Design should work with any JIT (java, C#) and with multiple, > simultaneous JITs. > 3) Works whether the JIT is started before or after the daemon. > 4) Location of JIT mapping data is up to the logger (but must > coordinate with daemon) > > > Disadvantages > ------------- > 1) Increased overhead for profiled scenario: > - JIT mapping ELF file must be built as scenario runs, adding trace > overhead. > 2) Requires new "JIT Data Logger" code. > - There are complexity issues since JITs, if they support it, may > want to > provide mapping data which allows mapping to a source file/line, > mapping > to generated bytecode (in java), and whatever other JIT-specific > mappings > they may want to generate. > - You may choose to limit what mapping data is allowed via the > JIT-to-Logger > interface. > 3) Daemon will have to look up every single anonymous sample before > writing > it to the SFILE (to convert from a VMA to an offset). Yes, this look up process to convert the VMA to an offset could be expensive. Need to know whether the mapping file has changed before processing the sample and there could be very many small regions in the mapping file due to the JIT generating a region for each JITed method. > 4) Hole for re-using memory during re-jitting is not closed, but > can be closed if the JIT does not re-use memory during re-jitting. -Will |
From: Enio P. <en...@so...> - 2004-09-28 19:20:35
|
> >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Proposal 1 changes to daemon >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ... > > So the instrumentation in JVMPI in JVM is going to need to relay that > data to daemon. The profiled java and oprofile daemon may start and stop > at different times: > > 1) oprofile daemon started followed by java program started > 2) java program already running with JIT'ed code then oprofile daemon > started. > 3) oprofile daemon shutdown while java program still running > 4) java program shutdown followed by oprofile daemon > > cases 1 and 4 would be the normal expectation. However, cases 2 and 3 > also need to be handled. Also there the oprofile daemon would need to be > able to handle multiple messages from multiple JVMs. Would this be some > kind of socket that the JVM would send things to and they may or may > not be read? Reading farther down this appears to be a file. > The thought was that the "logger" would really not be part of the daemon itself, but a shared library provided by OProfile, that each JIT would invoke to log their data. The library would also provide APIs for the daemon to request information. As long as the JIT is running the logger is logging data, regardless of whether or not the daemon is running. I think that would handle all four cases. For case 2, there would already exist a JIT data log file and any new samples would be resolved correctly by the daemon. For case 3, the java program would continue to log data until it completes. Since no new samples are coming in, there is no problem. Handling multiple JITs shouldn't be a problem because each one would be logging data to a separate file. > >> Changes to opreport >> ------------------- >> 1) Add option --anon-samples >> This option allows including anonymous samples "bucket" in reports >> and does symbol resolution when required. >> Default is to not include anonymous samples in reports. > > > Currently different pid/tgid samples are listed as separate columns in > opreport and this leads to very wide opreports output. In some cases it > might be useful to have data for different pids for the same executable > next to each other, but for the generic "opreport" it makes the output > difficult to read. The way that the sample files are stored are likely > to trigger this undesirable behavior. > We set the default so that the generic report would look just as it does without your patch - you never see the anonymous samples. I think we would like to see anonymous samples all the time though. The non-defautl behaviour would be to list the anonymous samples as rows, not columns. There would be a row for each tgid with anonymous samples. Agree that separate columns make the reports hard to read. Unless asked to report by pid, the report I had in mind looks something like this: tid:all| samples| %| ------------------ 112263 33.44 anon.2028 5432 17.17 anon.2988 13 0.0037 aic7xxx 36 0.0102 bash ... Where the % of anonymous samples is the % of all samples. That's what I meant by different buckets. I haven't looked at the PP tools in much detail so I don't know if it's possible to list anonymous samples by row. > >> JIT data logger >> --------------- >> 1) Implemented as a shared library. >> 2) Receives tgid and JIT mapping data from the JIT and maintains a >> ELF file >> containing each mapping as a symbol. Typically the symbol name >> will be >> some sort of method or function name (ie. name in the sample data). >> A suggested location for the ELF file is /tmp/oprofile/anon.tgid. > > > One reason for the ELF file is to allow people developing the JVM to > look at the performance of the code generated. Application developers > are not probably not too interested in that, and will want the data > mapped back to the Java methods and ideally, be able to use opannotate > to map it back to the source code, but I don't know whether the JITs in > the JVMs provide enough information for that level of detail of > debugging information. > True. The problem with using ELF is that you have to be able to store the debug information in the file in order for the PP tools to be able to map samples back to source line numbers. Since each JIT can potentially provide diffent ways of doing that, it becomes the responsibility of the logger to collect the data in a way that "fits" into the ELF debug data format. That also means the interface between the JITs and the logger must be flexible enough to allow the JITs to specify that information, if they have it, and if the PP tools want to provide that functionality. In addition to source line numbers I can see a java JIT wanting to allow mapping a sample to a bytecode stream and event to a generated machine op code. The JITs may not provide that level of detail now but the design should be easily extensible to provide that functionality in the future if needed. >> Anonymous sample resolver >> --------------------------- >> 1) Receives a tgid and a VMA from the daemon and returns an offset >> into the >> JIT mapping (ELF file and some sort of cookie identifying the file. >> The data retuned is saved in the SFILE. >> >> >> Flow >> ---- >> * Closed JIT mapping files are erased. (at opcontrol --start) > > > Ah, so the JVM instrumentation is writing the JIT mapping files and > these files are being used to communicate the information between the > JVM and the oprofile daemon? Right. I think the only communication with the daemon is when the daemon needs to convert a sample VMA to an offset into the log ELF file. > Does the JVM guarantee that the JIT and > related profiling hook is run before the JIT'ed code is run? Would like > to avoid the situation where the JVMPI doesn't get around to writing out > the data until after the oprofile daemon is processing a sample. > I believe the JVMPI notification is done first, before actually running the jitted code. Enio. |
From: John L. <le...@mo...> - 2004-09-28 19:54:38
|
On Tue, Sep 28, 2004 at 02:20:23PM -0500, Enio Pineda wrote: > We set the default so that the generic report would look just as it does > without your patch - you never see the anonymous samples. I think we > would like to see anonymous samples all the time though. > The non-defautl behaviour would be to list the anonymous samples as > rows, not columns. There would be a row for each tgid with anonymous > samples. > Agree that separate columns make the reports hard to read. > Unless asked to report by pid, the report I had in mind looks something > like this: > > tid:all| > samples| %| > ------------------ > 112263 33.44 anon.2028 > 5432 17.17 anon.2988 > 13 0.0037 aic7xxx > 36 0.0102 bash > ... Hmm, this is problematic. Unlike static programs we're in the situation where separate tgid's mean separate reporting rows, yet tgid's also have a meaning in terms of columns. What we might consider doing is thinking of the tgid as an opaque differentiator of JIT'd programs. We'd then format the results to make it clearer that it's opaque. On top of that, we'd have the usual handling of -m tid,tgid etc. behaviour. So, perhaps: tgid:all| samples| %| ------------------ 112263 33.44 MyJavaProgram (run 1) 5432 17.17 MyJavaProgram (run 2) 13 0.0037 aic7xxx 36 0.0102 bash ... Or we could still use --separate=thread and get results as before. regards john |
From: Enio P. <en...@so...> - 2004-09-28 20:27:41
|
John Levon wrote: > On Tue, Sep 28, 2004 at 02:20:23PM -0500, Enio Pineda wrote: > > >>We set the default so that the generic report would look just as it does >>without your patch - you never see the anonymous samples. I think we >>would like to see anonymous samples all the time though. >>The non-defautl behaviour would be to list the anonymous samples as >>rows, not columns. There would be a row for each tgid with anonymous >>samples. >>Agree that separate columns make the reports hard to read. >>Unless asked to report by pid, the report I had in mind looks something >>like this: >> >> tid:all| >> samples| %| >> ------------------ >> 112263 33.44 anon.2028 >> 5432 17.17 anon.2988 >> 13 0.0037 aic7xxx >> 36 0.0102 bash >> ... > > > Hmm, this is problematic. Unlike static programs we're in the situation > where separate tgid's mean separate reporting rows, yet tgid's also have > a meaning in terms of columns. > > What we might consider doing is thinking of the tgid as an opaque > differentiator of JIT'd programs. We'd then format the results to make > it clearer that it's opaque. On top of that, we'd have the usual > handling of -m tid,tgid etc. behaviour. So, perhaps: > > tgid:all| > samples| %| > ------------------ > 112263 33.44 MyJavaProgram (run 1) > 5432 17.17 MyJavaProgram (run 2) > 13 0.0037 aic7xxx > 36 0.0102 bash > ... > > Or we could still use --separate=thread and get results as before. > What I had in mind, and didn't get across, is that the two rows were two different processes, maybe JITs, maybe not, that had anonymous samples, in a single profiling session. All I was saying is that I'd prefer to see two rows as opposed to two columns in that case. I could still slice the report by tid (ie. multi-culumn) if I wanted. You are correct also in that the two rows could be two runs of the same scenario and there would need to be a way to differentiate them. Your example report looks OK to me. When analysts here profile java they don't do cumulative/conscutive/multiple runs. They tend to: run scenario, generate report, reset/clean up, start over. The contents of a report are cumulative up to the point it is generated. It would not be two runs of the same program, just one long run. OProfile does allows cumulative reports so there has to be a way to handle that. Enio. |