From: Maynard J. <may...@us...> - 2013-08-13 19:43:01
|
Hi, A user reported a problem trying to run Java under Valgrind 3.8.1 when specifying a large value for the Java '-Xmx' option (128G). The user was attempting to use Valgrind to do some analysis of "Watson" (of Jeopardy fame), running on a big IBM POWER7 system with tons of memory. I reproduced the problem using a lower value for -Xmx on my IBM POWER7 development system, as well as on my laptop (an Intel Core 2 Duo). On both systems, both the Valgrind and Java executables are 64-bit. Below is the failing output from my laptop: [maynard@oc3431575272 myJavaStuff]$ valgrind --tool=none java -Xmx16G ThreadLoop 1 ==13873== Nulgrind, the minimal Valgrind tool ==13873== Copyright (C) 2002-2012, and GNU GPL'd, by Nicholas Nethercote. ==13873== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==13873== Command: java -Xmx16G ThreadLoop 1 ==13873== JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 16G requested Could not create the Java virtual machine. ==13873== -------------------------------------------- On both systems, if I decreased the -Xmx value to 15G, the error did not occur. Is this a known limitation? Is there any way to work around this issue? I dumped the /proc/<pid>/maps file on my laptop for several runs, specifying different values for '-Xmx' (lower than 16G, of course). For each run, I saw a "[heap]" entry whose size matched whatever I specified for -Xmx. On the POWER7 system, the /proc/<pid>/maps file didn't have a '[heap]' entry. I've attached maps files from the POWER7 and my laptop in case they may be of help. I also tried the same test with current upstream Valgrind. On the POWER7 system, I did see a little improvement -- I was able to use -Xmx values up to 31G --but not enough improvement for the user doing the Watson analysis. I wasn't able to run the java test using upstream Valgrind on my laptop because Valgrind dies with a SIGSEGV (I'll report that problem in a separate posting). Thanks for any help anyone can offer. -Maynard |
From: Julian S. <js...@ac...> - 2013-08-14 08:44:09
|
For 3.8.1, I would expect that the biggest mmap you can do is limited to about 15G, so your -Xmx15G sounds about right. For trunk that got doubled, so I am also not surprised to hear you can manage -Xmx31G, If you are prepared to do some minor fiddling with constants, I don't see why you can't do two more doubling steps so as to allow a 128G mmap: To do a doubling step, do this: 1. in aspacemgr-linux.c, find this # if VG_WORDSIZE == 8 aspacem_maxAddr = (Addr)0x1000000000ULL - 1; // 64G Change that to (Addr)0x200... 2. in mc_main.c, find this # define N_PRIMARY_BITS 20 3. in mc_main.c, find this tl_assert(MASK(1) == 0xFFFFFFF000000000ULL); Change 0xFFFFFFF000000000ULL to 0xFFFFFFE000000000ULL ditto the other assertions (the change moves the 1-0 boundary in the constant one bit upwards) I suspect you may be able to get away with just (1) (worth a try), although you'll wind up with quite a large performance hit with Memcheck when accessing memory above the 64G boundary. Adding in (2) and (3) should bring the performance level back to normal. I'd be interested to hear if this actually works. J On 08/13/2013 09:40 PM, Maynard Johnson wrote: > Hi, > A user reported a problem trying to run Java under Valgrind 3.8.1 when specifying a large value for the Java '-Xmx' option (128G). The user was attempting to use Valgrind to do some analysis of "Watson" (of Jeopardy fame), running on a big IBM POWER7 system with tons of memory. I reproduced the problem using a lower value for -Xmx on my IBM POWER7 development system, as well as on my laptop (an Intel Core 2 Duo). On both systems, both the Valgrind and Java executables are 64-bit. Below is the failing output from my laptop: > > [maynard@oc3431575272 myJavaStuff]$ valgrind --tool=none java -Xmx16G ThreadLoop 1 > ==13873== Nulgrind, the minimal Valgrind tool > ==13873== Copyright (C) 2002-2012, and GNU GPL'd, by Nicholas Nethercote. > ==13873== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info > ==13873== Command: java -Xmx16G ThreadLoop 1 > ==13873== > JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 16G requested > Could not create the Java virtual machine. > ==13873== > > -------------------------------------------- > > On both systems, if I decreased the -Xmx value to 15G, the error did not occur. Is this a known limitation? Is there any way to work around this issue? > > I dumped the /proc/<pid>/maps file on my laptop for several runs, specifying different values for '-Xmx' (lower than 16G, of course). For each run, I saw a "[heap]" entry whose size matched whatever I specified for -Xmx. On the POWER7 system, the /proc/<pid>/maps file didn't have a '[heap]' entry. I've attached maps files from the POWER7 and my laptop in case they may be of help. > > I also tried the same test with current upstream Valgrind. On the POWER7 system, I did see a little improvement -- I was able to use -Xmx values up to 31G --but not enough improvement for the user doing the Watson analysis. I wasn't able to run the java test using upstream Valgrind on my laptop because Valgrind dies with a SIGSEGV (I'll report that problem in a separate posting). > > Thanks for any help anyone can offer. > -Maynard > > > > ------------------------------------------------------------------------------ > Get 100% visibility into Java/.NET code with AppDynamics Lite! > It's a free troubleshooting tool designed for production. > Get down to code-level detail for bottlenecks, with <2% overhead. > Download for free and get started troubleshooting in minutes. > http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Valgrind-users mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-users > |
From: Maynard J. <may...@us...> - 2013-08-14 20:03:27
|
On 08/14/2013 03:43 AM, Julian Seward wrote: > > For 3.8.1, I would expect that the biggest mmap you can do is limited to > about 15G, so your -Xmx15G sounds about right. For trunk that got doubled, > so I am also not surprised to hear you can manage -Xmx31G, > > If you are prepared to do some minor fiddling with constants, I don't see > why you can't do two more doubling steps so as to allow a 128G mmap: Julian, thanks for the tips. To get up to 127G, I did a "quadruple" step as described below. > > To do a doubling step, do this: > > 1. in aspacemgr-linux.c, find this > # if VG_WORDSIZE == 8 > aspacem_maxAddr = (Addr)0x1000000000ULL - 1; // 64G > Change that to (Addr)0x200... First, I just changed this to "(Addr)0x400..." and then ran my toy java app under valgrind memcheck. Specifying '-Xmx' values up to 127G. The higher I went, the slower memcheck appeared to run, varying from ~15% slowdown with -Xmx16G to ~20% slowdown with -Xmx127G. > > 2. in mc_main.c, find this > # define N_PRIMARY_BITS 20 Then I changed this value to '22'. > > 3. in mc_main.c, find this > tl_assert(MASK(1) == 0xFFFFFFF000000000ULL); > Change 0xFFFFFFF000000000ULL to 0xFFFFFFE000000000ULL and changed these assertions to check MASK(x) against 0xFFFFFFC00... . . . and also had to change the MAX_PRIMARY_ADDRESS assertion to 0x3FFFFFFFFFULL. > ditto the other assertions > (the change moves the 1-0 boundary in the constant one bit upwards) > > I suspect you may be able to get away with just (1) (worth a try), although > you'll wind up with quite a large performance hit with Memcheck when > accessing memory above the 64G boundary. Adding in (2) and (3) should > bring the performance level back to normal. Once all these had been done, memcheck still ran slower (but not as much) -- between 8 and 12% slower, again, depending on the value of -Xmx. Since the performance level didn't come back to "normal", maybe there's some other spot to change. I presume there's a reason why these address ranges are set as they are. Can you enlighten me? Is there a possibility of making this configurable? Thanks. -Maynard > > I'd be interested to hear if this actually works. > > J > > > On 08/13/2013 09:40 PM, Maynard Johnson wrote: >> Hi, >> A user reported a problem trying to run Java under Valgrind 3.8.1 when specifying a large value for the Java '-Xmx' option (128G). The user was attempting to use Valgrind to do some analysis of "Watson" (of Jeopardy fame), running on a big IBM POWER7 system with tons of memory. I reproduced the problem using a lower value for -Xmx on my IBM POWER7 development system, as well as on my laptop (an Intel Core 2 Duo). On both systems, both the Valgrind and Java executables are 64-bit. Below is the failing output from my laptop: >> >> [maynard@oc3431575272 myJavaStuff]$ valgrind --tool=none java -Xmx16G ThreadLoop 1 >> ==13873== Nulgrind, the minimal Valgrind tool >> ==13873== Copyright (C) 2002-2012, and GNU GPL'd, by Nicholas Nethercote. >> ==13873== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info >> ==13873== Command: java -Xmx16G ThreadLoop 1 >> ==13873== >> JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 16G requested >> Could not create the Java virtual machine. >> ==13873== >> >> -------------------------------------------- >> >> On both systems, if I decreased the -Xmx value to 15G, the error did not occur. Is this a known limitation? Is there any way to work around this issue? >> >> I dumped the /proc/<pid>/maps file on my laptop for several runs, specifying different values for '-Xmx' (lower than 16G, of course). For each run, I saw a "[heap]" entry whose size matched whatever I specified for -Xmx. On the POWER7 system, the /proc/<pid>/maps file didn't have a '[heap]' entry. I've attached maps files from the POWER7 and my laptop in case they may be of help. >> >> I also tried the same test with current upstream Valgrind. On the POWER7 system, I did see a little improvement -- I was able to use -Xmx values up to 31G --but not enough improvement for the user doing the Watson analysis. I wasn't able to run the java test using upstream Valgrind on my laptop because Valgrind dies with a SIGSEGV (I'll report that problem in a separate posting). >> >> Thanks for any help anyone can offer. >> -Maynard >> >> >> >> ------------------------------------------------------------------------------ >> Get 100% visibility into Java/.NET code with AppDynamics Lite! >> It's a free troubleshooting tool designed for production. >> Get down to code-level detail for bottlenecks, with <2% overhead. >> Download for free and get started troubleshooting in minutes. >> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk >> >> >> >> _______________________________________________ >> Valgrind-users mailing list >> Val...@li... >> https://lists.sourceforge.net/lists/listinfo/valgrind-users >> > |