[Valgrind-users] running callgrind with JNI (libhdfs) does not work

[Eqbal <eqb...@gm...>]

I am trying to profile an application with callgrind. The application uses
hadoop's libhdfs api which in turn makes JNI calls to a JVM. This causes a
crash in the JVM. When running normally the application seems to work fine.

I am running on Centos 6.7 with Valgrind 3.11.0. Java is 1.8 Oracle JDK (I
have also tried open jdk without much luck).
Here is my commandline:

valgrind -v --tool=callgrind --dump-instr=yes --trace-jump=yes
--trace-children=yes --smc-check=all --collect-jumps=yes
--simulate-cache=yes ./example

 Below are some of the output from valgrind:

[CodeBlob (0x00000000064d8f50)]
Framesize: 84
SafepointBlob
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is
disabled
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (sharedRuntime.cpp:834), pid=37623, tid=0x0000000005c138e0
#  fatal error: exception happened outside interpreter, nmethods and vtable
stubs at pc 0x00000000064d9007
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build
1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode
linux-amd64 compressed oops)
# Core dump written. Default location: /home/myuser/libhdfs/core or
core.37623
#
# An error report file with more information is saved as:
# /home/myuser/libhdfs/hs_err_pid37623.log
==37623== brk segment overflow in thread #1: can't grow to 0x4900000
==37623== brk segment overflow in thread #1: can't grow to 0x49a7000
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
==37623==
==37623== Process terminating with default action of signal 6 (SIGABRT):
dumping core
==37623==    at 0x3127032625: raise (in /lib64/libc-2.12.so)
==37623==    by 0x3127033E04: abort (in /lib64/libc-2.12.so)
==37623==    by 0x554A604: os::abort(bool) (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623==    by 0x56E9A62: VMError::report_and_die() (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623==    by 0x5127098: report_fatal(char const*, int, char const*) (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623==    by 0x55E7479:
SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned
char*, SharedRuntime::ImplicitExceptionKind) (in /usr/java/jdk1.8.0_92
/jre/lib/amd64/server/libjvm.so)
==37623==    by 0x5550169: JVM_handle_linux_signal (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623==    by 0x55465C2: signalHandler(int, siginfo*, void*) (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623==    by 0x312703269F: ??? (in /lib64/libc-2.12.so)
==37623==    by 0x64D9006: ???
==37623==    by 0x44: ???
==37623==    by 0x649A03F: ???

----------
Here is the stack trace from jvm error log:

Current thread (0x0000000004022800):  JavaThread "main" [_thread_in_Java,
id=37623, stack(0x0000000ffef01000,0x0000000fff001000)]

Stack: [0x0000000ffef01000,0x0000000fff001000],  sp=0x0000000ffeff1ae0,
 free space=962k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V  [libjvm.so+0xabd65a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4fb099]  report_fatal(char const*, int, char const*)+0x59
V  [libjvm.so+0x9bb47a]
 SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned
char*, SharedRuntime::ImplicitExceptionKind)+0x33a
V  [libjvm.so+0x92416a]  JVM_handle_linux_signal+0x48a
V  [libjvm.so+0x91a5c3]  signalHandler(int, siginfo*, void*)+0x43
C  [libc.so.6+0x326a0]
C  0x0000000000000045
j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68e746]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x7275bc]  JVM_DoPrivileged+0x27c
j
 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
j  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
J 327 C1
java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
bytes) @ 0x000000000663a8e4 [0x0000000006639f80+0x964]
j
 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+81
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68e746]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x68ec51]  JavaCalls::call_virtual(JavaValue*, KlassHandle,
Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x68f0a6]  JavaCalls::call_virtual(JavaValue*, Handle,
KlassHandle, Symbol*, Symbol*, Handle, Thread*)+0x56
V  [libjvm.so+0xa349a0]  SystemDictionary::load_instance_class(Symbol*,
Handle, Thread*)+0x3f0
V  [libjvm.so+0xa3388c]
 SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle,
Thread*)+0x7cc
V  [libjvm.so+0xa34dc3]  SystemDictionary::resolve_or_fail(Symbol*, Handle,
Handle, bool, Thread*)+0x33
V  [libjvm.so+0x70cd6e]  find_class_from_class_loader(JNIEnv_*, Symbol*,
unsigned char, Handle, Handle, unsigned char, Thread*)+0x3e
V  [libjvm.so+0x7142b1]  JVM_FindClassFromCaller+0x2e1
C  [libjava.so+0xe2d0]  Java_java_lang_Class_forName0+0x130
j
 java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class;+0
j
 java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+49
j
 com.sun.beans.finder.ClassFinder.findClass(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljava/lang/Class;+11
j
 com.sun.beans.finder.InstanceFinder.instantiate(Ljava/lang/Class;Ljava/lang/String;)Ljava/lang/Object;+13
j
 com.sun.beans.finder.InstanceFinder.find(Ljava/lang/Class;)Ljava/lang/Object;+34
j
 java.beans.Introspector.findExplicitBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+7
j  java.beans.Introspector.<init>(Ljava/lang/Class;Ljava/lang/Class;I)V+121
j
 java.beans.Introspector.getBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+60
j
 java.beans.Introspector.getBeanInfo(Ljava/lang/Class;Ljava/lang/Class;I)Ljava/beans/BeanInfo;+10
j  java.beans.Introspector.<init>(Ljava/lang/Class;Ljava/lang/Class;I)V+157
j
 java.beans.Introspector.getBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+60
j
 java.beans.Introspector.getBeanInfo(Ljava/lang/Class;Ljava/lang/Class;I)Ljava/beans/BeanInfo;+10
j  java.beans.Introspector.<init>(Ljava/lang/Class;Ljava/lang/Class;I)V+157
j
 java.beans.Introspector.getBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+60
j  org.apache.log4j.config.PropertySetter.introspect()V+7
j
 org.apache.log4j.config.PropertySetter.getPropertyDescriptor(Ljava/lang/String;)Ljava/beans/PropertyDescriptor;+8
j
 org.apache.log4j.config.PropertySetter.setProperties(Ljava/util/Properties;Ljava/lang/String;)V+113
j
 org.apache.log4j.config.PropertySetter.setProperties(Ljava/lang/Object;Ljava/util/Properties;Ljava/lang/String;)V+10
j
 org.apache.log4j.PropertyConfigurator.parseAppender(Ljava/util/Properties;Ljava/lang/String;)Lorg/apache/log4j/Appender;+694
j
 org.apache.log4j.PropertyConfigurator.parseCategory(Ljava/util/Properties;Lorg/apache/log4j/Logger;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V+280
j
 org.apache.log4j.PropertyConfigurator.configureRootCategory(Ljava/util/Properties;Lorg/apache/log4j/spi/LoggerRepository;)V+63
j
 org.apache.log4j.PropertyConfigurator.doConfigure(Ljava/util/Properties;Lorg/apache/log4j/spi/LoggerRepository;)V+134
j
 org.apache.log4j.PropertyConfigurator.doConfigure(Ljava/net/URL;Lorg/apache/log4j/spi/LoggerRepository;)V+246
j
 org.apache.log4j.helpers.OptionConverter.selectAndConfigure(Ljava/net/URL;Ljava/lang/String;Lorg/apache/log4j/spi/LoggerRepository;)V+129
j  org.apache.log4j.LogManager.<clinit>()V+156
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68e746]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x640807]
 InstanceKlass::call_class_initializer_impl(instanceKlassHandle,
Thread*)+0xd7
V  [libjvm.so+0x642cfc]
 InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V  [libjvm.so+0x6430c1]  InstanceKlass::initialize(Thread*)+0x41
V  [libjvm.so+0x7f7df6]  LinkResolver::resolve_static_call(CallInfo&,
KlassHandle&, Symbol*, Symbol*, KlassHandle, bool, bool, Thread*)+0x246
V  [libjvm.so+0x7f807f]  LinkResolver::resolve_invokestatic(CallInfo&,
constantPoolHandle, int, Thread*)+0x23f
V  [libjvm.so+0x7f9131]  LinkResolver::resolve_invoke(CallInfo&, Handle,
constantPoolHandle, int, Bytecodes::Code, Thread*)+0x4f1
V  [libjvm.so+0x687cc2]  InterpreterRuntime::resolve_invoke(JavaThread*,
Bytecodes::Code)+0x1b2
j
 org.apache.log4j.Logger.getLogger(Ljava/lang/String;)Lorg/apache/log4j/Logger;+1
j
 org.apache.commons.logging.impl.Log4JLogger.getLogger()Lorg/apache/log4j/Logger;+27
j
 org.apache.commons.logging.impl.Log4JLogger.<init>(Ljava/lang/String;)V+16
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68e746]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x995797]  Reflection::invoke(instanceKlassHandle,
methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle,
bool, Thread*)+0x5d7
V  [libjvm.so+0x996413]  Reflection::invoke_constructor(oopDesc*,
objArrayHandle, Thread*)+0x313
V  [libjvm.so+0x71cf5a]  JVM_NewInstanceFromConstructor+0x10a
j
 sun.reflect.NativeConstructorAccessorImpl.newInstance0(Ljava/lang/reflect/Constructor;[Ljava/lang/Object;)Ljava/lang/Object;+0
j
 sun.reflect.NativeConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+85
j
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+5
j
 java.lang.reflect.Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+79
j
 org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(Ljava/lang/String;Ljava/lang/String;Z)Lorg/apache/commons/logging/Log;+397
j
 org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(Ljava/lang/String;)Lorg/apache/commons/logging/Log;+187
j
 org.apache.commons.logging.impl.LogFactoryImpl.newInstance(Ljava/lang/String;)Lorg/apache/commons/logging/Log;+9
j
 org.apache.commons.logging.impl.LogFactoryImpl.getInstance(Ljava/lang/String;)Lorg/apache/commons/logging/Log;+18
j
 org.apache.commons.logging.impl.LogFactoryImpl.getInstance(Ljava/lang/Class;)Lorg/apache/commons/logging/Log;+5
j
 org.apache.commons.logging.LogFactory.getLog(Ljava/lang/Class;)Lorg/apache/commons/logging/Log;+4
j  org.apache.hadoop.fs.FileSystem.<clinit>()V+3
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68e746]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x640807]
 InstanceKlass::call_class_initializer_impl(instanceKlassHandle,
Thread*)+0xd7
V  [libjvm.so+0x642cfc]
 InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V  [libjvm.so+0x6430c1]  InstanceKlass::initialize(Thread*)+0x41
V  [libjvm.so+0x70cdab]  find_class_from_class_loader(JNIEnv_*, Symbol*,
unsigned char, Handle, Handle, unsigned char, Thread*)+0x7b
V  [libjvm.so+0x6e4874]  jni_FindClass+0x424
C  [libhdfs.so.0.0.0+0x364c]  globalClassReference+0xac
----

Any pointers on how I can make this work?