How Do I Analyse a Java Core Dump?
<p>I have several legacy Java webapps in production running in Tomcat on RHEL6 using the latest Oracle Java Server JRE (1.8.0_91). Over the last few weeks, I've had a total of 7 JVM crashes over 4 instances with no obvious pattern of time or which instance crashes. In order to try figure out the cause, I enabled core dumps.</p>

<p>With the latest crash a core dump was produced, however I can't seem to get any information through the standard Java tools like jstack and jmap:</p>

<pre><code>[root@c6e6c1b0b90e core-dump]# jstack -J-d64 /usr/jdk/jdk1.8.0_91/bin/java core.24182 
Attaching to core core.24182 from executable /usr/jdk/jdk1.8.0_91/bin/java, please wait...
Error attaching to core file: Can't attach to the core file
sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:286)
    at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:673)
    at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
    at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
    at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
    at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:156)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:191)
    at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
    at sun.tools.jstack.JStack.main(JStack.java:106)
</code></pre>

<p>I have limited success with gdb, in that I can inspect the backtrace and frame but debug symbols are not available and there doesn't seem to be much information available to point to the source of the issue:</p>

<pre><code>(gdb) bt
#0  0x00000036186325e5 in raise () from /lib64/libc.so.6
#1  0x0000003618633dc5 in abort () from /lib64/libc.so.6
#2  0x00007fe7462f8aa5 in os::abort(bool) () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#3  0x00007fe746497593 in VMError::report_and_die() () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#4  0x00007fe7462fe2cf in JVM_handle_linux_signal () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#5  0x00007fe7462f4a63 in signalHandler(int, siginfo*, void*) () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#6  &lt;signal handler called&gt;
#7  0x00007fe745f9d655 in G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*) () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#8  0x00007fe745f9e20b in G1ParScanThreadState::trim_queue() () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#9  0x00007fe745f78af7 in G1ParEvacuateFollowersClosure::do_void() () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#10 0x00007fe745f84623 in G1ParTask::work(unsigned int) () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#11 0x00007fe7464b762f in GangWorker::loop() () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#12 0x00007fe7462f9f78 in java_start(Thread*) () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
#13 0x0000003618a07aa1 in start_thread () from /lib64/libpthread.so.0
#14 0x00000036186e8aad in clone () from /lib64/libc.so.6

(gdb) frame 7
#7  0x00007fe745f9d655 in G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*) () from /usr/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so
(gdb)
</code></pre>

<p>So my questions are:</p>

<ol>
<li>How can I use jstack/jmap to analyse the core dump?</li>
<li>Can I dig in further with gdb? (It's not my forte)</li>
<li>This feels like a JVM bug to me. How can I confirm that?</li>
</ol>

<p>Thanks.</p>