pragmatic profiling - find out quickly why your java app is hanging

First published on March 02 2006 and tagged Tech . Java .

Hunting for thread locks by creating and studying thread dumps

Lately, our latest nightly built JBoss application was hanging and eating up to 100% CPU. No deadlock per se was detected. But we definitely had some profiling to do.

There are quite a few usefull links on jboss wiki regarding Monitoring, Management, and Profiling Tools for JBoss Software

The first options use a profiler, we first tried JProbe. It fails the 10 minutes test.

But we finally succeeded in debugging our application in a much simpler and yet efficient manner : by creating and studying thread dumps.

Here is the story:

JProbe 10 minutes test on Linux RedHat

we downloaded the JProbe community edition. we had a few issues to make it run on top of our Linux RedHat (ES version 3) OS to profile our JBoss-3.2.6 j2ee application. The first was known and fixed by Quest, we applied a patch for jboss profiling. But then the JVM crashed, I tried one or two workarounds found in the jprobe doc, but it didn’t help.

No need to say this took me more than 10 minutes. May be, we should have tried JBossProfiler, did you ?

thread dumps were helpful enough and so much quicker

Then we found very helpfull tips on javaone 2004 talk : View from the Trenches: Thread Dumps

When the plane crashes, what do you do?

  • You retrieve the Black Box

When the application server freezes in mid-flight, what do you do?

  • You retrieve the Thread Dumps

A Java thread dump is a log containing threads and monitors diagnostics. Depending on how complex your application is, your stack trace can range from a few lines to thousands.

On our version Jboss, we unfortunately don’t have the nice JMX Console thread dump generation feature starting from JBoss v3.2.8/4.0.2.

But thanks to the nice and cool Samurai 2.0.0 tool we did even better without. We forced the thread dump (kill -3 on Linux, Ctrl+Break on Dos), piped this dump into a text file and import it into Samurai. We found out in just a few seconds that many of our threads were endlessly waiting for JMS resources, this information lead us to the debugging path.

image

Note also that by just adding ``-verbose:gc'' option when launching jboss, you also get this kind of graphic for free:

image

references