JVM Profiling, Monitoring & Troubleshooting Guide

Diagnosing production issues in high-throughput Java applications—such as unexpected OutOfMemoryError crashes, high CPU thread utilization, memory leaks, and GC pause spikes—requires a structured troubleshooting approach. The HotSpot JVM exposes rich diagnostic APIs, command-line utilities, and low-overhead profiling mechanisms designed to inspect live production environments safely.

In this operational guide, we map out a production troubleshooting decision tree, examine CLI diagnostic utilities (jcmd, jstack, jmap), walk through high CPU thread correlation workflows, and analyze Java Flight Recorder (JFR) profiling.

1. Production Troubleshooting Decision Flowchart

When an application exhibits operational degradation in production, follow a systematic diagnostic workflow to isolate the root cause before attempting code changes.

flowchart TD
    Symptom["Production Issue Detected"] --> CheckType{"What is the Primary Symptom?"}

    CheckType -- "High CPU Usage (100% Core Load)" --> TopH["1. Run top -H -p <pid> to find hot OS LWP Thread ID"]
    TopH --> HexConv["2. Convert Thread LWP ID to Hexadecimal (e.g. 15423 -> 0x3c3f)"]
    HexConv --> JStack["3. Match nid=0x3c3f in jstack thread dump to identify hot method"]

    CheckType -- "OutOfMemoryError / Memory Leak" --> HeapDump["1. Generate HPROF Heap Dump (jcmd <pid> GC.heap_dump)"]
    HeapDump --> MAT["2. Open Dump in Eclipse MAT / Inspect Dominator Tree & Retained Size"]

    CheckType -- "Unresponsive / Latency Spikes" --> JFR["1. Trigger JFR Recording (jcmd <pid> JFR.start duration=60s)"]
    JFR --> JMC["2. Analyze GC Safepoint Pauses & Lock Contention in JDK Mission Control"]

2. Core Command-Line Diagnostic Toolkit

The JDK packages command-line tools that interface directly with running JVM processes via native OS sockets and shared memory.

3. High CPU Troubleshooting Workflow

When a JVM process consumes 100% CPU on a production server, use this step-by-step procedure to locate the exact line of Java code responsible:

flowchart LR
    Step1["1. top -H -p <pid>\nFind LWP ID (e.g. 12580)"] --> Step2["2. printf '%x\n' 12580\nConvert to Hex (0x3124)"]
    Step2 --> Step3["3. jstack <pid>\nSearch nid=0x3124"]
    Step3 --> Step4["4. Exact Java Code Line Found!\n(e.g. OrderProcessor.java:142)"]

1. Find Host Process ID: Identify the JVM PID using jps -l or top. (Assume PID = 12345).
2. Find Hot Thread ID: Run top -H -p 12345 to list thread-level CPU utilization. Identify the Light-Weight Process (LWP) ID consuming max CPU. (Assume LWP Thread ID = 12580).
3. Convert LWP ID to Hexadecimal: Convert the decimal thread ID to hexadecimal format:

   printf "%x\n" 12580
   # Output: 3124

4. Locate Thread in Thread Dump: Generate a thread dump using jstack 12345 and search for nid=0x3124:

   "http-nio-8080-exec-4" #42 daemon prio=5 os_prio=0 tid=0x00007f8a nid=0x3124 runnable
      java.lang.Thread.State: RUNNABLE
           at com.enterprise.service.OrderProcessor.calculateRegex(OrderProcessor.java:142)
           at com.enterprise.service.OrderProcessor.process(OrderProcessor.java:88)

5. Root Cause Identified: Line 142 of OrderProcessor.java is executing a catastrophic backtracking regular expression in a tight loop!

4. Low-Overhead Profiling: Java Flight Recorder (JFR)

Unlike traditional profilers that inject bytecode instrumentation and degrade application throughput by 20-50%, Java Flight Recorder (JFR) is built directly into the HotSpot JVM kernel. JFR gathers execution events (CPU allocation, thread locks, file I/O, GC safepoints) with less than 1% CPU overhead, making it completely safe for continuous production monitoring.

flowchart LR
    subgraph HotSpotEngine["HotSpot JVM Kernel"]
        JFREngine["JFR Event Engine\n(Emits Allocations, Locks, Safepoints <1% Overhead)"]
    end

    subgraph DiagnosticUI["JDK Mission Control (JMC)"]
        JMCApp["Visual Analytics UI\n(Inspect CPU Spikes & Lock Contention)"]
    end

    JFREngine -->|Write .jfr Binary Stream| JMCApp

# Start a 2-minute JFR recording on a running production JVM
jcmd 12345 JFR.start name=ProdProfile duration=120s filename=/tmp/prod_recording.jfr

# Check recording status
jcmd 12345 JFR.check

# Dump recording to file for analysis in JDK Mission Control (JMC)
jcmd 12345 JFR.dump name=ProdProfile filename=/tmp/prod_recording.jfr

5. Automated Memory Dumps on OutOfMemoryError

Always configure production JVM startup flags to generate an automatic heap dump if the application encounters an OutOfMemoryError:

java -XX:+HeapDumpOnOutOfMemoryError \
     -XX:HeapDumpPath=/var/logs/java_heap_dump.hprof \
     -XX:+ExitOnOutOfMemoryError \
     -jar application.jar