OutOfMemoryError:PermGen Space: How to access local SUN 1.5 JVM JMX MBeans from command line without opening a JMX remote port

Recently, I encountered a production server crash problem that PermGen is out of memory. The error is:

java.lang.OutOfMemoryError: PermGen space
     at java.lang.Class.getDeclaredMethods0(Native Method)
     at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
     at java.lang.Class.getDeclaredMethod(Class.java:1907)
     at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1354)
     at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
     at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:421)
     at null
     at null

The application is running on Tomcat 4.1 and Sun JDK 1.5. Frankly, it's very hard to locate the problem of PermGen and our operational team declared they never reloaded the app at the runtime, which is well known of the primary cause of class loader leaking. So I decided to trace the PermGen usage after the server restarted. The tomcat had been turned on the JMX with "-Dcom.sun.management.jmxremote=true" to hook up with some SNMP library based on MBean adaptation.


Questions:

• How to find the local SUN 1.5 JVMs from the command line
• How to access local SUN 1.5 JVM JMX MBeans from command line without opening a JMX remote port
  

You can easily use jps to list all the JVMs locally.

We all know that we can benefit from the JMX support of SUN JDK from 1.5 and above by simply adding -Dcom.sun.management.jmxremote option in JVM command line. Then, you can use JMX client tools such as JConsole, JVisualVM etc. to have an insight of the JVM's runtime status either remotely or locally.

However, it will not always be the case that you can use those GUI based tools to hook up the JMX of the JVMs. One typical example is that most Unix or Linux servers are only accessible through command line or the X window is not supported at all. So when you trying to run JConsole from the remote command line, it will simply complain:

root@host01:/# /jdk1.5.0_16/bin/jconsole
Exception in thread "AWT-EventQueue-0" java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
       at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:159)
       at java.awt.Window.(Window.java:318)
at java.awt.Frame.(Frame.java:419)
at javax.swing.JFrame.(JFrame.java:194)
at sun.tools.jconsole.JConsole.(JConsole.java:65)
at sun.tools.jconsole.JConsole$4.run(JConsole.java:666)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:163)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

So you would think to find a utility can query the JMX bean directly from the command line. This will work as long as you specify the "-Dcom.sun.management.jmxremote.port=port number" where the port is used to compose the JMX service URL that the utility can connect with. However, in same cases for security reason, the port is not even specified for the JVM instance you want monitor. Now, you start to scratch your head and say, " why JConsole can connect to any local JVM enabled with pure -Dcom.sun.management.jmxremote but not mine?"

This is caused by without openning a JMX port externally, the JMX service url is somehow generated by JVM and you can connect to it unless you know exactly what is it. It's in a weird form like this:

jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjIyOC4zMi44NQAAD8I51MfIMsJGsloaHuQAAAEnPvBov4ABAHg=

This super long token generated by JVM to indicate the rmi proxy you can connect to. No wonder, you can't connect it. How comes you know this magic or even you could make up this encoded string?

On JDK 1.6, you can use attach API to find the local JVM and attach to its JMX server or start an agent if there is not one there. However, on JDK 1.5, it's not supported. Well, since JConsole can, it's best to refer to JConsole's source codes how it makes it.

It uses private SUN API to connect with local JVM. Simply saying, it utilizes the classes:

sun.jvmstat.monitor.MonitoredHost

sun.jvmstat.monitor.MonitoredVm

to facilitate its job. Those classes are in tools.jar under /lib/.

So here are some examples how to find a local 1.5 JVM enabled by "-Dcom.sun.management.jmxremote=true" and connect to it to get the PermGen memory usage.

import sun.jvmstat.monitor.*;
import sun.management.ConnectorAddressLink;

import javax.management.remote.JMXServiceURL;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.MBeanServerConnection;
import java.util.Set;
import java.util.Date;
import java.lang.management.RuntimeMXBean;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;

/**
* @author Blues in Java
*/
public class Example {
String name;

public static void main(String[] args) throws Exception {
Example app = new Example("SimpleMain");
app.permGenUsage();
}

public Example(String name) {
this.name = name;
}

private void permGenUsage() throws Exception {
String url = findLocalMonitoredVM(name);
if(url == null){
  System.out.println("Can't find jvm "+name);
  return;
}
memoryUsage(new JMXServiceURL(url));
}

/**
* Find a local monitored VM whose name matches the given parameter.
* @param name
*/
private String findLocalMonitoredVM(String vmName) {
MonitoredHost host;
Set vms;
try {
  host = MonitoredHost.getMonitoredHost(new HostIdentifier((String) null));
  vms = host.activeVms();
} catch (Exception e) {
  throw new InternalError(e.getMessage());
}

for (Object vmid : vms) {
  if (vmid instanceof Integer) {
      try {
          int pid = (Integer) vmid;
          String name = vmid.toString(); // default to pid if name not available
          String address;
          MonitoredVm mvm = host.getMonitoredVm(new VmIdentifier(name));
          // use the command line as the display name
          name = MonitoredVmUtil.commandLine(mvm);
          address = ConnectorAddressLink.importFrom(pid);
          mvm.detach();
          if(name.contains(vmName)){
              return address;
          }
      } catch (Exception e) {
          e.printStackTrace();
      }
  }
}
return null;
}

private void memoryUsage(final JMXServiceURL target) throws Exception {

// Connect to target (assuming no security)
final JMXConnector connector = JMXConnectorFactory.connect(target);

// Get an MBeanServerConnection on the remote VM.
final MBeanServerConnection remote =
      connector.getMBeanServerConnection();

final RuntimeMXBean remoteRuntime =
      ManagementFactory.newPlatformMXBeanProxy(
              remote,
              ManagementFactory.RUNTIME_MXBEAN_NAME,
              RuntimeMXBean.class);
System.out.println("Target VM is: " + remoteRuntime.getName());
System.out.println("VM version: " + remoteRuntime.getVmVersion());
System.out.println("VM vendor: " + remoteRuntime.getVmVendor());
System.out.println("Started since: " + remoteRuntime.getUptime());
System.out.println("With Classpath: " + remoteRuntime.getClassPath());
System.out.println("And args: " + remoteRuntime.getInputArguments());
System.out.println("");

final MemoryPoolMXBean memoryBean=
      ManagementFactory.newPlatformMXBeanProxy(
              remote,
              ManagementFactory.MEMORY_POOL_MXBEAN_DOMAIN_TYPE + ",name=Perm Gen",
              MemoryPoolMXBean.class);

System.out.println("---Memeroy Usage--- "+new Date());
System.out.println("Committed Perm Gen:" + memoryBean.getUsage().getCommitted());
System.out.println("init Perm Gen     :" + memoryBean.getUsage().getInit());
System.out.println("max Perm Gen      :" + memoryBean.getUsage().getMax());
System.out.println("Used Perm Gen     :" + memoryBean.getUsage().getUsed());

connector.close();
}
}

The test app SimpleMain is like:

public class SimpleMain{
public static void main(String[] args) throws Exception{
for(;;){
Thread.sleep(1000);
}
}
}

Run SimpleMain like this:

java -Dcom.sun.management.jmxremote=true SimpleMain

Run Example like this:

java -cp .:[java_home]/lib/tools.jar Example

The output is:

Target VM is: 1316@Host1
VM version: 1.5.0_19-b02
VM vendor: Sun Microsystems Inc.
Started since: 5976
With Classpath: .;C:\Program Files\Java\jre6\lib\ext\QTJava.zip
And args: [-Dcom.sun.management.jmxremote=true]

---Memeroy Usage--- Mon Mar 08 15:58:15 PST 2010
Committed Perm Gen:12582912
init Perm Gen     :12582912
max Perm Gen      :67108864
Used Perm Gen     :2231888

Extending from this, you can easily get any JMX mbean's status of local JVM on JDK 1.5 from the command line to facilitate the server side JAVA application troubleshooting.