Monday, March 8, 2010

OutOfMemoryError:PermGen Space: How to access local SUN 1.5 JVM JMX MBeans from command line without opening a JMX remote port

Recently, I encountered a production server crash problem that PermGen is out of memory. The error is:

java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
at java.lang.Class.getDeclaredMethod(Class.java:1907)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1354)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:421)
at null
at null


The application is running on Tomcat 4.1 and Sun JDK 1.5. Frankly, it's very hard to locate the problem of PermGen and our operational team declared they never reloaded the app at the runtime, which is well known of the primary cause of class loader leaking. So I decided to trace the PermGen usage after the server restarted. The tomcat had been turned on the JMX with "-Dcom.sun.management.jmxremote=true" to hook up with some SNMP library based on MBean adaptation.


Questions:
  • How to find the local SUN 1.5 JVMs from the command line
  • How to access local SUN 1.5 JVM JMX MBeans from command line without opening a JMX remote port
You can easily use jps to list all the JVMs locally.

We all know that we can benefit from the JMX support of SUN JDK from 1.5 and above by simply adding -Dcom.sun.management.jmxremote option in JVM command line. Then, you can use JMX client tools such as JConsole, JVisualVM etc. to have an insight of the JVM's runtime status either remotely or locally.

However, it will not always be the case that you can use those GUI based tools to hook up the JMX of the JVMs. One typical example is that most Unix or Linux servers are only accessible through command line or the X window is not supported at all. So when you trying to run JConsole from the remote command line, it will simply complain:


root@host01:/# /jdk1.5.0_16/bin/jconsole
Exception in thread "AWT-EventQueue-0" java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:159)
at java.awt.Window.(Window.java:318)
at java.awt.Frame.(Frame.java:419)
at javax.swing.JFrame.(JFrame.java:194)
at sun.tools.jconsole.JConsole.(JConsole.java:65)
at sun.tools.jconsole.JConsole$4.run(JConsole.java:666)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:163)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

So you would think to find a utility can query the JMX bean directly from the command line. This will work as long as you specify the "-Dcom.sun.management.jmxremote.port=port number" where the port is used to compose the JMX service URL that the utility can connect with. However, in same cases for security reason, the port is not even specified for the JVM instance you want monitor. Now, you start to scratch your head and say, " why JConsole can connect to any local JVM enabled with pure -Dcom.sun.management.jmxremote but not mine?"

This is caused by without openning a JMX port externally, the JMX service url is somehow generated by JVM and you can connect to it unless you know exactly what is it. It's in a weird form like this:


jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjIyOC4zMi44NQAAD8I51MfIMsJGsloaHuQAAAEnPvBov4ABAHg=


This super long token generated by JVM to indicate the rmi proxy you can connect to. No wonder, you can't connect it. How comes you know this magic or even you could make up this encoded string?

On JDK 1.6, you can use attach API to find the local JVM and attach to its JMX server or start an agent if there is not one there. However, on JDK 1.5, it's not supported. Well, since JConsole can, it's best to refer to JConsole's source codes how it makes it.

It uses private SUN API to connect with local JVM. Simply saying, it utilizes the classes:

  • sun.jvmstat.monitor.MonitoredHost
  • sun.jvmstat.monitor.MonitoredVm

  • to facilitate its job. Those classes are in tools.jar under /lib/.

    So here are some examples how to find a local 1.5 JVM enabled by "-Dcom.sun.management.jmxremote=true" and connect to it to get the PermGen memory usage.


    import sun.jvmstat.monitor.*;
    import sun.management.ConnectorAddressLink;

    import javax.management.remote.JMXServiceURL;
    import javax.management.remote.JMXConnector;
    import javax.management.remote.JMXConnectorFactory;
    import javax.management.MBeanServerConnection;
    import java.util.Set;
    import java.util.Date;
    import java.lang.management.RuntimeMXBean;
    import java.lang.management.ManagementFactory;
    import java.lang.management.MemoryPoolMXBean;

    /**
    * @author Blues in Java
    */
    public class Example {
    String name;

    public static void main(String[] args) throws Exception {
    Example app = new Example("SimpleMain");
    app.permGenUsage();
    }

    public Example(String name) {
    this.name = name;
    }

    private void permGenUsage() throws Exception {
    String url = findLocalMonitoredVM(name);
    if(url == null){
    System.out.println("Can't find jvm "+name);
    return;
    }
    memoryUsage(new JMXServiceURL(url));
    }

    /**
    * Find a local monitored VM whose name matches the given parameter.
    * @param name
    */
    private String findLocalMonitoredVM(String vmName) {
    MonitoredHost host;
    Set vms;
    try {
    host = MonitoredHost.getMonitoredHost(new HostIdentifier((String) null));
    vms = host.activeVms();
    } catch (Exception e) {
    throw new InternalError(e.getMessage());
    }

    for (Object vmid : vms) {
    if (vmid instanceof Integer) {
    try {
    int pid = (Integer) vmid;
    String name = vmid.toString(); // default to pid if name not available
    String address;
    MonitoredVm mvm = host.getMonitoredVm(new VmIdentifier(name));
    // use the command line as the display name
    name = MonitoredVmUtil.commandLine(mvm);
    address = ConnectorAddressLink.importFrom(pid);
    mvm.detach();
    if(name.contains(vmName)){
    return address;
    }
    } catch (Exception e) {
    e.printStackTrace();
    }
    }
    }
    return null;
    }

    private void memoryUsage(final JMXServiceURL target) throws Exception {

    // Connect to target (assuming no security)
    final JMXConnector connector = JMXConnectorFactory.connect(target);

    // Get an MBeanServerConnection on the remote VM.
    final MBeanServerConnection remote =
    connector.getMBeanServerConnection();

    final RuntimeMXBean remoteRuntime =
    ManagementFactory.newPlatformMXBeanProxy(
    remote,
    ManagementFactory.RUNTIME_MXBEAN_NAME,
    RuntimeMXBean.class);
    System.out.println("Target VM is: " + remoteRuntime.getName());
    System.out.println("VM version: " + remoteRuntime.getVmVersion());
    System.out.println("VM vendor: " + remoteRuntime.getVmVendor());
    System.out.println("Started since: " + remoteRuntime.getUptime());
    System.out.println("With Classpath: " + remoteRuntime.getClassPath());
    System.out.println("And args: " + remoteRuntime.getInputArguments());
    System.out.println("");

    final MemoryPoolMXBean memoryBean=
    ManagementFactory.newPlatformMXBeanProxy(
    remote,
    ManagementFactory.MEMORY_POOL_MXBEAN_DOMAIN_TYPE + ",name=Perm Gen",
    MemoryPoolMXBean.class);

    System.out.println("---Memeroy Usage--- "+new Date());
    System.out.println("Committed Perm Gen:" + memoryBean.getUsage().getCommitted());
    System.out.println("init Perm Gen :" + memoryBean.getUsage().getInit());
    System.out.println("max Perm Gen :" + memoryBean.getUsage().getMax());
    System.out.println("Used Perm Gen :" + memoryBean.getUsage().getUsed());

    connector.close();
    }
    }


    The test app SimpleMain is like:

    public class SimpleMain{
    public static void main(String[] args) throws Exception{
    for(;;){
    Thread.sleep(1000);
    }
    }
    }


    Run SimpleMain like this:

    java -Dcom.sun.management.jmxremote=true SimpleMain

    Run Example like this:

    java -cp .:[java_home]/lib/tools.jar Example

    The output is:


    Target VM is: 1316@Host1
    VM version: 1.5.0_19-b02
    VM vendor: Sun Microsystems Inc.
    Started since: 5976
    With Classpath: .;C:\Program Files\Java\jre6\lib\ext\QTJava.zip
    And args: [-Dcom.sun.management.jmxremote=true]

    ---Memeroy Usage--- Mon Mar 08 15:58:15 PST 2010
    Committed Perm Gen:12582912
    init Perm Gen :12582912
    max Perm Gen :67108864
    Used Perm Gen :2231888


    Extending from this, you can easily get any JMX mbean's status of local JVM on JDK 1.5 from the command line to facilitate the server side JAVA application troubleshooting.


    Tuesday, December 11, 2007

    Is Sun's developer over smart or just stupid?

    Look at this code, what it will come out?

    import java.text.SimpleDateFormat;
    import java.util.Date;

    public class Test{
      public static void main(String[] args){
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
        Date dt = null;
        try{
            dt = sdf.parse("2000999878-3454555-35344");
            System.out.println(sdf.format(dt));
        }catch (Exception e){
            System.out.println("error");       
        }
     }
    }

    The result is:

    247625706-07-03

    Is this what you expected? You thought it would have thrown an exception of invalid format of text you passed in.

    It's very common for a developer do date and time data conversion from text and vice versa in different format. Will it breach your boundary validation if you purely rely on this stupid SimpleDateFormat class?

    Work around so far like:

    import java.text.SimpleDateFormat;
    import java.util.Date;

    public class Test{
    public static void main(String[] args){
      SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
      Date dt = null;
      try{
        dt = sdf.parse("2000999878-3454555-35344");
        String s = sdf.format(dt);
        if(!s.equals("2000999878-3454555-35344"))
          System.out.println("Data format wrong!");
      }catch (Exception e){
        System.out.println("error");  
      }
    }

    Ugly codes, but works.

    How many online Web systems relying on Java and J2EE have those codes?

    Wednesday, November 7, 2007

    Manageable Spring Application - enable JMX in minutes

    We are keeping talking about manageability of an application. Why is it so important? Because at any stage of the application lifecycle, you need a way to probe the some key aspects of the application's internal status and take appropriate actions to change the application's behaviors as a consequence. Without it, you just guess the application's runtime status and never able to steer it per your needs.

    But hold on, it's easy to talk about manageability. And it's really a great idea in the air until you want put it in a concrete base in your application. It's so cumbersome, tedious and error prone to make an implementation. There are couples of option you could choose to inject the manageability in your application:

    1. Your own proprietary mechanism.
    2. Standard SNMP
    3. JMX

    Forget about option 1, smart as you, don't want invent any wheel. Well, SNMP (Simple Network Management Protocol) sounds good: standardized, bunch of tools and applications and really structured, until you dig in the details of all the ugly OID (Obejct Identification) definitions, binary format, pure data driven approach. And difficulties of debugging. Plus extra cost for those usable commercial SNMP tools and editors.

    Fortunately,  we are in campus of Java, which is so far the only language and platform that put the serious crucial enterprise aspects intrinsically in the body, especially manageability in an offer as JMX. For the people working for .Net, either they just don't know what is the manageability, or struggling with various proprietary approaches or annoying SNMP stuffs.

    Best of the best in Java application manageability is that we have the generous platform MBean Server as a gift from SUN in new version of JVM, which save your efforts looking for a MBean server; we have the JConsole as tool to directly craft your JMX management GUI frontend; and the offer from Spring's JMX supports. Combine them together, you can make any Java application JMX enabled in minutes.

    Here is a simple example called MockServer. It's just a simple socket server for any mock testing purpose. With JMX, you can get the information and stop it in runtime.

    Following is the partial code snippets, which is definitely the beautiful POJO. No any special MBean or MXBean stuffs in it, see!

    /**
     * A mock Socket server.
     */
    public class MockServer implements Runnable{
        private String name="";
        private int port = 80;
        private boolean bSSL = false;
        private int sotimeout = 2*60*1000;//2 minutes

        private ServerSocket listeningSocket = null;
        private boolean bStop=false;
        private long connCounts = 0; //Ongoing total connection counter.
       

    public MockServer(String name, boolean isSSL, int port, int sotimeout){
            this.name = name;
            this.bSSL = isSSL;
            this.port = port;
            this.sotimeout = sotimeout;
        }

        public String getName(){
            return name;
        }

        public int getPort(){
            return port;
        }

        public boolean isSSL(){
            return bSSL;
        }

        public int getSotimeout(){
            return sotimeout;
        }

        public boolean isStopped(){
            return bStop;
        }

        public void stop(){
            bStop = true;
            try {
                listeningSocket.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        protected void createListeningSocket() throws IOException {
            ServerSocketFactory factory = (bSSL? SSLServerSocketFactory.getDefault():ServerSocketFactory.getDefault());
            listeningSocket = (ServerSocket) factory.createServerSocket(port);
        }

        public void run(){
            System.out.println("Server: "+getName()+" started.");
            try {
                createListeningSocket();
                while (!isStopped()){
                    Socket worker = listeningSocket.accept();
                    //Spawn a thread to handle the request.
                    fork(worker);
                }
            }catch (Exception e){
                e.printStackTrace();
            }finally {
                if (null!=listeningSocket){
                    try {
                        listeningSocket.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
            }
            System.out.println("Server: "+getName()+" stopped.");
        }

        protected void fork(Socket worker){
            Thread wt = new Thread(new Worker(worker));
            wt.start();
        }

        public long getConnCounts(){
            return connCounts;
        }
        protected synchronized void addCount(){
            this.connCounts++;
        }
        protected synchronized void minusCount(){
            this.connCounts--;
        }

        protected class Worker implements Runnable {
            private Socket socket=null;

            protected Worker(Socket socket){
                this.socket = socket;
            }

            public void run() {
                addCount();
                byte[] buf = new byte[1024*10];//10K buffer
                InputStream is;
                OutputStream os;
                try {
                    //Read request from socket input stream.
                    is = socket.getInputStream();
                    int size = is.read(buf);
                    is.close();
                    //Process the request.
                    byte[] resp = processRequest(buf, size);

                    //Write back the response to socket output stream.
                    os = socket.getOutputStream();
                    os.write(resp);
                    os.flush();
                    os.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }finally{
                    if(null != socket){
                        try {
                            socket.close();
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    }
                    minusCount();
                }
            }
        }

        //Subclasses to override it.
        protected byte[] processRequest(byte[] buf, int size){

             return buf;//echo it.

       }
    }

    /**
     * A mock server services to manage the mock servers.
     */
    public class MockServerService {
        private Set<MockServer> servers = null;

        public void setServers(Set<MockServer> servers){
            this.servers = servers;
        }

        protected void init(){
            if(null != servers){
                for(MockServer server: servers){
                    new Thread(server).start();
                }
            }
            System.out.println("Service initialized.");
        }

        public Set<MockServer> getServers(){
            return servers;
        }

        public void stop(){
            if(null != servers){
                for(Server server: servers){
                    server.stop();
                }
            }
            System.out.println("Service stopped.");
        }
    }

    Then we need an application entry point and integrate with Spring.

    public class MockServer {
        public static void main(String[] args){
            AbstractApplicationContext ctxt = new ClassPathXmlApplicationContext("context-mockserver.xml");
            ctxt.registerShutdownHook();
            MockServerService service = (MockServerService) ctxt.getBean("mockServerService");
            service.init();
        }
    }

    the context file for Sring defining the beans.

    <?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="
    http://www.springframework.org/schema/beans"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">
        <bean id="mockProcessor0" class="MockServer" >
            <constructor-arg index="0" value="Server1"/>
            <constructor-arg index="1" value="false"/>
            <constructor-arg index="2" value="80"/>
            <constructor-arg index="3" value="120000"/>
        </bean>
        <bean id="mockProcessor1" class="MockServer" >
            <constructor-arg index="0" value="Server2"/>
            <constructor-arg index="1" value="false"/>
            <constructor-arg index="2" value="90"/>
            <constructor-arg index="3" value="120000"/>
        </bean>
        <bean id="mockProcessor2" class="MockServer" >
            <constructor-arg index="0" value="Server3"/>
            <constructor-arg index="1" value="false"/>
            <constructor-arg index="2" value="100"/>
            <constructor-arg index="3" value="120000"/>
        </bean>

        <bean id="mockServerService" class="MockServerService">
            <property name="servers">
                <set>
                    <ref local="mockProcessor0"/>
                    <ref local="mockProcessor1"/>
                    <ref local="mockProcessor2"/>
                </set>
            </property>
        </bean>
    </beans>

    Until now, nothing to do with JMX. You can run it as a normal Spring application. You can use JConsole to connect with it locally, if you run the application in this command line:

    java -cp . -D-Dcom.sun.management.jmxremote MockServer

    This is the snapshot of JConsole MBeans tab. Besides the default threads, memory etc. JVM MXBeans, you can't do anything else to this application.

    Now, let's just simply tweak the context file, then see what happens. Add this extra block just at the end of the context file.

    ...

        <bean id="mockServerService" class="MockServerService">
            <property name="servers">
                <set>
                    <ref local="mockProcessor0"/>
                    <ref local="mockProcessor1"/>
                    <ref local="mockProcessor2"/>
                </set>
            </property>
        </bean>
        <bean id="exporter" class="org.springframework.jmx.export.MBeanExporter">
            <property name="beans">
                <map>
                    <entry key="bean:name=MockProcessor0" value-ref="mockProcessor0"/>
                    <entry key="bean:name=MockProcessor1" value-ref="mockProcessor1"/>
                    <entry key="bean:name=MockProcessor2" value-ref="mockProcessor2"/>
                    <entry key="bean:name=MockServerService" value-ref="mockServerService"/>
                </map>
            </property>
        </bean>
    ...

    Here it is! The new JConsole MBeans tab populated with your beans. Now you can see the name of each bean and stop it just by invoking the corresponding stop() method of that bean. Done! You can manage your Spring application now!

    In a nutshell, following this pattern, you can tweak any of your applications to be manageable in minutes:

    1. Define the management interfaces for your Object.
    2. Spring your application and expose the object as MBean you want control. Nevertheless, Spring is a extremely noninvasive container, don't be afraid. The things need you Springlize is just make a context file for beans, add less than 5 lines of code to create the application context and put the spring.jar in your classpath. Everything is so familiar to you in a POJO world.
    3. Enable the JVM JMX platform MBean server in command line with -Dcom.sun.management.jmxremote and run it.
    4. Launch the JConsole and connect to the application then control it in your hands.

    Wednesday, October 24, 2007

    Bean Definition Tip1: Avoid Circular Reference

    Spring is a really powerful framework, which is not only providing the JEE similar or much better and lighter container services, but also providing a straightforward declarative and configuration driven bean definition model. With the flexibility and capability of bean definition mechanism, especially when you unleashing the power of XML schema based context file introduced since Spring 2.0, you can almost do any sorts of bean wiring, parameter configuration, application assembling tasks. After you use it, I bet you will forget all the unpleasant and cumbersome JEE configuration files and the coarse grained Enterprise Beans. However, when dealing with bean definitions, it's still very tricky and the similar problems probably happen as in your traditional programmatic approach.

    One typical problem is that the bean definition circular referencing. This is the circumstance under which there are beans relying on each others instance before they could be instantiated by Spring container.

    For below example:

    public class BeanA {
    BeanB b;
    public BeanA(BeanB beanB){
    b = beanB;
    }
    public void print(String s) {
    System.out.print(s);
    }
    public void foo(){
    b.foo();
    }
    }

    public class BeanB {
    private BeanA a;
    public BeanB(BeanA beanA){
    a = beanA;
    }

    public void print(String s){
    a.print(s);
    }

    public void foo() {
    System.out.print("foo!");
    }
    }

    public class TestStub {
    public static void main(String[] args){
    AbstractApplicationContext ctxt= new ClassPathXmlApplicationContext("beandef.xml");
    }
    }

    ...bean definitions in beandef.xml ...

    <bean id="beanA" class="BeanA">
    <constructor-arg ref="beanB" />
    </bean>
    <bean id="beanB" class="BeanB">
    <constructor-arg ref="beanA"/>
    </bean>

    When you run TestStub, it will show following exception (pay attention to the underline):

    Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'beanA' defined in class path resource [beandef.xml]: Cannot resolve reference to bean 'beanB' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'beanB' defined in class path resource [beandef.xml]: Cannot resolve reference to bean 'beanA' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'beanA': Requested bean is currently in creation: Is there an unresolvable circular reference?
    Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'beanB' defined in class path resource [beandef.xml]: Cannot resolve reference to bean 'beanA' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'beanA': Requested bean is currently in creation: Is there an unresolvable circular reference?
    Caused by: org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'beanA': Requested bean is currently in creation: Is there an unresolvable circular reference?

    It's a simple sample of bean circular reference. Specifically, the container tried to instantiate "beanA" first, then it found "beanA" need a reference of "beanB" as constructor argument; subsequently, it tried to create an instance of "BeanB" as "beanB", but this time the "beanB" need a "beanA" instance as a constructor reference argument that is still creation pending waiting for "beanB". This situation confused the container and it can not resolve all the bean references. Then it came the complains of exception. This circular reference problem was caused by circular references of bean by means of Constructor Injection.

    To solve this issue, one option is that replace constructor reference to "beanA" of "beanB" with a setter injection. Then the container will not try to create the "beanB" with the "beanB" reference at same time. Instead, it will defer this by injecting the "beanA" reference latter on via "setter" method of "BeanB" after the "beanA" has been created. The same changes happen to "BeanA". It needs some tweaks to the code and bean definitions as well.

    public class BeanA {
    BeanB b;
    public BeanA(){
    }
    public void setBeanB(BeanB b){
    this.b = b;
    }
    public void print(String s) {
    System.out.print(s);
    }
    public void foo(){
    b.foo();
    }
    }


    public class BeanB {
    private BeanA a;
    public BeanB(){
    }

    public void setBeanA(BeanA a){
    this.a = a;
    }
    public void print(String s){
    a.print(s);
    }

    public void foo() {
    System.out.print("foo!");
    }
    }


    ...

    <bean id="beanA" class="BeanA">
    <property name="beanB" ref="beanB"/>
    </bean>
    <bean id="beanB" class="BeanB">
    <property name="beanA" ref="beanA"/>
    </bean>

    ...

    This time, the Sping was very happy to create all the beans.

    An alternative option is still workable for this case but a lit bit tricky related with bean instantiating order. We can ask "beanA" injected "beanB" by setter, whereas "beanB" can still use the constructor injection. But wait, the order of bean definition is very important. You can not let constructor injected "beanB" defined before setter injected "beanA".

    public class BeanA {
    BeanB b;
    public BeanA(){
    }
    public void setBeanB(BeanB b){
    this.b = b;
    }
    public void print(String s) {
    System.out.print(s);
    }
    public void foo(){
    b.foo();
    }
    }

    public class BeanB {
    private BeanA a;
    public BeanB(BeanA a){
    this.a = a;
    }

    public void print(String s){
    a.print(s);
    }

    public void foo() {
    System.out.print("foo!");
    }
    }

    ...

    <bean id="beanA" class="BeanA">
    <property name="beanB" ref="beanB"/>
    </bean>
    <bean id="beanB" class="BeanB">
    <constructor-arg ref="beanA"/>
    </bean>

    ...

    This works very well. If you move the "beanB" definition before "beanA", it will not work because the "beanA" will need a creating "beanB".

    In a nutshell, the circular reference problem is very typical for a sophisticated "Spring" application if you did not cook it very well. It will cost you time to debug and fix it. But not just that. Because the nature of configuration driven of "Spring", the same bean definition file itself could have huge chances to be manipulated by different personnel, thus, more chances of introducing new errors regarding the "Spring" competency level. I will talk about in future sessions how to address this issue by establishing your project or organization level bean definition schema extended from "Spring" base schema. In this way, it mandates some of important project wise constraints for bean definition, such as available tags, bean types, data types etc. Thus, it could reduce some errors caused by arbitrary string parameters or typos in bean definition files for your project.

    It's strongly recommended to use setter injection for bean wiring and bean reference injection. If it's very necessary to do constructor based reference injection, and unfortunately involved with circular reference, please well comment or document the bean definition and put notices for future maintaining people. You don't just want someone else happened to ruin your whole fragile bean constructor injection hierarchy someday, do you?