Write high-performance RMI servers and Swing clients

Sometime when you have a few minutes to burn, cruise into your local discount brokerage firm and watch the brokers operating the service counter. During a typical lunch hour, customers crowd the service counter like software engineers ravaging a convention booth giving away free t-shirts. The most effective brokers help multiple customers fill out forms, process checks, direct customers to the appropriate prospectus, and answer the phone simultaneously. The brokers not destined to make a living in the world of “churn and burn” simply serialize all of their tasks: help a customer fill out a form, or handle a phone call, or process a check, or — you get the idea. The efficient broker keeps the line moving by handling multiple tasks concurrently, while the inefficient broker lets the tasks queue up as he or she works on a single task.

Which broker type best resembles your RMI, CORBA, or DCOM server process? Does your process perform synchronous writes to the log while executing client requests? Are client callbacks executed within the same method (process thread) that forced the notification? Does your server process perform callbacks in synchronized blocks? If you answered affirmatively to any of these questions, your process may resemble that of the inefficient broker. This article will provide techniques for improving the efficiency of your server, enabling it to handle more clients and return control to client processes more quickly.

Our example distributed program, SouthBay, provides a client application that allows users to bid on products and provides realtime updates on the current high bids. The client application interacts with a server application that maintains the current product bids and notifies clients when it receives a new high bid. This application uses RMI to illustrate the programming techniques, but the techniques are equally valid for CORBA-, DCOM-, or TCP-based applications.

SouthBay application overview

The SouthBay distributed application provides the user an application that displays the current list of products for sale and a server application that maintains the current high bid and informs registered clients when a price changes. Figure 1 shows the appearance of SouthBay’s Swing-based, client user interface.

Selecting an item in the Products list causes the product detail to be displayed in the Current Product panel, which shows the product ID, description, current bid, and current bid acceptance time. This being a startup e-commerce venture, we don’t have to make a profit for the foreseeable future, so no provision has been made to track users or allow them to pay for a product; after all, there has to be some incentive for them to upgrade. Clicking on the Bid on Item button pops up an input message panel into which the user can enter a new bid (Figure 2).

Pressing the OK button causes the client application to submit the bid to the server. If the new bid is higher than all previous bids, the new bid replaces the previous high bid and the server broadcasts the new bid to all registered clients. Our client registers with the server when it starts up and unregisters when the user closes the application.

RMI and threads

Since this article deals with server-side threading techniques, I would like to talk a little about how RMI uses threads to handle client requests before proceeding into the design of the SouthBay application. Although we’re dealing with RMI, the principles discussed in this section equally apply to CORBA and DCOM.

Many programmers are unsure how RMI handles concurrent calls to a single RMI object; and compounding the confusion, the RMI specification doesn’t set a policy for how calls should be dispatched by the RMI system. Essentially, a separate thread dispatches every RMI call. The implication for RMI objects is they must be thread safe, since the object may have to handle multiple requests simultaneously.

The RMI specification doesn’t identify a specific thread-dispatching policy, so each RMI implementation exhibits different runtime characteristics. In addition to the implementation that ships with the JDK, BEA Software provides an implementation with its WebXpress application server. One of the main differences between the two RMI implementations is the remote method call dispatching mechanisms. Of the two, the standard Sun implementation has the least efficient threading architecture, for it suffers from several shortcomings:

• There is no configurable upper limit on the number of server threads to handle client requests: Failure to enforce an upper limit means the server could be overwhelmed with requests and thereby exhaust its resources. Developing a solution to this problem is difficult at best because RMI doesn’t provide a hook for implementing custom threading policies as it does for custom sockets.

• Call-dispatching threads aren’t reused: This is a very simplistic means of implementing a multithreaded server that wastes resources and limits scalability. To demonstrate this behavior, I ran five SouthBay clients and alternated bid requests. Figure 3 shows the thread list at a breakpoint in the server at the time of servicing the first client update request:

  I’ve highlighted the RMI dispatcher threads, the blue arrow indicates the thread handling the client request. Figure 4 shows the stack trace after updating a bid 67 times:

  

  Notice that many of the original threads have died, and that new threads have been created in their place. Figure 5 shows the thread list after stopping the first batch of processes and starting five additional clients:

  Imagine how this would scale on a server under heavy load of multiple requests per second.

Despite these shortcomings, RMI is adequate for applications that don’t involve handling a large number of concurrent users. BEA WebXpress provides an alternative RMI implementation with vastly improved scalability.

The key point to take away from this is that RMI server processes are multithreaded and your RMI implementation must be prepared for the load.

Anatomy of a callback

The SouthBay’s RMI objects will run in a multithreaded environment, but we need to ensure the design facilitates an efficient use of the server’s resources. Ideally, the design should minimize the number of client RMI requests so the server can scale to handle more users. An effective means of reducing the number of client requests is to implement a callback model for asynchronous state change notifications.

When using server callbacks, the server’s broadcast forcefully synchronizes the clients’ state with the server. Without using a callback model, clients would have to continuously query the server for updates, thus wasting valuable network resources and server processing time.

Implementing an RMI application that supports callbacks is fairly straightforward. The first step is to declare two remote interfaces: the server-side interface and the client-side interface. The server-side interface defines two registration methods and at least one other method that triggers the callback. I will call this the state change method. The registration methods accept a remote reference to the client-side interface as their sole parameter, although some implementation may call for additional parameters. The client-side interface declares one or more methods called by the server when some client invokes one of the server’s state change methods.

The client-side and server-side remote interfaces for the SouthBay application are very simple. The client-side interface is called Bidder and declares two callback methods:

package callback;
import java.rmi.Remote;
import java.rmi.RemoteException;
public interface Bidder extends Remote {
    void updateBid(Bid b) throws RemoteException;
    void serverShuttingDown() throws RemoteException;
}

The server-side interface is called Auctioneer and declares two registration methods, a single state change method, and a query method that returns the current bids:

package callback;
import java.rmi.Remote;
import java.rmi.RemoteException;
public interface Auctioneer extends Remote {
BidException;
    void addBidder(Bidder b) throws RemoteException;
    void removeBidder(Bidder b) throws RemoteException;
    Bid[] getCurrentBids() throws RemoteException;
    void updateBid(Bid b) throws RemoteException, 
}

The registration methods normally follow the AWT event model convention for registering event listeners by using the naming convention addXxx(Xxx) and removeXxx(Xxx). Since the Auctioneer notifies Bidder instances of current bid changes, the two registration methods are called addBidder() and removeBidder(). The state change method, called updateBid(), takes a Bid instance and throws a BidException if the bid isn’t valid due to a low price or the product is not currently for sale.

Bids and products are represented using the Bid class, which is defined as:

package callback;
public class Bid implements Serializable {
   private String fId;
   private String fDesc;
   private URL fImage;
   private Date fTimeOfBid = new Date();
   private double fCurrBid;
   private transient ImageIcon fIcon;
   public Bid(String id, String desc, URL image, double price)
   public String getId()
   public String getDescription()
   public Icon getIcon()
   public Date getTimeOfBid()
   public double getCurrentBid()
   public int comparePrice(Bid b)
   public int comparePrice(double p) 
   public void updateBid(double price) 
   public boolean equals(Object o)
   public String toString()
}

Figure 6 illustrates how the Bidder, Auctioneer, and Bidder types fit together to generate callback notifications when a product bid changes.

The interaction diagram shows clients that are registering with the server by invoking the addBidder() remote method, which causes the Auctioneer to store a reference to the Bidder client in a list, such as a java.util.Vector. When a client invokes the updateBid() with a new, higher bid, the Auctioneer iterates over the list of remote client references and invokes the updateBid() Bidder method.

Basic auctioneer implementation

To begin our efforts to demonstrate simple techniques that improve the responsiveness and efficiency of the server process, we need a simple baseline implementation of the Auctioneer remote interface. The client side will be addressed briefly in a later section; however, this article primarily focuses on the server side.

The SouthBay application uses the AuctioneerImpl class as the RMI server implementation. The class is declared as:

public class AuctioneerImpl extends UnicastRemoteObject 
implements Auctioneer 
{
   Vector fClients = new Vector();
   Hashtable fProducts = new Hashtable();

The fClients Vector serves as the list of Bidder client references and the fProducts Hashtable stores the current Bid instances keyed by the ID property value. The products are installed in the constructor:

public AuctioneerImpl(Bid products[]) throws RemoteException { 
   //put products into hashtable for quick access
   for (int i=products.length-1; i>=0; i--) {
      fProducts.put(products[i].getId(), products[i]);
   }
}

The real meat of the AuctioneerImpl class occurs in the registration methods and the updateBid() method. The registration methods are very straightforward:

public synchronized void addBidder(Bidder b) {
   fClients.addElement(b);
}
public synchronized void removeBidder(Bidder b) {
   fClients.removeElement(b);
}

The updateBid() method is more involved, for it must perform some checks before accepting the bid. Specifically, it must verify the product is currently active in the auction and the new bid is for a higher dollar value than the current bid. Remember that the bid may have been the highest when the user typed it in, but another client may have jumped in just before the first bid arrived. Once the bid is accepted, the server iterates over fClients and broadcasts the new Bid to all registered clients. One or more of the client callbacks may fail; possible reasons for this include network failure, abrupt client process failure, a skeleton or stub version problem, or the client simply may have shut down without unregistering. If a client cannot be successfully updated due to a communication failure, it is removed from the list of clients. RMI generates a RemoteException to indicate a communication failure. A more sophisticated approach would allow a certain number of successive failures before rejecting a client or employing a leasing scheme as dictated in the Jini specification. With all that logic swimming in your head, here’s the updateBid() implementation:

public synchronized void updateBid(Bid bid) throws BidException {
   //make sure the bid is greater than the current bid
   Bid b = (Bid)fProducts.get(bid.getId());
   if (b==null) {
      throw new BidException("Could not locate product: " + bid);
   } else if (bid.comparePrice(b)<=0) {
       throw new BidException("Bid is not higher than the current price");
   }
   fProducts.put(bid.getId(), bid);
   if (fClients.size() == 0)  return; 
   sendBidToClients (bid);      
}
//Send the new bid to all clients. Remove any client that cannot be successfully contacted
//This is indicated by the RMI system generating a RemoteException.
synchronized void sendBidToClients(Bid bid) {
   Bidder bidder = null;
   for (int i=fClients.size()-1; i>=0; i--) {
      try {
         bidder = (Bidder)fClients.elementAt(i);
         bidder.updateBid(bid);
      } catch(RemoteException ex) {
         //client is not valid anymore so remove it from client list
         fClients.removeElement(bidder);
      } catch(Exception ex) { /* ignore */}
   }
}

The sendBidToClients() method iterates through the Bidder references in reverse order (n-1 to 0). Iterating backwards allows elements to be safely removed from the fClients Vector when a RemoteException is handled without skipping any Bidder references. The following scenario illustrates the problem encountered when iterating in the forward direction over a Vector that contains five elements and the notification of the element at index 2 fails:

1. Element 2 is removed — fClients.removeElement(bidder): all the elements are shifted down to fill the empty index (the element at index 3 moves to index 2, and the element at index 4 moves to index 3).

2. The for loop increments the counter: i is now equal to 3.

3. Element at index 3 is retrieved, effectively skipping the element that slid from index 3 to index 2!

Iterating in reverse order prevents this type of logic error without having to resort to complex and error-prone loop counter adjustments.

Notice how the registration and state change methods are synchronized. This prevents the registration methods from adding and removing elements while the sendToClients() method is performing the broadcast notification.

Use cloning to reduce synchronization requirements

With the current implementation, any client attempting to register, unregister, or update is blocked while the Auctioneer is busy performing an update, which could be quite time consuming. An AuctioneerImpl instance’s ability to concurrently handle multiple requests can be greatly increased by removing all of the synchronized blocks from the code. Sound dangerous? Yes, but it is relatively safe once a little tweak is made to the sendToClients() method:

void sendBidToClients(Bid bid) {
 <b>Vector v = (Vector)fClients.clone();</b>
   Bidder bidder = null;
 <b> for (int i=v.size()-1; i>=0; i--) {</b>
      try {
 <b>       bidder = (Bidder)v.elementAt(i);</b>
         bidder.updateBid(bid);
      } catch(RemoteException ex) {
         //not valid anymore so remove it from client list
 <b>       fClients.removeElement(bidder);</b>
      } catch(Exception ex) { /* ignore */}
   }
}

The fClients Vector is cloned, thus creating a shallow copy: same elements, different Vector. This ensures changes to the element ordering of the fClients Vector won’t change the ordering of the Vector v. If a client cannot be updated, it’s removed from the fClients Vector but not from the temporary Vector. It’s possible that the server could update a client after it has unregistered, but that’s rarely a problem worthy of note. If your application cannot tolerate any unwanted callbacks, however, you may have little choice but to use a heavy dose of synchronization.

Use asynchronous callbacks

A second opportunity for improving the AuctioneerImpl class is to change the broadcast from a synchronous to an asynchronous notification scheme. RMI synchronously executes all methods, which means an RMI method won’t return until the server completes the method and returns control to the client. If the server takes 5 minutes to perform a bid update, the poor user will see an hourglass for 5 minutes. Just think what would follow if a client happened to deadlock during an update! Remember those two stockbrokers mentioned in the beginning of the article? Well, our current implementation is the slow one, and he’s on the verge of being fired! I think our slow broker needs an upgrade. What’s needed is a way to place the new bid in a queue of updates, return control to the client process, and use a separate thread to retrieve the queued bid and perform the broadcast update. The asynchronous callback system for this example possesses the following characteristics:

• Contains a configurable thread pool size: One thread serializes the broadcast updates while an unbounded thread pool size can uncontrollably chew up resources and degrade performance. This scheme allows the developer or administrator to set the appropriate size.

• Contains a reusable pool of threads: As discussed earlier, why throw away threads when they can be reused?

• Allows thread pool priority to be specified: This feature allows the developer to decide how important it is to perform a callback; not all callbacks are created equal.

• Provides a framework that supports reusability: You may be surprised how many types of applications can benefit from asynchronous callbacks. Two additional uses for them will be demonstrated later in the article.

• Easy to use: If it ain’t easy, software engineers won’t use it.

Before discussing the implementation, let me show you the new code. The only requirement for interfacing to the asynchronous callback thread pool is to implement the com.kasoftware.Callback interface:

public interface Callback {
   public void executeCallback(Notifier n, Object arg);
}

The interface contains a single method called executeCallback(). Sounds simple so far, but what are those two parameters? The Notifier is a reference to the “notifier thread” that is performing the callback, and the Object parameter is any argument you wish to pass to the callback method (or null if one isn’t specified). The com.kasoftware.AsyncCallback class encapsulates the configurable thread pool and provides the doCallback() method to request an asynchronous callback. The class provides two forms of doCallback(): one with and one without parameter passing capability.

public class AsyncCallback {
   public AsyncCallback(int priority, int num) {
   public synchronized void start() {
   public synchronized void stop() {
   public void doCallback(Callback c, Object arg) {
   public synchronized void doCallback(CallbackRequest c) {
}

So here’s the new updateBid() method, which is used to synchronously invoke sendToClients():

public class AuctioneerImpl extends UnicastRemoteObject 
implements Auctioneer
{  //create pool of 5 threads at a slightly lower priority - more important to
   //provide quick response to client method calls than do updates
   AsyncCallback fClientCallback = new AsyncCallback(Thread.NORM_PRIORITY - 1, 
                                                     5 /*thread pool size*/);

   public AuctioneerImpl(Bid products[]) throws RemoteException { 
      //put products into hashtable for quick access
      for (int i=products.length-1; i>=0; i--) {
         fProducts.put(products[i].getId(), products[i]);
      }
    //initialize the thread pool - creates and starts the threads
      fClientCallback.start();
   }
   public void updateBid(final Bid bid) throws BidException {
      //make sure the bid is greater than the current bid
      Bid b = (Bid)fProducts.get(bid.getId());
      if (b==null) {
         throw new BidException("Could not locate product: " + bid);
      } else if (bid.comparePrice(b)<=0) {
          throw new BidException("Bid is not higher than the current price");
      }
      fProducts.put(bid.getId(), bid);
      if (fClients.size() == 0)  return; 
      //register for async callback
      fClientCallback.doCallback(new Callback() {
         public void executeCallback(Notifier n, Object arg) {
            sendBidToClients(bid);
         }
      });
   }
}

If the anonymous inner class is too obfuscated for your tastes, then use this more traditional approach:

public class AuctioneerImpl extends UnicastRemoteObject 
implements Auctioneer, Callback  
{ 
public void executeCallback(Notifier n, Object arg) {
sendBidToClients((Bid)arg);      
}

public void updateBid(final Bid bid) throws BidException {
//make sure the bid is greater than the current bid
Bid b = (Bid)fProducts.get(bid.getId());
if (b==null) {
throw new BidException("Could not locate product: " + bid);
} else if (bid.comparePrice(b)<=0) {
throw new BidException("Bid is not higher than the current price");
}
fProducts.put(bid.getId(), bid);
if (fClients.size() == 0)  {
//no clients so return control without a callback
return; 
}
//register for async callback
     fClientCallback.doCallback(this, bid);
   }
}

If you compare the latest updateBid() implementation with the original version, the only difference is a single, simple modification that enables the new Auctioneer implementation to use asynchronous callbacks. If this class were a stockbroker, it would be planning how to spend a fat commission check.

The AsyncCallback class creates and starts a pool of “notifier” threads when the start() method is invoked. These new threads immediately block, waiting for a Callback instance to show up in a queue (implemented using a Vector). When the AsyncCallback places a Callback instance in the queue, a notifier thread is unblocked. The notifier thread performs three actions:

1. Inspects the queue
2. Removes the first Callback in the queue
3. Invokes the Callback instances executeCallback() method

Adding asynchronous logging to SouthBay

Many server implementations provide logging capabilities so developers, administrators, and users can examine the runtime behavior and tweak configuration settings to improve performance, security, and capabilities. With SouthBay being an aspiring startup venture destined for the big time, you would expect the AuctioneerImpl class to provide world-class logging capability. And SouthBay doesn’t disappoint: it provides a handy-dandy Swing-based logging system that graphically lists all server activities. The logging window is shown in Figure 8.

Here is a code snippet from the AuctioneerImpl class that demonstrates how to use the log:

public class AuctioneerImpl extends UnicastRemoteObject 
implements Auctioneer  
{
  Logger fLog = new Logger(this);
   public void addBidder(Bidder b) {
      fClients.addElement(b);
     fLog.log("Added new client");
   }

Here is the code for the Logger class:

public class Logger extends JDialog implements Callback {
 <b> //asynchronous logging with pool of one thread at lowest priority
   //not all thread pools have to be big - having one thread ensures
   //log entries are serialized
   AsyncCallback fLogCallback = new AsyncCallback(Thread.MIN_PRIORITY, 1);
</b>   AuctioneerImpl fAuctioneer;
   JList fLog = new JList(new DefaultListModel());
   public Logger(AuctioneerImpl auctioneer) {
      fAuctioneer = auctioneer;
      <b>    fLogCallback.start();
    </b>  
      JScrollPane sp = new JScrollPane(fLog);
      getContentPane().add(sp);
      setSize(200, 300);
      show();
      addWindowListener(new WindowAdapter() {
         public void windowClosing(WindowEvent e) {
            fAuctioneer.closing();
            System.exit(0);
         }
      });
   }
   <b> public void log(String msg) {
      fLogCallback.doCallback(this, msg);
   }
   public void log(Exception ex) {
      fLogCallback.doCallback(this, ex);
   }
   public void executeCallback(Notifier n, Object arg) {
      ((DefaultListModel)fLog.getModel()).addElement(arg);
      fLog.repaint();
   }</b>
}

As you can see from the log() method, this logger performs all logging asynchronously using the lowest possible thread priority. Generating logs typically involves relatively slow I/O operations to the filesystem. Writing to the log using a low-priority thread allows the RMI server to immediately respond to new service requests, yet it allows logging to take place during idle time.

Swing doesn’t like RMI, CORBA, or DCOM callbacks

Until this point I’ve concentrated on the server-side implementation of callback broadcasts, but there is one important Swing characteristic that needs to be addressed: Swing isn’t multithread safe. This means most changes to Swing components need to occur in the AWT event dispatching thread. In the SouthBay application, the client process needs to perform special processing to update the Current Product panel’s bid price when the Auctioneer asynchronously invokes the Bidder‘s updateBid() method. A screenshot of the SouthBay client application is shown in Figure 9. The Swing Connection contains detailed information about this requirement, but the implementation is fairly painless. Swing provides the javax.swing.SwingUtitlities class, which defines the invokeLater(Runnable doRun) method that causes doRun.run() to be executed asynchronously on the AWT event dispatching thread. The BidderFrame.BidderImpl class uses the following code to implement the updateBid() method:

public class BidderFrame extends JFrame {
  void updateUI() {... }
   class BidderImpl extends UnicastRemoteObject implements Bidder {
       public void updateBid(final Bid b) {
         //locate bid and replace in array
         for (int i=fProducts.length-1; i>=0; i--) {
            if (fProducts[i].equals(b)) {
               fProducts[i] = b;
            }
         }
         if (b.equals(fCurrItem)) {
           SwingUtilities.invokeLater(new Runnable() {
               public void run() {
                  updateUI(b);
               }
            });
         }
      }

Using Swing in a multithreaded environment is the topic of several articles written by the fine engineers of Sun’s Swing team. In these articles the authors explain the underlying reasons for Swing’s single-thread model and the programming techniques for utilizing Swing in a multithreaded environment. References to the Swing articles can be found in Resources at the end of this article. My goal here is to bring this subtle yet important topic to your attention now so that you aren’t caught unaware when a customer calls with a bug report.

Conclusion

So, which stockbroker does your server processes resemble? The slow one or the efficient one? I hope your answer is the latter, which allows clients to get in, do their transactions, and get out quickly. If your server processes remind you of the broker serially servicing a line of customers, I hope this article provided some techniques that will enable you to quickly and effortlessly upgrade your programs.

Be happy in your coding.