Why ETrade banks on Java to keep bulls and bears in line

Bears and bulls wreak havoc on Wall Street, so imagine the chaos they create when storming the information highway. At San Francisco-based ETrade’s Mutual Fund Center, keeping the animals in well-ordered queues is a formidable task now entrusted to Java.

Online trading rose 44 percent between the first quarter of 1997 and the close of the third quarter, as service providers slashed commissions and saw their own firms’ stock values increase 50 percent in the third quarter alone. As online trading firms jockeyed for position, the lure of the Internet and lower commissions continued to attract more customers from traditional brokerages and boost online demand.

The problem for the online trading industry was clearly demonstrated in late October 1997, when trading firms were caught generally flat-footed by more than 200,000 online trades placed in response to a sharp market downturn. That is significantly higher than the 138,000 trades-per-day third-quarter average, and obviously does not include traders locked out by overloaded systems. While stock prices quickly rebounded, the online trading firms were left with the long-term task of repairing customer relations and shoring up systems that sagged or buckled under the load.

Shoring up ‘Net-based trading infrastructures is crucial because online traders now garner 30 percent of all the brokerage fees in the discount trading industry, and only 5 percent of all retail brokerage fees. In short, there is plenty of room to grow, and a lot of motivation for both consumers and service providers to facilitate a continued shift to online trading.

Using Java and Kiva to tame the beasts

Long before the events of late October underscored the entire online trading market’s need to beef up its trading infrastructures, ETrade Securities Inc. began addressing this need. The firm had already looked at its options, worked furiously for a few months, and rolled out its “Mutual Fund Center” in mid-November. The infrastructure relies heavily on Java and uses the Java-based Kiva Enterprise Server. (Following this deployment, Netscape Communications completed its acquisition of Kiva Software, so the third-party technology now belongs to the Mountain View maven of Web browsers and servers.)

ETrade’s Mutual Fund Center supports more than 3,500 funds, provides tools for assisting users in conducting research (historical charting, comparative performance charting, price quotations, and so on) and placing trades. ETrade provides its own full-service back-end trade processing center.

Why Java

Pick your favorite buzzword — scalability, reliability, maintainability, ease-of-deployment — the senior technical folks at ETrade will tell you that is why they opened their checkbooks and bought Kiva. Regardless of Kiva’s Java-based internals, ETrade had the option of writing its Kiva “App Logic” components in either C++ or Java. It chose Java, says Tim Fleming, vice president of technical planning and development for ETrade, in spite of the expectation that Java would execute more slowly than compiled C++ code. The upside, says Fleming, was twofold: He could much more easily find quality Java programmers, and Java was the better direction for long-term technical investments.

“Everything we buy or build has to scale very well, and the Kiva server is fully multithreaded,” says Bennett Ting, ETrade’s senior software developer. “We use Kiva to drive the entire conversation between what we offer and the customer’s Web browser — the session state, dynamic page generation, where the user came from, what the user has selected, where they go to, etcetera,” he says.

ETrade’s infrastructure consists of 30-plus Web servers, a series of Kiva application servers, a series of Tuxedo application servers, and a series of databases. Both the Kiva layer and the Tuxedo layer can handle and distribute load factors over multiple servers, says Fleming. Moreover, this entire infrastructure — Web servers, Kiva servers, Tuxedo servers, and databases — is fully reproduced and mirrored at the firm’s Sacramento data center, making everything fully redundant and thus minimizing the possibility of online service downtime.

The Kiva layer is used to normalize the Web conversation — interactions between the Web servers and clients’ browsers — and then pass requests to a series of services. The Kiva layer serves up applications to get a quote, buy or sell something, do research, and so on. These requests are simplified into discrete requests and, in turn, converted into a series of API calls that are handed off to Tuxedo services such as order management, quote management, and portfolio management. These in turn go to various databases.

“Everything turns out to be a series of simple requests that are passed from one layer to the next, and then back to the user,” says Fleming. “This gives us a tremendous amount of flexibility to write a service only once but be in a position to deploy it over multiple gateways.”

With six months of experience, ETrade is now incrementally deploying a similar Kiva-Tuxedo architecture under the rest of its electronic commerce infrastructure. The entire ETrade Australia portion of its business and selected portions of its North American applications beyond mutual funds are now running on top of Kiva.

Failover code

The Kiva version 1.5 infrastructure deployed by ETrade in November delivered tremendous functionality, says Fleming, but needed additional code to ensure robust “failover” capabilities to prevent (or at least minimize) downtime.

“If you’re running a storefront, it is one thing to go down for awhile and delay your customers from purchasing books, but with a rapidly shifting [stock] market, downtime is simply not acceptable,” says Fleming.

Custom code was written, with Kiva’s help, to ensure the infrastructure would provide uninterrupted operations 24 hours a day, 7 days a week.

The Kiva Enterprise Server version 2.1 now in place provides very similar failover support out-of-the-box. Session keys (which contain all the crucial information about who is doing what) are distributed over several servers. If one of the servers goes down, the users’ Web session automatically rolls to another server with only a momentary delay, and perhaps only requiring a second mouse-click.

Version 2 also incorporates a just-in-time compiler (JIT) to speed execution, something Ting says he eagerly awaited. Kiva still is certifying and deploying the new version and has not yet generated hard performance metrics to quantify the impact of the JIT, but anecdotal experience leads Fleming to expect the performance boost to be notable.

Capacity and performance

ETrade’s goal is to operate on typical trading days at 25 percent of its end-to-end transaction throughput capacity. Then when the market goes haywire, it has 75 percent of its capacity in reserve to deal with frantic traders.

According to Fleming, the biggest challenge in the original development and deployment process was the performance and tuning of the Java code. ETrade wrestled with striking a balance between how many distinct objects to create versus how big objects should be, and how many objects to put together.

Smaller objects maximize object reuse because they deliver a small, discrete bit of functionality. The downside is that this approach requires many more objects than a large-object approach.

“The first time we did it we put together over 100 objects,” says Ting, taking the goal of reuse to an extreme.

The Mutual Funds project was developed in about three months. More than two weeks of that time was spent figuring out performance issues. “Once we got that one nailed down, it was fairly straightforward to bang out the methods and put things together. It took some pretty long hours and hard work, but those hours were very productive,” says Fleming. “Java probably cut two-thirds or more off the timeline. Doing this in C++ would have taken a year or more,” he says.

On The GUI side

There is certainly a lot more Java-related work going on at ETrade than just the middleware infrastructure. The GUI is another area in which Java is being utilized.

For more than a year, ETrade has offered Java-based historical charting and comparative performance charts. These applets, which were originally developed by Neural Applications as part of its NetProphet™ software, have garnered a great deal of praise for the information they can provide — but also some criticism for long download times.

Applet size and download time are concern shared by other companies offering online trading information and services — including Quote.com, an information service provider that serves the public and online brokerages such as ETrade, Schwab, and Fidelity. (See sidebar below.)

One of the principal lessons learned from spending a year delivering Java applets to customers, according to ETrade, is that the firm must do a better job of managing user expectations and of finding ways to speed performance.

“Once you have the applets downloaded, they run very quickly,” says Pam Kramer, ETrade’s vice president of product management, “but we are certainly looking to cut download times.”

One possible solution is to partition some of the functionality in the applet so that more is done on the server and less needs to be downloaded to the client browser, but currently no such implementation effort is underway.

ETrade currently is prototyping new applets and applications using the Java Foundation Classes (JFC), which Kramer says look very good. And with JFCs becoming a part of core Java, Kramer hopes greater GUI functionality will be delivered without requiring clients to download as much code.