Linux clusters serve the low end

When researchers unveil the biannual Top 500 list of the world’s fastest supercomputers at the SC2004 Conference in Pittsburgh this week, much of the focus will be on the two-year reign of Japan’s Earth Simulator as the fastest of the fast. But a more interesting story may be developing at the lower end of the list, where Linux clusters are making supercomputers smaller and cheaper to manufacture than ever before.

Linux will power 60 percent of the high-performance computing systems on the list, once dominated by Unix-based supercomputers.

Thanks largely to inexpensive and powerful processors from Intel and AMD — and to the distribution of the Linux operating system known as Beowulf — users are able to build supercomputers on servers with as few as 64 processors, said Thomas Sterling, one of the creators of Beowulf and a faculty associate with the Center for Advanced Computing Research at the California Institute of Technology.

The majority of Beowulf clusters are not creating massive Linux supercomputers but are running small systems that perform the kind of analysis and simulation applications that previously were done on more costly Unix systems. As the focus on Linux clusters moves beyond high-performance computing, Unix software vendors are starting to pay attention, Gartner analyst George Weiss said.

Oracle, which began promoting software for clustered architectures with the release of Oracle9i in 2002, is a prominent example of a vendor porting commercial software to a clustered architecture. The company bills the software as an ideal application for four- to eight-node clusters.

And, new applications are evolving. Penguin Computing, which in 2003 acquired cluster software vendor Scyld Computing, has created a version of the Beowulf Linux distribution that can be used to deploy, provision, and centrally monitor clustered applications, said Donald Becker, Penguin’s CTO and a founder of Beowulf.

Becker believes that many of the features used to provision and manage scientific applications with Beowulf clusters can eventually be used as the basis of a software platform that could be used for a wider variety of applications, similar to Sun’s N-1 platform or IBM’s concept of on-demand computing.

The idea of extending Beowulf to make it a management platform for Linux applications makes sense to Rob Shanks, president of GlobeXplorer, which uses a cluster of approximately 50 dual-processor Linux servers to provide online access to aerial images and maps. “When you install multiple applications in a clustered environment and they’re not meant to be clustered, you have a lot of problems with the Tomcat and Apache software working together. The clustering environment requires that kind of management,” Shanks said.