How we tested the workstations

They say the devil is in the details, but nowhere is this more evident than in PC and workstation benchmarking. After receiving a flood of e-mails from readers requesting additional information on the test methodology behind our recent Opteron vs. Xeon workstation comparison, I’ve decided to publish the following for those rabid fans with a need to read between the box scores. Here you’ll find a more detailed disclosure of the test-bed hardware, as well as a detailed review of each workload object and its core functionality/behavior.

As I mentioned in my original sidebar, the target of the underlying test methodology was a next-generation, high-performance workstation user environment similar to those I’ve analyzed at leading financial services companies. These companies typically employ very sophisticated data-mining and workflow-automation systems that involve numerous parallel connections to back-end client/server database and messaging servers. They also run some unexpected workloads, including streaming playback of concurrent multimedia feeds for tracking market indexes and monitoring key business-news media reports. It’s a truly dynamic run-time environment, one that puts tremendous stress on the full range of hardware subsystems.

To reproduce this kind of compute-intensive workload, each test bed was configured with a clean installation of Windows XP Professional Service Pack 1, appropriate device drivers (chip set, disk, video, NIC), and any necessary software to support the various Clarity Studio workload packages (SQL Server 2000 Developer Edition, Windows Media 9 Series Encoder). Clarity Studio is the multiprocess, client/server workload-simulation framework we used to generate the parallel workload scenarios. (Linear scenarios were accomplished using OfficeBench 4.0, which is also a component of the Clarity Studio framework.)

We scaled the workload by increasing the number of concurrent instances of each workload object and then repeating the test package — a 5-minute window in which the parallel workload objects executed in a loop as their individual completion times were recorded and averaged. By starting with a relatively lightweight scenario (3x each workload object) and then scaling the test package to include 5x, 7x, and 10x each workload object, respectively, we were able to monitor both individual package performance and the effect of steady workload growth over time.

Test-bed configurations

To better focus the report on CPU and chip-set platform performance, we took steps to deliberately level the playing field in terms of memory size, disk type, and video adapter. This served to eliminate these highly dynamic elements from the analysis process, ensuring an apples-to-apples comparison.

Database workload

The database workload consisted of a multi-instanced ADO (ActiveX Data Objects) client simulation object conducting OLE DB-based transactions against a local installation of Microsoft SQL Server 2000 Developer Edition. The actual transaction script — embodied in the ADO Stress 4.0 Workload Simulation object, part of the CSA Research Clarity Studio load simulation and testing framework — involved copying the contents of a source table to a temporary target table and then executing various T-SQL SELECT commands before copying, moving, or deleting records. The script then deleted the temporary table and closed the connection before repeating the process.

The database workload was scaled in concert with the other workloads per the schedule we outlined previously. Each instance employed six concurrent execution threads, mostly related to OLE-DB connection maintenance and cursor management. The workload was configured to read from a source table with 250 rows and to write 100 percent of the row data as part of its interaction with the target table.

Workflow workload

As with the database workload, the workflow workload involved the use of a discreet, task-specific scripting object, in this case the Clarity Studio MAPI Stress 4.0 Workload Simulator. Once again, multiple instances of the script were executed at the client, but instead of conducting SQL transactions, this object employed the Microsoft CDO (Collaboration Data Objects) APIs to access a PST (Microsoft Personal Folders) message store.

The actual test script logic consisted of copying 25 different message and document objects — Word, Excel, PowerPoint, and Adobe PDF files — from a single source folder located in the aforementioned PST store to five target folders located in the same message store. The script then deleted the target folders and closed the MAPI connection before repeating the transaction.

The workflow workload was scaled in concert with the other workloads, according to the schedule outlined at the beginning of the article. Each instance employed nine concurrent execution threads, mostly to maintain the CDO/MAPI connection and to manage the nearly 1.5MB of message and attachment data being copied per execution loop.

Media playback workload

The media playback workload (Clarity Studio WMP Stress 4.0) employed an embedded Windows Media Player object (msdxm.ocx) to access and play a streaming Windows Media clip (welcome2.asf). Each instance of the object spawned 14 concurrent execution threads, most dedicated to servicing the Media Player object and managing I/O to the .asf file.

As were the other workloads detailed here, the media playback workload was scaled in concert, increasing the number of active playback tasks according to the test schedule outlined at the beginning of the article.

Media encoding workload

The media encoding workload consisted of a multi-instanced encoding application (Clarity Studio WME Stress 4.0) that leveraged the Windows Media Encoder libraries to re-encode a streaming media clip (welcome2.asf), changing both the clip’s target playback bit rate and screen resolution. The recoded file was then written to disk and the process repeated.

As with the other workloads, the media encoding workload was scaled according to the schedule outlined at the beginning of the article. Each instance of the workload process spawned 23 concurrent execution threads, most to service the Windows Media Encoder libraries and the I/O operations for source and target clip files.

Other active processes

In addition to launching and controlling the workload processes, Clarity Studio also maintained its own background metrics agent process for tracking system and process run-time behavior. This multithreaded COM server utilized the Windows Performance Data Helper libraries to instantiate and sample performance counters for various system metrics, as well as process metrics for each running task. This data was later aggregated along with the test result data to a centralized SQL Server database where it was analyzed using the IT Clarity Service Web Portal’s Report Card template.

For some, only the raw data — without any massaging, correlation, or interpretation — will satisfy their quest for answers. Here, in all their gory detail, are the raw scores for each of our test-bed platforms.