Futuremark says Nvidia didn’t cheat, but broke rules

BOSTON – Futuremark, the developer of the benchmarking software 3DMark03, has retracted accusations of cheating it leveled against graphics chip vendor Nvidia over modified software drivers on Nvidia products. However, it remains opposed to Nvidia’s techniques and is not reinstating the original benchmark results.

Futuremark has changed its description, but not its opinion, of Nvidia’s decision to use application-specific optimizations in its drivers that detected the presence of specific 3DMark03 tests and altered the performance of its graphics chip to achieve a higher benchmark result. Benchmarks are software tests used to simulate application behavior on a processor or device. They are designed to be used by buyers of hardware to compare the performance of different products.

On May 23, Futuremark published a seven-page report detailing eight specific examples of where it believed Nvidia unfairly modified drivers after detecting certain types of shaders in 3DMark03 tests. Some shaders — code that denotes how the surface of an object should appear — in 3DMark03 were replaced in favor of ones in Nvidia’s drivers that rendered the image more quickly, but at the expense of overall image quality, Futuremark said in that report. 

“Please note, that the cheating described here is totally different from optimization. Optimizing the driver code to increase efficiency is a technique often used to enhance game performance and carries greater legitimacy, since the rendered image is exactly what the (game) developer intended,” Futuremark said in its report.

However, the rhetoric has cooled down, at least on Futuremark’s part.

Futuremark, in Espoo, Finland, released a statement on its Web site Monday that read, “Futuremark now has a deeper understanding of the situation and Nvidia’s optimization strategy. In the light of this, Futuremark now states that Nvidia’s driver design is an application specific optimization and not a cheat.”

However, application-specific optimizations are still forbidden under Futuremark’s rules, said Tero Sarkkinen, executive vice president of sales and marketing for Futuremark. The benchmarking company put out its most recent statement because it believes the industry will not benefit from an extended “soap opera” between itself and Nvidia, he said.

Nvidia stands by its drivers, which have not been changed and are still available for download on its Web site, said Derek Perez, an Nvidia spokesman. The Santa Clara, California, company still believes it did nothing wrong and will continue to use optimizations, despite Futuremark’s objections, Perez said.

“They made a bunch of rules without understanding the intricacies of the architecture,” he said. “There are application-specific optimizations in every CPU and every GPU.”

Rival ATI Technologies was quite willing to continue the mudslinging. “It’s a shame that a large company with a lot of money and access to expensive lawyers can beat a small company into submission,” said Chris Evendon, director, public relations for ATI.

Along with its report on May 23, Futuremark released a new build of 3DMark03 with slightly different code that did not affect the workload required by the tests. That release lowered the performance of Nvidia’s GeForceFX 5900 Ultra processor by 24 percent and is the most accurate representation of processor performance, Sarkkinen said.

The issue stems from the different architectures used by Nvidia and ATI in developing their graphics processors, said Jon Peddie, president of Jon Peddie Research, in Tiburon, California.

Nvidia has both a 16-bit and a 32-bit architecture in its GeForce FX chips, while ATI uses a 24-bit architecture for its Radeon products, he said. 3DMark03 was written for the latest version of Microsoft’s DirectX 9.0 graphics technology, which also uses a 24-bit architecture, he said.

Generally speaking, a 24-bit architecture will render a image faster than a 32-bit architecture, but the 32-bit architecture will produce sharper images on applications that can take advantage of that sharper image. In this situation, Nvidia replaced the 24-bit shaders with its own 12-bit shader that rendered the image more quickly, but at the expense of image quality, Peddie said.

This was immediately decried by Futuremark and hundreds of gamers as cheating, since Nvidia changed the outcome of the image. Perez denied Tuesday that the optimization resulted in a poorer image.

Futuremark also mentioned in its report that ATI used optimization in its drivers, but ATI defended that move last week by pointing out that ATI only changed the order in which the 3DMark03 instructions flowed through its processor, without changing the instructions themselves.

However, because ATI’s driver detected the 3DMark03 test, it was also prohibited under Futuremark’s rules, Sarkkinen said. ATI has already said it will remove this optimization from its drivers and will release a new driver as part of its regular release schedule within about three weeks, Evendon said.

The end result of this saga has further blurred the line between cheating and optimizing. Futuremark has nothing against general optimizations that improve gameplay, but the use of optimizations that detect the presence of benchmarking software and adjust the performance accordingly will mean Futuremark cannot guarantee the reliability of its results, Sarkkinen said.

Any user turning to benchmarks for support in making buying decisions will make the best choice after consulting numerous sources and not relying on any one result, Peddie said.

“You cannot use one benchmark that produces one score to evaluate multiple architectures. It’s impossible, misleading, and lazy,” he said.