Companies lining up to root out rootkits

Stealthy, remote system access programs called “rootkits” could fuel the next big wave of malicious code, and are already beginning to influence the design of new Internet worms and viruses, according to security experts. Now security software companies are sitting up and taking notice, releasing software that can spot and remove rootkits from infected systems.

In recent weeks a handful of companies, including antivirus company F-Secure, Sana Security and free software site Sysinternals released products they claim can ferret out kernel rootkit programs that manipulate Microsoft’s Windows operating system and evade security software. But the buzz about rootkits may be overblown, according to one leading malicious code expert who says that the powerful programs, while dangerous, will never become as widespread as current viruses, worms or spyware.

Rootkits are malicious programs that are designed to be invisible, often replacing core operating system functionality with a version of the same functionality that provides remote attackers with a back door into compromised systems, said Al Huger, senior director of engineering at Symantec.

Kernel rootkits have been around since 1994, when the first “proof of concept” program was developed that evaded detection by loading and hiding in the Solaris kernel, or core processing center, he said.

While they’re not new, rootkits have been the focus of increased energy and attention in underground malicious code-writing communities, and have begun to influence more common threats, such as e-mail viruses and worms, said Mikko Hyppönen of F-Secure.

Two recent viruses, Myfip.H and Maslan.A, both have stealth features borrowed from rootkits, Hyppönen said. Maslan.A hides files and folders it needs to run, so that they cannot be seen from within Windows by an administrator. Myfip.H manipulates the Windows kernel to hide the memory process used by the virus, according to F-Secure.

Those features make it very difficult for most antivirus products, including F-Secure’s, to spot the programs, because antivirus software typically relies on telltale virus “signatures,” such as executable file names, memory processes, or folders that are evidence of infection, Hyppönen said.

To counter the new threats, F-Secure released an evaluation version of a rootkit detection program called BlackLight on March 10. The software program looks for telltale rootkit behavior, such as programs that are attempting to hide processes, files, folders or configuration settings, he said.

F-Secure is planning to roll BlackLight into its consumer and enterprise antivirus products, which will allow the company to spot rootkits before they are installed on customer systems, and detect infections on machines that have already been compromised, Hyppönen said.

Another free program, named RootkitRevealer, takes a similar approach to BlackLight, said Mark Russinovich, chief software architect of Winternals Software LP of Austin, Texas, which operates the SysInternals free software site.

RootKitRevealer analyzes instructions from application program interfaces (APIs) at the kernel level and in the Windows user environment, and then compares the results of those scans. The approach is designed to spot rootkits by recognizing when they manipulate system data at either location, he said.

Sana Security, of San Mateo, California, said the latest version of the Primary Response intrusion prevention system technology can spot rootkits. Primary Response 3.0 uses technology called Active Malware Defense Technology (Active MDT) that analyzes the behavior of memory processes or applications on a machine over time and flags malicious behavior, or software that is trying to evade detection, the company said March 7.

However, there are limitations to some of the new detection programs, said Jamie Butler, director of engineering at HBGary, Inc. and author of the FU rootkit and VICE rootkit detection programs.

For example, Rootkit Revealer can’t detect instances of FU, because it looks only for manipulated Windows registry entries and hidden files, not the temporary memory processes where FU runs, Butler said.

F-Secure’s BlackLight can detect FU, but “not for long,” Butler warns. A new version of FU will counter BlackLight’s detection mechanism and force F-Secure to go deeper into the kernel to spot FU installations, he said.

“It’s a chess game. Right now, F-Secure has beaten FU, but I’ll come up with a better technique and that will make them get better,” he said.

But not everybody is convinced that rootkits are a major threat worthy of new products.

“If we have seen an increase in (rootkit infections) it’s not significant enough to warrant attention,” said Symantec’s Huger.

Unlike spyware or Trojan horse programs, rootkits are typically used in targeted attacks on systems that malicious hackers hope to control for a long time, not scatter-shot virus or worm attacks, Huger said. Symantec’s antivirus software can spot many different kinds of rootkits, but treats them the same as other kinds of malicious code, and can remove some kinds of rootkit infections, he said.

“At the end of the day, if you have a piece of malicious code on your system that you don’t want there, its all the same thing to us. We’ll find it, remove it, and keep you safe,” Huger said.

Russinovich of Sysinternals agrees that rootkit infections are not widespread.

More than 49,000 copies of RootkitRevealer were downloaded from the Sysinternals Web site after the free product was released on Feb. 22. However, the company hasn’t received many reports of infections from those who downloaded the program since then. Still, rootkits could be very useful to online criminal groups, or even cyberterrorists, especially when married to rapid propagation features of worms and viruses, Russinovich said.

Researchers at F-Secure agree. Despite scant evidence of widespread rootkit infections, the company is betting that technological advances from rootkit authors will be making their way into the next generation of mass-propagating malicious code, and has had a separate team of researchers working exclusively on rootkits for the past year, Hyppönen said.

The back and forth between rootkit authors and security companies is also forcing security companies to stay sharp, said Butler. “Commercial companies only become as good as they have to be. Before FU came along, (antivirus companies) were doing more or less simple scans — signature based scan, that look for the presence of a file on the disk. Now, with rootkits like FU and Hacker Defender, they’re starting to take notice and realize that they have to get better.”