Whodunit: A Hyper-V failure may reveal fabled ‘escape attack’

This week, I was called into an office to look at an odd situation with a Hyper-V server. The server has been up and running for years, with Windows Server 2008 as the Hyper-V parent and three VM servers running Active Directory, SharePoint, and various other server applications. For some reason, none of the VMs was running. Upon investigation, it was easy to see why: Hyper-V was no longer enabled as a server role. Somehow it had uninstalled that role and wouldn’t reinstall itself.

That was a new issue for me. I’ve seen Hyper-V hiccup before, but never uninstall itself. I put that concern aside for the moment and pulled the three VMs over to another system running Hyper-V. They all mounted without issue. Phew! Good news.

[ Don’t look now, but your antivirus may be killing your virtualization infrastructure. InfoWorld’s Matt Prigge shows you how to detect the warning signs. | Stay atop key Microsoft technologies in our Technology: Microsoft newsletter. ]

I thought, “We fix the Hyper-V server, we restore these VMs, and we’re good to go.” Not so fast — the SharePoint server wasn’t serving sites, as it had faithfully done for five years. Upon further investigation, I found that the SQL databases were still in place — but not SharePoint. It was as if it had been uninstalled from the system. Only three people have access to these servers, and none of them uninstalled SharePoint or disabled Hyper-V.

The mystery remains as I write this, and we’re poring through event logs to see if we can get to the bottom of it.

An instance of the fabled escape attack? Someone suggested the possibility of a hack going through SharePoint (which is Internet-facing) and into the Hyper-V parent. That particular hack is known as an escape attack. A few years ago, it was just a theoretical possibility, but recent news reports indicate that these vulnerabilities do in fact exist and can be exploited. My colleague David Marshall recently relayed a security flaw in which someone can cause a system exception in virtualized code and escape from the guest OS into the host environment with elevated privileges.

Although many people are skeptical about the existence of attacks such as the escape attack, Gartner Fellow Neil MacDonald says, “It’s just a matter of time before a widespread publicly disclosed enterprise breach is tied back to a hypervisor vulnerability.” He recommends you take a look at the National Institute of Standards and Technology’s Guide to Security for Full Virtualization Technologies.

I’m left wondering if the Hyper-V server in my client’s case was hit by an attack. It feels purposeful, not like a simple virus. If it was an attack, it’s absolutely the result of a cavalier attitude on the administrator’s part. Every rule in the book was ignored here. You always think, “What are the odds?” until you become part of the statistics. Thankfully, there are valid backups of the VMs, so only a little data will be lost.

Key steps to reducing virtualization security risks This hack should make everyone with virtualized systems question whether they are doing everything they can to ensure they are secure. To help you do that, I’ve put together a preventative checklist, including items that can apply specifically to Hyper-V:

• Use Server Core for your Hyper-V parent. Server Core is a minimalist OS that reduces the attack surface and potential exploits. Fewer patches and updates are needed for Server Core, and you manage the server remotely using your Hyper-V management tools rather than directly on the server.
• Do not run applications on a Hyper-V parent. Applications are for your child VMs, not the parent. I cannot tell you how often I see this rule ignored by admins who want to use that parent as a normal server. It already has a job, and it should do only that job.
• Do not give admins who work with child VMs the same permissions as on the parent. This is common sense, but what if you’re the same admin for both? In that case, you should have two separate accounts: one account for the child VMs and one for the parent. In security circles you hear the principle of least privilege preached all the time. This principle applies no matter what the platform, even if the execution methods differ. For example, EMC VMware’s ESX has roles that can be used to give different levels of security.
• Update all VMs regularly. VMs are no different than physical servers in that they need to be updated and patched. In fact, you should update and patch your VMs before you even turn them on. For Hyper-V, Microsoft has a tool called the Virtual Machine Servicing Tool (VMST) 3.0 that can do this, which helps avoid introducing vulnerabilities into your environment. You can also use tools like System Center Configuration Management to assist with patch management. If you use ESX, it has its own tools, as well as third-party ones.
• Use a separate NIC for your VMs and parent. I found this recommendation on Microsoft’s TechNet community. It says, “by default, NIC0 is for the parent partition. Use this for management of the Hyper-V server and don’t expose it to untrusted network traffic. Don’t allow any VM to use this NIC. Use one or more different dedicated NICs for VM networking.”

There are plenty of other security practices to keep in mind, some of which are specific to the vendor you choose for virtualization. For example, ESX provides a built-in firewall whose configuration you should double-check. Regardless of the vendor whose products you use, most of your VMs will run some form of the Windows OS, so you need to make sure your security-hardening process takes into consideration every angle, including the OSes and applications running on those VMs.

This story, “Whodunit: A Hyper-V failure may reveal fabled ‘escape attack’,” was originally published at InfoWorld.com. Read more of J. Peter Bruzzese’s Enterprise Windows blog and follow the latest developments in Windows at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.