VMware pushes new storage paradigm for virtualized world

As the saying goes, you can’t make an omelet without cracking a few eggs.

That axiom applies to every disruptive technology ever to hit the market. But it’s been a long time since the primary storage market saw serious disruption.

Sure, the storage market has seen plenty of innovation, such as object-based cloud storage or cheaper, better SSDs. Great advances such as automated tiering, deduplication, eager thin provisioning, and storage virtualization have delivered a ton of value to storage professionals. But all of the above has largely been improvements on existing tech. The same goes for the storage protocols that have emerged over the past decade; iSCSI, FCoE, and the ATA-over-Ethernet standard are all block-level SCSI protocols that use different transports.

Allow me to predict, however, that the relatively staid world of storage technology is about to be turned on its head. If the vision laid out in the VMworld 2011 VSP3205 tech preview session by VMware and the big-five storage companies is any indication, things are going to get pretty interesting over the next few years.

How interesting? Let’s just say I wouldn’t want to be an egg right now. To understand the extent of the disruption, though, you need to know the context.

A bit of history

Virtualization has been a boon for storage management. The layers of abstraction inherent in virtualized infrastructures has made incredibly useful capabilities — such as those offered by VMware’s Storage vMotion — a reality.

But the world of possibilities that virtualization has opened for us has also placed a strain on other segments of the data center infrastructure. As virtualization tries to do new tricks, the storage and networking worlds are forced to play catch-up — or find themselves steamrolled by proprietary work-arounds.

So far, VMware has used its market dominance to meet these challenges, in much the same way Cisco has used its clout to try and remake the networking market in its own image. Anyone who’s worked with Cisco hardware for a long time has been exposed to proprietary functionality that has since been supplanted (or joined) by other standards-based technology; CDP, EIGRP, and pre-802.11af PoE are just a few examples. In some cases, this has caused compatibility problems and vendor lock-in.

The same is coming true for VMware. Faced with high-density cloud networking environments that make layer-two network isolation increasingly difficult or even impossible, VMware hatched its own MAC-in-MAC tunneling protocol in the form of vCD-NI. This protocol first showed up in the now defunct VMware Lab Manager, then made its way into VMware vCloud Director. Based on my own experience, it’s universally reviled by most networking gurus as being very difficult to manage.

In enabling layer-two network isolation, vCD-NI made the impossible possible — at the cost of being a proprietary work-around. Fortunately, VMware has partnered with a number of prominent networking companies (Cisco included) to introduce the new VXLAN standard to the IETF, which should eventually replace vCD-NI’s MAC-in-MAC with a standards-based MAC-in-L3 implementation.

Similarly, VMware’s vStorage APIs for Array Integration (VAAI) are VMware’s answer to a number of different virtualization-related storage challenges. VAAI includes the use of SCSI primitives that allow VMware to offload such tasks as copying entire VMs from one storage LUN to another (Full/Extended Copy), enabling array-based thin provisioning of virtual disks (Block Zeroing), and high-performance file locking (Atomic Test and Set) directly onto the storage array.

These extensions are being rapidly adopted by VMware’s storage partners (that is, pretty much everyone who’s anyone), but not without objection. Much of the grumbling derived from the fact that the SCSI primitives used by VAAI when it was first released in vSphere 4.1 were not yet ratified by the INCITS T10 working group (the standards body that governs SCSI), effectively forcing storage vendors to implement production functionality that could change once the standards were ratified. You can’t really blame VMware for wanting to get to market quickly with sought-after features, regardless of whether standards bodies feel like following along. That’s what market leaders do.

VMware’s bold proposal

As virtualization environments become more complex and customers demand more functionality, VMware is yet again under pressure to think outside the box. The VSP3205 presentation at this year’s VMworld offered a glimpse into what VMware and its primary storage partners have envisioned for the future of virtualization-integrated storage.

The presentation covered a lot of ground (check out this recap by EMC’s Scott Lowe). In a nutshell, it centered on the APIs and protocol extensions required to give the storage infrastructure visibility into what virtualized resources are stored on top of it. As it stands now, regardless of what kind of storage array you run (block-level SCSI or NFS), storage arrays know very little, if anything, about what data they’re working with. Especially on block-level devices, the storage device doesn’t really know the difference between an NTFS volume attached to a Windows server and a VMFS volume that might be hosting virtual disks for 50 Windows servers.

This lack of visibility presents a number of serious management and functionality challenges that will need to be overcome for virtualization to continue to evolve — especially as it relates to the cloud, but even beyond that as nondisruptive workload migration to mobile devices starts to become a reality. Basically, VMware thinks that storage arrays should be smarter, with the ability to see each virtual disk on the storage array as a discreet object. This opens the possibility of per-VM (or per-disk) storage QoS polices, more efficient deduplication, and much easier implementation of stretched clusters (see my original wrap-up post on VMworld 2011 for more on that).

Though the design presented by VMware and its storage partners in this session isn’t likely to survive to implementation exactly as they’ve described it, VMware is essentially proposing massive changes to the way that storage is provisioned and managed at a very basic level. These changes may initially be implemented through extensions and APIs stapled on top of existing storage protocols and devices. But it’s easy to imagine this concept as an entirely new storage protocol — one that will bear little resemblance to existing storage in terms of how it’s managed.

That’s a big deal.

To be sure, this concept is still very much in its infancy. Even if the proposed design survives the test of time, I imagine we’re at least two or three years away from production products that make use of it. It’ll be interesting to see how the storage market reacts to this concept as it’s refined. At the moment, VMware appears to have the interest (if not full support) of the five largest storage players, but that itself has already raised suspicions that VMware — primarily controlled by EMC — will gain too much power and stifle innovation among smaller storage players.

Putting it all together

The advances VMware and its storage partners are proposing are sorely needed. Virtualized environments require better integration with the storage they run on if we’re going to see stretched clusters and fully hybridized public/private clouds. But how we get there remains an open question. VMware can continue to develop standards on its own and try to force then down everyone’s throat; historically, that’s the way the sausage tends to get made in the tech industry. But it doesn’t always work. If VMware isn’t careful, it risks pushing customers into the arms of other less proprietary competitors.

This article, “VMware pushes new storage paradigm for virtualized world,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.