The law of unintended storage consequences

The increasing complexity of modern storage arrays seems to know no bounds. While all of us are understandably excited when hot, new storage features arrive on the scene, it’s important to remember that all of them come at a price. Their complexity tends to make these systems harder to troubleshoot — and harder for storage vendors to test fully before they hit the market.

Each customer has its own way of using a new storage product to meet specific needs, so expanding the feature set multiplies the number of scenarios the vendor needs to test against. It’s almost inevitable that something will slip past the folks in QA. That’s precisely what happened after the release of vSphere 5.0’s new INCITS T10-based VAAI implementation, resulting in VMware hastily telling customers to manually disable the offending feature before it could cause problems.

There’s a lot we can learn from this particular case. Storage is no longer an amorphous bucket you pour your data into — it’s a living, breathing entity that can have as much of an impact on your environment as your users or the applications you run. Heavier integration with hypervisors and other software not only enhances the storage’s ability to solve problems, but also to create them.

Thin provisioning and VMware

VMware’s VAAI (vSphere APIs for Array Integration) is an umbrella term that that applies to a number of extensions to the T10 SCSI standard, which are designed to allow hosts to make more intelligent use of the storage they are attached to. Among these features, the SCSI UNMAP command has a lot of promise to dramatically increase the effectiveness of thin provisioning in virtualized environments.

Using this command, vSphere hosts are able to inform the storage array when a given set of blocks are being vacated, so that the array can mark those blocks as being free. Whether you’re simply deploying and then tearing down large amounts of virtual machines or taking advantage of more advanced features such as Storage vMotion and Storage DRS, it’s very common to see large swaths of vSphere datastores allocated and then de-allocated as virtual disks are created, grown, and then moved or deleted.

Since block-level storage arrays are entirely unaware of what data is actually being placed on top of them, array-based thin provisioning on virtualized storage arrays can only keep track of which blocks have yet to be written to. Once those blocks are written, they are forever “trapped” as being part of that volume and can’t be reused later. If the administrator uses Storage vMotion to move a virtual machine off one datastore and onto another, the storage array isn’t really aware that this has happened and can’t free the blocks that were being used to store that virtual machine’s disks.

That’s where the SCSI UNMAP command comes into the picture. By using this SCSI primitive, the host can inform the array when a given set of blocks are no longer actively being used and the storage array can move them into its free pool, allowing them to be allocated by other volumes. When it works, capacity efficiency improves dramatically.

The devil is in the details

Not long after the release of vSphere 5.0, it became clear that real-world use of the SCSI UNMAP command could have some unexpected consequences. When vSphere performs a Storage vMotion, it does a live copy of a virtual machine’s disk from one datastore to another and then transitions the operation of that virtual machine to the new disk location. Once this transition is complete, the last step is to delete the old disk, thereby freeing the space on the old datastore’s file system to be used for other purposes (not unlike moving a file from one drive to another on your PC).

When the disk array being used supports the T10 UNMAP extension, the vSphere host will also issue UNMAP commands for the range of disk blocks that the virtual machine’s disks had been using — freeing not only the space within the VMFS filesystem, but also the underlying disk blocks on the array. Though they seem similar, these two processes couldn’t be more different.

In the first case, vSphere simply needs to mark those blocks as being free in its file allocation table, which can be done almost instantaneously. In the second, depending upon the size of the virtual machine that has been moved, vSphere may have hundreds of millions of block UNMAP commands to send to the array — a process that’s often far from instantaneous and depends heavily upon how the storage vendor in question has implemented the UNMAP command within its product. If you’ve formatted a disk in Windows using the “quick format” option which simply rewrites the allocation tables versus a full format which touches every block on the disk, you’ve seen something similar play out.

In the wild, this left Storage vMotion waiting for these commands to be executed — long enough so that the Storage vMotion operation would time out and fail — potentially leaving the migration in an inconsistent state. Because the support for this and other VAAI commands is automatically detected and enabled, VMware’s only option was to issue a blanket declaration that removes support for the UNMAP command until the problem can be worked through with storage vendors and a permanent solution could be found.

Though we’ll all have to wait to see how VMware and its storage vendors solve this problem, it’s not as straightforward as you might hope. The crux of the problem is that vSphere has to wait for the array to actually free the blocks as it sends the UNMAP commands. In other words, they must be executed synchronously. Doing this asynchronously opens up a range of nasty situations in which VMware might subsequently attempt to write to a disk block which the array hasn’t yet finished clearing from a previous UNMAP request — either resulting in a failure of the write or, worse, data destruction or inconsistency.

A rough road to storage utopia

Though this problem is specific to vSphere 5.0 and has a relatively easy workaround, the moral of the story applies to a wide range of storage innovations. As storage arrays and hypervisors become smarter, the chances that you’re going to experience pain associated with living on the bleeding edge grows exponentially. Now, I’m not suggesting you hide under a rock clutching your tried-and-true, last-gen tech. But complications like these serve to remind us to expect the unexpected — and to have a rollback plan in case some fancy new technology gets too clever for its own good.

This article, “The law of unintended storage consequences,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.