Clustered storage solutions offer safety and throughput in numbers, but still on the fringe Commodity hardware and virtualization is a powerful cocktail, not only in computing but also in storage. A number of vendors including Caringo, Exanet, Intransa, Isilon, and Verari offer clustered storage systems using commodity Intel boxes, each with a few drives, and storage virtualization to stripe data across multiple nodes. The nodes in a storage cluster are typically connected through iSCSI over Gigabit Ethernet, often with a 10GbE (Gigabit Ethernet) backbone, although interconnects including InfiniBand, Mellanox ConnectX, and AoE (ATA over Ethernet) are sometimes used. Claims for storage clusters include better redundancy than typical storage systems, and fast rebuild times if drives or even nodes are lost. As with server clustering, low cost and high performance are the draws. But if these systems are better than the typical Fibre Channel storage silo, why aren’t they spreading faster? One reason is that many administrators tend to be conservative, going with technologies they are familiar with, and solutions that they understand and trust. Another is that getting performance as good as you’ll get with the latest Fibre Channel systems will require 10GbE — at least to the servers, if not to all the nodes — or even InfiniBand or another high-bandwidth, low-latency interconnect. These technologies will require an investment in learning to run them, if not a bigger price tag than Fibre Channel infrastructure. A third reason is that these systems are often tough to pilot test. For optimum performance, you’ll need lots of nodes, plus a switch that can handle 10GbE connections to the servers. When I looked at reviewing a couple of these systems, the recommendation to install a 24-node system with four servers connected via 10GbE stopped me in my tracks. It was more than my lab would support. And yet, the promise of these systems can be realized. I tested Intransa’s solution about a year ago, and found it offered very high performance, approaching 8Gbps of throughput with dual controllers. In general, as the Intransa offering demonstrated, iSCSI can provide very high throughput over 10GbE, although IOps won’t be as high as with 8Gbps Fibre Channel or 10Gbps InfiniBand, due to the higher latency of iSCSI. This means that database applications that need lots of random reads or writes may still perform better over Fibre Channel. But applications where IOps performance is not so important — such as disk-to-disk backup, medical imaging, video processing, and video on demand — have proven to perform well on clustered storage platforms, and at lower costs than Fibre Channel hardware. Where low cost and high resilience are more important than maximum performance, storage clusters will continue to gain ground.