iSCSI storage networking: What you need to know

Over the past two weeks, I’ve written about some of the commonly overlooked aspects of building a bulletproof IP storage network and how to best use that network with NFS. This week, I’ll show you the ins and outs of configuring iSCSI for performance and redundancy, as well as how it compares with NFS.

The first thing to understand is that although NFS and iSCSI are both IP storage protocols supported by many server operating systems and hypervisors, that’s about as far as the similarity between the two extends. NFS is a high-performance file-sharing protocol in the same vein as SMB/CIFS, while iSCSI is a block-level storage protocol more akin to Fibre Channel in that it encapsulates raw SCSI commands. This distinction is important because it has significant bearing on how you get redundancy and performance scalability.

iSCSI vs. NFS on the network: How they differ

When comparing the networking requirements of NFS and iSCSI, the largest practical difference is that the iSCSI protocol has built-in redundancy and link aggregation capabilities, called multipath input/output (MPIO). NFS lacks MPIO. You can use MPIO to offer both link redundancy and additional throughput — entirely without the use of the problematic NIC/link teaming that’s required to do the same for NFS.

In a typical NFS configuration, servers are each configured with a single-storage IP address bound to a NIC team spread across two stacked switches. On the storage side, the same configuration is duplicated, except that the storage is generally configured with a second IP address alias to allow better load balancing over the team members because the NIC teaming algorithms use source and destination IP address to load-balance traffic. In this scenario, the failure of an individual link or entire switch stack member is handled by the NIC teaming software — the NFS client and server software isn’t involved.

By contrast, iSCSI configurations use separate, unteamed NICs each with its own IP address on both the server and storage side. When properly configured, the iSCSI initiator software on the server and target software on the storage array build multiple storage connections across those links, so each of the two interfaces on the server has a connection to both of the interfaces on the storage array. If an individual link or one of the redundant storage switches failed, the iSCSI initiator software would recognize that two of its four storage paths had become unavailable and would funnel all traffic over the other two paths.

The exact means by which iSCSI initiators (servers) and targets (storage arrays) work together depends somewhat on what server-side software is used and what array it is connected to. That’s why the Microsoft iSCSI initiator works slightly differently than the software iSCSI initiator built into EMC VMware’s vSphere. Likewise, the iSCSI target software in an EMC VNXe SAN differs slightly from that in a Dell EqualLogic SAN in how it load-balances connections and offers targets and LUNs.

In all cases, it’s vitally important to read your storage vendor’s documentation and check out support forums for tips because different OS/array combinations can have significantly different best practices. If you really want to get the most out of your array and network, it’s very rarely ever as simple as punching in some IP addresses and calling it a day.

iSCSI setup in practice: An example

Let’s say you’ve been charged with connecting a VMware vSphere host to a Dell EqualLogic PS4100 over the physical network that I outlined in an earlier post. This entry-level array is fairly simple in that it has only two storage interfaces on each of its two controllers.

Attaching to the switch is simply an issue of plugging the first interface from the first controller and second interface from the second controller into your first network switch, then plugging the remaining two ports into the second switch. Because this array’s controllers are used in an active/passive configuration, staggering the NICs across the switches ensures that the failure of a switch won’t isolate all the interfaces on the active controller at the same time, which could prompt a controller failover.

EqualLogic arrays are configured with a group IP address (the portal address), as well as individual IP addresses assigned to each active physical interface. At the end of the day, you end up with three IP addresses on the storage array, but the group address is the only one you’ll type in anywhere. The other two are used behind the scenes.

After the array is up and running, it’s time to configure the vSphere host. On your host, you should have at least two NICs dedicated solely to iSCSI traffic. You add those NICs to a single vSwitch much in the same way you configure other vSphere networking teams. After that, you need to add three VMKernel IP interfaces to that vSwitch. The first interface is configured to give the vSphere management stack visibility to the storage array so that it can ping it and determine if it’s alive (called a heartbeat VMK), while the other two actually move storage traffic.

Before you can bind (assign) the two storage VMKernel interfaces to the iSCSI initiator, you need to reconfigure their teaming properties so that one VMKernel interface is assigned to only use the first physical NIC attached to the vSwitch and the other interface only uses the second physical NIC.

Unlike in a normal team where you want both interfaces to be available if one of the physical NICs had a link failure, in this scenario you specifically don’t want that — you want to let the iSCSI initiator handle the link failure. This is also why the first heartbeat VMKernel interface is needed: The first defined VMKernel interface on that vSwitch always needs to be able to ping the storage regardless of which link might have failed.

After you have the vSwitch and VMKernel interfaces configured properly, you can enable the Software iSCSI initiator and bind the second two VMKernel interfaces on your storage vSwitch to it. Then it’s simply an issue of punching the array’s group IP address into the discovery IP addresses section of the iSCSI initiator configuration.

During the storage rescan that happens after you save the configuration, each VMKernel interface should connect to the array’s group address, at which time the array refers those interfaces to connect back to its individual interface IPs. At the end, you should have four active iSCSI connections that effectively build a fully redundant mesh across both host NICs and both array NICs.

However, you’re not quite done yet. In the default MPIO configuration (Most Recently Used), the vSphere host uses only one of the four connections to move storage traffic to the array. If there’s a link failure that disables that active path, it will dynamically switch to using a different path. But it will not try to take advantage of the additional bandwidth that might be available from using the other three paths.

The simplest way to put those paths to use is by reconfiguring the MPIO policy on each target device to use round-robin load balancing. (On an EqualLogic array, there is one target for each volume you create.) In this load-balancing scheme, the vSphere host cycles through the four paths, sending a little bit of I/O down each one. Aggregating the traffic from several hosts can make much better use of the array’s interfaces, but it isn’t particularly smart as this load-balancing scheme can send traffic down a path that might be busier than the others.

You need to do a little more work to use those other paths’ capacity without worry of oversaturating one channel periodically. Although not all vendors offer a device-specific module (DSM) for their arrays, Dell EqualLogic does. The DSM is effectively a software plug-in for vSphere that lets vSphere make better path load-balancing choices. Dell’s DSM, the Multipath Extension Module, adds a load-balancing mode that distributes storage traffic based on the queue depth of the individual storage paths, effectively implementing an intelligent load-balancing algorithm that ensures that all the paths/interfaces/links are equally loaded.

Real optimization requires real work

Whether it’s NFS or iSCSI, IP storage has a reputation for being simple and easy to configure. However, if you’re really trying to push the limits of what either protocol can do, you need to dig into the details and do a lot of careful planning to get the most out of your storage network. And you have to pay careful attention to how your physical network is assembled and how you’ve configured your servers and arrays.

No matter what you do, don’t go in blind. Do some careful best practice reading and really understand it. After all, no one wants to have to reconfigure the network and storage hardware after having gone live with it!

This article, “iSCSI storage networking: What you need to know,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.