How to choose the right iSCSI switch

I’ve written a few times about the many unexpected complexities involved in getting an IP storage network to work optimally. Whether you’re using NFS or iSCSI, it’s not surprising that the switches you choose — and how you configure them — can make an enormous difference in both the performance and reliability you can expect to achieve.

Though 10Gbps Ethernet is becoming far less expensive than it once was, you’ll still pay a substantial premium for it in comparison with 1Gbps Ethernet. This cost rears its head not only in the switches, but also in items as simple as cabling; even twin-ax direct-attached copper cables run around $150 each, a fair sight more expensive than a $5 CAT5e cable. For enterprises seeking the high bandwidths and simplicity that 10GbE can offer, it can be an excellent and entirely worthwhile investment, but most of us simply don’t need it yet.

[ Also on InfoWorld: Matt Prigge helps you answer the eternal question: NFS or iSCSI? | Sign up for InfoWorld’s Data Explosion newsletter for news and updates on how to deal with growing volumes of data in the enterprise. ]

That’s why the vast majority of IP-based storage implementations use garden-variety 1Gbps Ethernet technology and rely upon multipath I/O or link aggregation to achieve performance in excess of 1Gbps. Though it’s absolutely true you can use general-purpose networking gear for IP storage, not all 1Gbps Ethernet switches are well-suited to handling the loads associated with IP storage. Here are a few guidelines you can use to grade your options if you find yourself on the hunt.

Nonblocking backplane

The very first thing you want to look for is whether the switch in question has a nonblocking backplane. That simply means the switch is capable of operating at full bandwidth on all of its ports simultaneously. Some less expensive switches (and even very expensive ones not designed for data center workloads) utilize port-to-backplane oversubscription. In these switches, several gigabit ports may share a single gigabit pathway to the switch’s backplane. This is especially common in chassis-based modular switches, which are designed for high-density workstation aggregation and should be avoided wherever possible.

Stacking and link aggregation

In all but the very smallest IP SAN deployments, you’ll generally want to deploy at least two storage switches to offer path and switch redundancy to your storage infrastructure. Unlike in Fibre Channel storage infrastructures where the two storage fabrics remain completely isolated from one another, it’s important in IP storage infrastructures to provide a substantial pathway between your storage switches.

This is essential due to the fact that a host port attached to your first switch may well be accessing a storage port on the second. Moreover, many scale-out iSCSI SANs generate large amounts of inter-SAN member traffic — another traffic flow likely to use that interswitch pathway.

Thus, it’s important to offer a significant amount of bandwidth between the switches, generally equal to the amount of bandwidth available to the active interfaces on your IP SAN; in larger SANs, this can be quite a lot. In switches that offer a built-in (and generally proprietary) stacking mechanism, using it can be a good way to offer a high-bandwidth switch interconnect. Most stacking links available today give you at least 10Gbps of bandwidth; many offer much more.

For switches that do not have any built-in stacking, you’ll have to use general-purpose switch ports to do the job. If you’re working with switches that ship with a couple of 10Gbps Ethernet uplink ports, those can be used to link the pair. If not, you’ll have to create a link aggregation group (LAG, typically using 802.3ad LACP), which might consume a large number of 1Gbps ports. This is one of the reasons why bargain-basement unmanaged switches aren’t wise choices for IP storage, as they typically do not have the capability to create these kinds of LAGs.

Flow control and packet buffers

Another crucial switch tech is 802.3x flow control, a mechanism that allows an Ethernet interface to issue a packet called a “pause” frame that signals to the system on the other end it should stop sending traffic for a specified period of time (usually very, very small). This is extremely valuable in IP storage applications as a device, be it the client, SAN, or switch, can signal to the sending device that it is overwhelmed — a process very similar to asking a coworker who has had too much coffee to speak just a bit more slowly.

Without flow control, when a link or device becomes overwhelmed with traffic, some traffic will inevitably be dropped. When that occurs, the sending and receiving hosts will detect it and retransmit the missing packets. That can take a relatively large amount of time and result in noticeably decreased performance.

Thus, a good IP storage switch will both honor and retransmit flow control pause frames issued by servers and storage devices attached to it, then issue them itself when it becomes overwhelmed. Now, you might think, if I have a switch with a nonblocking backplane, how could I ever have a situation where the switch itself is overwhelmed?

Imagine a situation where a number of hosts are writing a large amount of data to a SAN and that traffic is all trying to get down a single gigabit Ethernet port. In that scenario, there will undoubtedly be times (even if they are extremely short) when a port’s available bandwidth will be less than the traffic that needs to be sent. In that case, the switch will typically use its output packet buffer to store the data waiting to be sent until there is adequate bandwidth available. However, if the packet buffer is filled, traffic will be unceremoniously dropped. By issuing a pause frame to the sending hosts, the switch can give itself time to empty its buffers.

However, the larger those buffers are, the better overall performance will be. Gigabit switches designed for desktop workloads will typically have relatively small buffers (2MB to 4MB/switch, assigned to ports on an as-needed basis), while those suited for data center workloads typically have much more available buffer and give you the flexibility to determine how it is assigned and used.

Jumbo framing

Another important storage-related network feature is jumbo framing. Most typical IP packets have a maximum size of 1,500 bytes. For general-purpose networking where traffic may go over a low-bandwidth WAN, this is generally a good size. However, for the high-bandwidth network flows associated with IP storage, this packet size limitation results in many more full-size packets being sent than you’d see generated by normal application workloads. By increasing the Maximum Transmission Unit (MTU or max packet size) to 9,000 bytes, the number of packet headers that need to be created, sent, and processed is dramatically decreased, and overall performance will go up.

Disabling storm control

Many intelligent switches have a feature called unicast storm control. This feature is designed to detect a particularly chatty network host and to disconnect it before it can harm the network. In most situations, if you have a network full of workstations and one of them maxes out its link sending traffic, something has probably gone awry. However, in storage environments, this is a normal and expected traffic pattern. Being able to disable unicast storm control (if it’s enabled by default) is an important capability to have.

Putting it all together

As it says on the tin, you can use just about any kind of general-purpose networking gear with either NFS or iSCSI. However, if you don’t choose the right gear, you risk artificially limiting the amount of performance you’ll be able to wring out of the storage infrastructure. If you’re planning an IP storage network that’s going to see a lot of traffic and are concerned about delivering the best performance possible, do the research and don’t skimp on your switches.

This article, “How to choose the right iSCSI switch,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.