Pushing the limits of virtual networking

Recently, I was faced with an interesting virtual networking problem: how to allow a large virtualization environment to be failed over to a recovery data center and thoroughly tested, without impacting the production network. As is often the case, my quest for an elegant solution took me down the rabbit hole and led to consideration of several new virtualization networking technologies. In the end, I didn’t achieve exactly what I set out to, but I did catch a glimpse of what the future might hold.

The challenge

In this case, a large VMware vSphere-based virtualization infrastructure had been configured to fail over to a duplicate secondary data center elsewhere on the same campus, the entirety automated through the use of VMware’s Site Recovery Manager (SRM).

Since both data centers had visibility to the same mutually redundant core route switches and associated layer two (L2) networks, an actual failover would be relatively seamless from a networking perspective, involving none of the dynamic routing or, worse, re-addressing that might be required in a typical geographic site failover. If the production data center were to become unavailable for whatever reason, SRM could simply bring the virtual machines back up at the secondary data center in virtual networks configured identically to those in the primary data center, and life would continue without any real changes to the network.

However, providing the means to thoroughly test this data center failover capability was a different matter. For a variety of reasons, the virtual machines running in the environment were divided into a fairly large number of L2 network segments, each consisting of a separate VLAN on the network and represented by a separate virtual port group on the vSphere hosts (in this case switched by Cisco’s Nexus 1000V virtual switch).

Fortunately, SRM treats a real site recovery and a test recovery differently, allowing the administrator to perform a test without interrupting ongoing SAN replication (by creating volume clones rather than promoting replica volumes) and by allowing the administrator to specify separate virtual networks to be used. With this feature, I could create a “shadow” set of virtual networks in which the virtual machines could be started during a test, while the “real” virtual machines continued to operate and serve clients in the original networks.

Unfortunately, providing that duplicated set of virtual networks while allowing all recovered virtual machines to speak to each other and without requiring substantial changes to the network turned out to be a tall order without a perfect solution.

Solution No. 1: VLANs and VRF

The first thing I considered was building a shadow set of VLANs that would duplicate the existing, production VLANs. Configuring virtual portgroups based on these VLANs would allow virtual machines running on different members of the secondary virtualization cluster to talk to each other regardless of whether they booted up on the same host. The downside is that a separate set of VLANs and their associated configuration within the virtualization environment would need to be maintained — an extra administrative headache, but not an enormous one.

In addition, simply creating a separate set of L2 VLANs wouldn’t allow virtual machines in one VLAN to speak to those in other VLANs. For that, I would need layer three (L3) routing. Fortunately, the existing core switches supported the definition of VRFs (Virtual Routing and Forwarding). By configuring a new VRF on the core switches, I could partition a subset of the core L3 VLAN interfaces into a “partitioned” router with its own route table, thus allowing identically addressed interfaces to exist for the production and test environments without allowing them to see or conflict with each other.

While that solution would undoubtedly work, it would add a substantial amount of complexity to the network, including the definition of new VLANs in the secondary data center and the core, as well as a substantial amount of new routing configuration within the core. There had to be a better way.

Solution No. 2: VXLANs and a virtual router

Not all that long ago, Cisco, VMware, and several other partners spearheaded the creation of a new network overlay protocol standard called VXLAN (Virtual eXtensible Local Area Network). VXLAN seeks to fix many of the scalability and provisioning problems that large cloud infrastructures experience — namely, dealing with the need to constantly provision new VLANs on both physical and virtual networking gear to support isolated customer networks and addressing the exhaustion of available VLAN IDs (stored in a 12-bit integer supporting a max of 4,096 VLANs).

VXLAN provides a solution to those challenges by tunneling L2 traffic in between virtualization hosts within L3 packets — in a sense, creating a set of completely isolated network segments that only are visible within the virtualization environment and that require little to no configuration outside of it. Additionally, VXLAN also utilizes a much larger integer for storing the so-called Virtual Network ID (or VNI), allowing the creation of millions of VXLAN segments within a shared infrastructure. Though I wasn’t concerned with creating millions of networks, utilizing VXLAN in the place of defining new VLANs would nearly eliminate any configuration changes to the physical network devices and would be easy to extend down the line — exactly what I wanted.

However, the VXLAN standard suffers some drawbacks due to its relative newness. At the moment, the only networking stacks that can actively participate in building a VXLAN-based network are the Cisco Nexus 1000V virtual switch and Open vSwitch; no physical network devices (that I’m aware of) are capable of acting as a VXLAN Virtual Tunnel Endpoint (VTEP). If I wanted to introduce L3 routing to allow cross-VXLAN traffic, I’d have to do so within a virtual machine.

To make matters worse, as with physical network gear, there aren’t any virtual appliance-based routers capable of natively participating in VXLAN, though people are asking for it. Today, the only way to allow a virtual router to have connectivity to the various VXLANs I might define would be to attach a virtual NIC to the router appliance for each VXLAN networks I wanted to allow routing across. Not only is this cumbersome, but it’s made all the more difficult by vSphere’s a hard limit of 10 virtual NICs per virtual machine — I’d need three or four virtual router appliances, each festooned with vNICs, to provide routing to all of the necessary networks. Despite the benefit of not having to touch the physical networking gear and the “cool” factor of using VXLAN to solve a real-life problem, this also introduced much more complexity than I wanted.

Solution No. 3: VLANs and a virtual router

In the end, I simply broke down and created a duplicate set of VLANs on the physical networking gear and configured the secondary data center’s virtual networking gear to talk to them. However, instead of creating a new VRF on the core switches to route the VLANs, I used a virtual appliance-based router. The difference here is that the router would not have to act as a VXLAN endpoint; it’d just have to understand tried-and-true 802.1q VLAN tagging — which enjoys nearly ubiquitous support.

I configured the virtual router with a single virtual NIC, which is in turn configured to be a VLAN trunk; this is called Virtual Guest Tagging or VGT — the missing piece in the VXLAN puzzle. That NIC can then have multiple virtual interfaces configured on it within the virtual router (one for each VLAN), which allow it to act as the default gateway for a relatively arbitrary number of different networks at the same time. Though the performance and redundancy of this kind of configuration might not be sufficient for some production environments, it’s perfectly suited for this kind of test environment.

This compromise wasn’t exactly what I wanted, but it was the best I could come up with and took me through the interesting process of learning some of the mechanics and current limitations of the available VXLAN implementations.

Future potential

Though I didn’t get to leverage VXLAN as I had hoped, I can certainly see its potential. At least in its Nexus 1000V implementation, Cisco has worked to expose an API within the vSwitch, which allows third-party services to automatically provision VXLAN network segments. At the moment, this capability is only really being used in conjunction with VMware’s vShield Edge and vCloud Director for automating the deployment of isolated public/private cloud tenants; however, there’s no reason we should expect it to stay that way.

The day is not far off when one might be able to fire up a full-featured virtual appliance-based router and integrate it with the Nexus 1000V or any other “soft” switch. Thus, you’ll get the dynamic provisioning of L2 network segments together with the L3 routing and higher-order network services, all through the same pane of glass. I can’t wait for that day.

This article, “Pushing the limits of virtual networking,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.