How to pick the provider for a solid Ethernet WAN

In my post last week, I discussed how Ethernet-based WAN services are typically delivered to your premise, and I touched on some caveats. While the type and quality of your last-mile connection options are typically the most important points to determine early on, the middle and long-haul distance of the connection are by no means insignificant.

Let’s say you have two offices you want to connect with 100Mbps service. If you’re lucky enough to find that both sites are near fiber and a carrier can deliver the 100Mbps EPL (Ethernet Private Line) — essentially a point-to-point circuit — you’re looking for, you might suppose there’s not much else to worry about. After all, you’re probably paying $800 to $1,200 per month for an item that might have cost an order of magnitude more in the previous generation.

However, a wide range of features may or may not be available to you depending upon whether the carrier has designed its network to support them — and how many carriers are involved in delivering the service. Here are a few important features to look for if you’re considering leveraging Ethernet-based services for your WAN.

Are multiple providers involved? The first question to ask a carrier is whose service will actually deliver your circuit. Within the telecom world, it’s incredibly common for carriers to wholesale last-mile links to their backbone network. For a circuit that’s not entirely “on net” (delivered by a single provider’s network), a number of concerns may arise.

For example, if you live in Northern New England, your incumbent local exchange carrier (the company that owns the copper PSTN lines on the telephone poles) is most likely FairPoint. If you decide to buy a T1 to plug into an MPLS mesh from a national MPLS provider such as Windstream, keep in mind that Windstream won’t hang new wires to all the poles to your building.

Instead, it will wholesale a T1 loop from FairPoint, which then runs from your premise to the nearest FairPoint central office with Windstream network equipment connected to Windstream’s national backbone. This arrangement typically works well, though it can mean longer service calls for a fix when it’s not immediately clear which provider is causing the problem. In the end, you get one invoice from Windstream and you never have to call FairPoint. As the owner of the wholesaled circuit, Windstream has sole responsibility for the service.

It’s much the same in the Ethernet world, but with a few critical differences. The first is that a T1 is a T1 regardless of where you are in North America. Sure, there are a couple of encoding options (AMI, B8ZS, and so on) that vary from carrier to carrier, but in general, it’s cut and dried. The same is absolutely not true for Ethernet. Two carriers’ Ethernet products might support entirely different features (more on those later). You’ll be left working with the lowest common denominator of the two.

The other major difference is that Ethernet-based data products are typically not regulated by Public Utilities Commissions that oversee the mixed data/voice services (such as T1s) offered by incumbent local exchange carriers (ILECs). This means that providers are working together because they want to, not because they’re forced to, as in the case of ILECs and competitive local exchange carriers (CLECs). If you’ve found two providers that will cooperate to deliver a circuit, that’s great. However, after the circuit is delivered, those providers often don’t (or won’t) work with each other. If one of your two offices sits next to a pole with Provider A’s fiber on it, and the other sits next to a pole with Provider B’s fiber, that lack of cooperation can be incredibly frustrating.

Another potential issue with an Ethernet-based WAN on multiple carriers; There’s usually only a single connection point between the providers. This point, referred to as a network-to-network interface (NNI), can represent a single point of failure. It’s important to know how both providers plan to work around such a failure if a disaster were to strike the equipment or facility where the NNI is located.

What features will the carrier(s) support? The next big area where you’ll need specifics: the features you’ll be able to use. While it’s true that Ethernet is a standard, many other standards get implemented on top of Ethernet that you might want to be able to leverage. Not all (or even most) carriers will support every single one. For example, a “good” carrier would allow you to use your own 802.1q VLAN tags, tunnel L2 protocols like Spanning Tree, honor your DSCP or COS tags as they apply traffic shaping within their network, and pass jumbo frames (along with extra fudge factor for VLAN tagging and other encapsulation techniques).

Lack of support for those features doesn’t necessarily mean that a given circuit is unusable, but it can significantly affect the way you use them. For example, one of the big advantages of an Ethernet-based link is that you can use it exactly the same way as you would a link between two switches in your data center. You can configure it with 802.1q VLAN tagging to pass multiple, isolated virtual networks across the same wire, and you can implement circuit redundancy using the Spanning Tree Protocol (or one of its more advanced derivatives such as PVRST+ or MSTP).

You can also ensure that higher-priority traffic is preferred over lower-priority traffic when too many transmissions are trying to travel a link at one time (though there are a wide variety of ways to implement Quality of Service or QoS). This is important in premise-based network infrastructures, but it’s critical in long-distance WAN links. This is partly because WAN links are usually much more bandwidth restricted than those you’ll find in your data center; also, WAN circuits are typically rate limited or traffic shaped by the provider.

For example, you can order a 200Mbps Ethernet circuit from a carrier, even though there’s no such thing as a 200Mbps circuit. Instead, the carrier delivers to you a 1Gbps Ethernet connection that it shapes down to 200Mbps of throughput. If you don’t do your own traffic shaping on your routers and switches or if the carrier doesn’t honor your tagging, then the carrier will indiscriminately throw traffic away when you try to send more than 200Mbps of traffic through the link. If the carrier honors your DSCP or COS tags (different tags you can apply to packets on your switches to denote how important they are), then it can throw away less important traffic before more critical traffic.

Frame size is another important facet of an Ethernet-based WAN link. Typically, most standard Ethernet packets are 1,500 bytes in size. However, there are a number of different kinds of encapsulation, encryption, and tagging that can result in slightly larger packets being created — MPLS over Ethernet is a great example. It can be incredibly helpful to be able to send marginally larger packets: so-called baby jumbos measuring up to 1,600 bytes in size.

For links that handle storage replication or other bandwidth-intensive flows, it can be extremely helpful to use full-jumbo frames, which are typically 9,000 bytes in size. The same encapsulation buffers apply here, so look for the ability to pass 9,100-byte packets if you plan to use jumbo frames in combination with your own MPLS or encryption overlays.

What SLA will you have and how will it be enforced? The last thing to watch out for is the SLA from your carrier. In general, SLAs are of relatively limited utility — a provider’s service credit stipulated in the SLA will never compensate for any amount of downtime you experience. However, SLAs set a line in the sand over which both you and the provider can agree that there is a problem. In telco services and even IP-based Ethernet services (that is, provider-managed MPLS), SLAs are typically reflected in terms of lost packets and maximum latencies. Both you and your provider can test your circuit by pinging from your equipment to the provider’s equipment, thus easily determining end-to-end latency and packet loss.

In raw Ethernet-based services, there are no gateways for the carrier to ping. From the carrier’s perspective, it is tunneling your Ethernet traffic through its network and spitting it out the other end. The provider has no real visibility into that stream of traffic, nor does (or should) it typically have the ability to inject its own traffic into your circuit. This makes it difficult to create a means by which both you and your carrier can mutually verify the performance of the circuit. Different carriers have different solutions to this problem, but it’s important to determine what they are before you buy. You never want to find yourself in a situation where you ping from one end to the other and learn that you have incredibly high, 200ms latency, but your carrier refuses to use that as evidence of an issue.

If you choose your carrier wisely, using Ethernet to build a WAN can be awesome. You can pull off all sorts of tricks that are extremely difficult, expensive, or impossible to do with other WAN technologies. However, flexibility comes at a cost. If you plan to leverage Ethernet-based WAN tech to build a WAN, make sure you ask all the right questions ahead of time.

This article, “How to pick the provider for a solid Ethernet WAN,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.