Migrating from T1 to fiber WAN

Back in the old days, the only realistic way to connect multiple remote sites was by T1 or T3 delivered either point-to-point or via Frame Relay. These were either slow and expensive or fast and unbelievably expensive. Then came MPLS, which dispensed with the need for point-to-point circuits from site to site, but was still bound by high expense. You got what you paid for. These circuits were not only reliable, but if a T1 or T3 circuit dropped, you could generally count on the carrier to jump on the problem quickly and resolve it with some expediency.

As cable and DSL networks began expanding, the ISPs introduced the concept of a business-class circuit. With significantly higher bandwidth than a T1 for far less money, these circuits are quite attractive — but susceptible to the vagaries of their physical plant, which is to say that they’re not as reliable as the T1s and T3s of old. In many cases, that trade-off is acceptable since the cost savings can be measured in the tens of thousands of dollars per year.

However, alternative ways to connect remote sites may enter the picture depending on their physical locations. The ideal solution is to be fortunate enough to find that all of your sites are served by a single fiber carrier, such as Optimum Lightpath. In other cases, you may find that your last-mile carrier (such as Verizon) has fiber to your locations, and a carrier such as Cogent can tie up all those ends into a connection served by a single end-to-end network. This way, you get bidirectional speeds up to 1Gbps between sites served by the same carrier, all for a lower monthly cost than a few 1.5Mbps T1s. If possible, this offers the ability to treat remote sites as local, enabling all kinds of replication and application delivery options that simply aren’t available with lower-bandwidth circuits.

The new WAN: Weighing the options

For this case study (see “How to slash your WAN costs” for the background), we’re detailing a migration from an MPLS network fed with multiple bundled T1 circuits at each site to 100Mbps fiber circuits installed at two sites with a common carrier and a business-class cable circuit feeding a smaller office. These sites are spread across three states, hundreds of miles apart. The previous design was simple: four T1 circuits at HQ, three at the large remote site, and a single T1 at the small office. Each location was linked to a single MPLS provider to handle internal routing.

The design and build were relatively simple, and the network functioned well, with an average latency of around 30ms between sites. However, with all Internet traffic flowing across the MPLS network for egress through an Internet circuit at HQ, the 4.5Mbps and 1.5Mbps pipes to the remote offices were suffering. Adding more T1s or a fractional T3 to the mix was a possible solution — but a very expensive option for a minimal increase in bandwidth.

After some research, it became clear that a single fiber carrier could serve all three offices, and plans were hatched to deliver 100Mbps service to the large sites and 10Mbps service to the small office. Each circuit would be configured as an Internet circuit and assigned public IP subnets. The first two parts of the build were delayed significantly by Verizon, the last-mile provider, but were eventually built and tested. The small site proved to be the biggest problem.

Due to lack of planning and poor design stewardship by the building owner, only a single conduit fed from the fiber interconnect at the street to the large office building housing the small site. Although four conduits were specified in the building plans, only a single conduit could be found in the basement wiring room; the others may or may not have been present on the other side of the concrete foundation. Naturally, the existing conduit was full, with no more room to pull the fiber through.

The costs of exploratory digging, permits, and other nonsense that would be required to find the other conduits was far too high, so the plans for adding the small office to the fiber network were scrapped. Instead, a business-class asynchronous cable circuit was ordered, and fingers were crossed. Fiber is a dedicated medium, whereas cable is shared. Fiber is also physically more robust than cable, and it generally elicits faster response times when trouble occurs. But cable would have to do.

Once the 100Mbps fiber circuits were built out at the two larger locations, Cisco ASA5510s were procured, configured, and tested. Each site would have an AES-256 VPN tunnel to the other sites, with VoIP and video traffic prioritized to ensure that phone calls and videoconferences would trump all other traffic. Further QoS was implemented to ensure that internal WAN traffic would supersede Internet traffic.

The new WAN: Upsides, downsides

The initial testing showed the best possible scenario: The latency between the two main sites was right around 10ms, roughly a third of the latency on the dedicated MPLS network — not bad for a 200-mile round trip. The smaller site had somewhat higher latency due to the fact that it was served via another carrier, but was still around 35ms.

The new network was stress-tested and cut over during a weekend maintenance window. The speed bump from 4.5Mbps to 100Mbps for internal traffic was lost on the users, but immediately noticeable to IT, which quickly put the abundance of bandwidth into use for SAN replication and backup consolidation. The voice and video traffic not only ran smoothly, but in fact the videoconferencing resolution could be upped to 1080p without a hiccup. And the monthly WAN costs dropped by one-third, saving nearly $40,000 per year — talk about a win for IT.

However, the new network involved trade-offs. For starters, Internet access was no longer centralized, as each site now had full Internet access via the same pipe carrying the WAN VPN traffic. This led to the demise of Websense and other Internet content control measures. With Websense gone, OpenDNS was pulled into action to mitigate nonbusiness Internet usage at each site. Although OpenDNS is not as deeply configurable as Websense, the cost reduction for the functionality offered made it worthwhile.

The other significant loss was redundancy. Whereas the previous MPLS network consisted of multiple bundled T1 circuits with disparate physical egress paths out of each building, the new network was built on a single fiber link. Previously, if one or two T1s in the bundle dropped due to upstream data problems, one or more T1s would likely remain operational, allowing the network to continue functioning, even if significantly degraded.

With only the single fiber connection at each site, upstream problems mean the loss of all data services: Internet, internal phone, and WAN. Apart from a backup circuit that would cost as much as the primary, there’s no suitable way around this problem other than to procure some different form of service delivery — such as cable — and go through the headaches of configuring backup VPN tunnels through different providers.

Over the many months since the new network was built and implemented, there have been a number of service disruptions caused by upstream provider problems (read: Verizon screwing things up) and more than a few scheduled maintenance windows that drop the circuits for a time, usually late at night. Otherwise, the stability and speed of the network have been exemplary. The business has no concerns over passing internal traffic via Internet circuits, since the traffic is not only encrypted with AES-256 but never leaves the carrier’s internal network. Overall, the network may not be as robust as the old T1’s, but it’s snappier and much faster — and the money saved is music to everyone’s ears.

This story, “Migrating from T1 to fiber WAN,” was originally published at InfoWorld.com. Read Paul Venezia’s The Deep End blog and follow the latest developments in networking at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.