Route Control
Taking Control of your Internet Routing.

Created: December 3, 2002

Note: For help designing your route control system or developing tools to help you audit/test the performance your network routing contact Birds-Eye.Net.

The use and popularity of multihoming is increasing in response to the need to provide reliable and low latency service for e-commerce. Just as those Internet businesses that have recently multihomed begin to rest easy, another ripple in the Internet fabric is forcing these companies to revisit their networks once again. This latest problem has to do with the fact that the Internet is a loose confederation of interconnected networks (see figure 1.0) that rely on self-serving information provided by Internet Service Providers (ISP). The ISPs have a monetary incentive to make their networks look better then the competition. If you choose their path, you pay them more money. Multihoming insures traffic has at least two paths (choices) that result in at least two, self-serving, opinions of which is the best path. However, these opinions leave business with the famous “trust but verify” dilemma. Which path should I pick? Should I trust the information provided by the ISPs who have a stake in the outcome? Can I trust that they (ISP or service provider) are working in my best interest? Or, should I attempt to verify their claim with my own measurements to insure I am getting the best possible path at this moment for my customers.

To compound matters, the validity of the information provided about routing is time dependent, but once distributed throughout the Internet, it is rarely updated by the ISPs due the complexity of re-computing the information. Each service provider has a time varying load imposed on the their network that causes a changing latency and packet loss experienced by traffic on their network throughout the day. Compound this with scheduled and unscheduled maintenance and the customer experience over time is unpredictable at best.

>

>Figure 1.0 Internet creates multiple connectivity paths for traffic to flow 

Being multihomed ensures that there are at least two or more paths that traffic can take as it flows between content source and consumer. The new ripple in the Internet fabric is that no two paths are equal over time. One path could be faster at 10:00 am and slower at 3:00 pm. One path could have substantial packet loss due to network maintenance at a critical trading time for your business. If your Internet business is time critical, for example stock trading, not capitalizing on the fastest possible paths between you and your customers, at all times, will ultimately sacrifice the speed of your service (and satisfaction of your customers). For a few Internet businesses, performance is negotiable, but for the majority, time is money. The value of multihoming can be enhanced beyond reliability if some way can be found of managing multiple routes for performance in real time.

Beyond the Original Design of the Internet

As shown in Figure 1.0 the Internet has a large number of paths to route traffic between any two points. If any one ISPs network goes down, the Internet can self-heal by re-routing traffic around these problem areas. The Internet communicates routing information about the "best" path from a source to a destination using BGP (Border Gateway Protocol). When a problem occurs, the owner of the network with problems or networks that are adjacent and observe the problem send out new routing instructions or BGP updates. These updates take time to propagate and may contain multiple views of the problem network. For instance, ISP A may be able to go through ISP B to get to ISP C, but ISP E can and not reach ISP C through ISP B. This difference of how to route traffic can cause routing instability and takes time to converge to a stable solution. During this convergence time, routing performance may be degraded as the network attempts to agree on the proper BGP representation of the routes available. BGP updates are few and far between and usually occur for only major network events. They typically do not substantially change over a 24-hour period. BGP is the de facto exterior routing protocol for the Internet.

Essentially BGP is used to learn routes to destinations out on the Internet and the router then uses this information when its forwarding traffic to those destinations. BGP communicates route information using an Autonomous System (AS) Number. AS Numbers are assigned by ARIN in the US, APNIC in Asia and RIPE in Europe. Routes are advertised by an ISP as a list of AS Numbers or hops that a packet can transit to get to the final destination. BGP selects the best route to these destinations using a decision process where by the router considers a number of criteria, and then selects the best path based on that criteria. Some of these criteria include:

• If the next hop is inaccessible, do not consider route.
• Prefer the route with the largest weight (a proprietary feature of one router vendor)
• If the local preference is the same, or if no route was originated by the local router, prefer the route with the shortest Autonomous System (AS) path.
• If all of the above is equal, pick the path with the lowest AS number (this biases the route to ISPs that were early providers of Internet service and thus have low AS Numbers. Genuity (formerly BBN) has AS1 and wins every time the decision process gets to this level).

Although it is beyond the scope of this article to review this decision process in detail, the purpose for introducing it is to convey that nearly all of the BGP decision criteria are static. It is mainly based on tangible metrics like hop count or the number of autonomous systems a particular path traverses – this information does not typically change over time. So while the Internet performance continually changes (backbone links become saturated, etc.), the path your traffic takes across the Internet (by in large) does not take into account these changes. As a result your traffic may not take the most optimum or efficient path (e.g. the path with the least latency and packet loss) to its destination because modern day routers are not smart enough maintain this time dependent information. One could also argue that routers needn’t ever be this smart.

Manual Intervention:

Most Internet businesses are intimately aware of these issues and usually employ some senior routing personnel to combat problems as they arise. Inevitably customers will complain about slow service and often these complaints fall upon the shoulders of the content provider to resolve these matters. However when there are numerous paths that traffic can traverse between any two points on the Internet, the complexity of resolving performance issues can be daunting – often taking weeks of research, testing, calls to service providers, etc. Finding the problem (if and when it repeats itself) is half the battle but finding a permanent solution to the problem regularly exceeds the capacity of even the most senior routing expert. While many companies still employ people to do just this sort of thing (and arguably they do a good job of this) the time is fast approaching when the complexity of this task can no longer be handled manually.

Introduction to Route Control Software

A common misconception here is that BGP (or even routers in general) are broken and need to be fixed. This is not the case. Rather they merely need some augmentation to make them more efficient. Route control, or intelligent routing software provides that assistance by conducting some type of continuous, real time performance measurements (probing) on a list of frequented Internet destinations to generate an optimized routing table for each service provider connection. Since all service providers are different (could be Tier One, Tier Two, or Tier Three) their advertised path to a particular destination is also likely to be different. Most route control software vendors don’t stop at measuring network performance when gauging which route traffic should take. They also take other external matters under consideration such as contract limitations, bandwidth costs, time of day, day of month or week, cheapest burst rates, monthly totals of traffic, etc. - all in a effort to deliver the best value, performance, or both to the Internet business owner.

Limitations of Route Control Software

While the Internet operates bi-directionally, most all route control software doesn’t. When route control software computes an optimized path for traffic that path is usually optimized in one direction (from the source where the route control software is located to the destination of where the data is headed). The return path is determined by the destination ISP that may or may not employee route control. The behavior of returning by a route other than the one you went out on is called asymmetric routing. When the paths are the exact inverse, it's called symmetric routing. However the inability (in most cases) to optimize the path of incoming traffic may not be a significant limitation. If you adhere to the notion that most all Internet traffic is downloaded (as opposed to uploaded) the need for Internet businesses to optimize the downstream path to their customers is perhaps of the most importance.

Scalability is also a limitation for many route control software vendors. Most all have built quite large platforms, but how well these platforms scale into the Gigabit Ethernet speeds and increasing fragmentation of networks at the enterprise level is a rising concern. Fragmentation of networks and Gigabit Ethernet increases the likelihood of more and more destinations or prefixes will have to be probed at much higher rates to maintain valid data about the best path.

The nice thing about route control software is that whether or not it continues to work, it will not restrict the flow of data through your network. That is because it runs along-side your existing routers to help them route more effectively. If route control goes down for any reason, your routers will continue to work using default paths learned from the last successful update from your ISPs rather than real-time updated paths. The scalability must merely address the increasing number of destinations (prefixes) rather than keep up with the speed of ongoing data traffic. Having route control software work in parallel (as opposed to it running within a router) helps maximize your routers’ horsepower for routing packets rather than determining the best path.

Primary Uses of Route Control Software

The two most common applications of route control software are enterprise VPN and web traffic. VPN services optimize paths for traffic destined for remote offices. Most VPN product offerings also feature a scaled down appliance for placement at the remote facilities. These smaller appliances only need to optimize a subset of available Internet prefixes, but this capability is more than sufficient to connect up with the company headquarters along with several remote employees. A larger appliance sits at the corporate headquarters and facilitates communications with these remote offices as well as the Internet as a whole.

Web traffic is representative of the content generated by popular web sites like Amazon.com, Application Service Providers (ASP), search engines like Google, or perhaps even a Broadband Service Providers (BSP). In the case of the BSP, the amount of data leaving their network is small as opposed to the amount of data coming into their network. Typically the ratio of packets leaving a consumers PC to packets received by a customers PC is 20:1 to 60:1 for multimedia sites and climbing. However, a need still exists to optimize BSPs’ multihomed connections to the Internet. Broadband customers pay a $20-$30 monthly premium over dial-up service for a single thing, speed. Broadband customers are extremely sensitive to issues like latency, packet loss and throughput. Route control provides the broadband vendor with the tools to mange the daily and sometimes hourly changes in the performance of their customer experience by automatically adjusting the routes for their customers’ traffic to the best performing provider using real time data. This real-time monitoring and correction reduces the number of trouble calls to customer care for slow performance and increase customer satisfaction.

In one well-documented case, a Tier 2 ISP experienced substantial packet loss and increase in latency overnight on their primary Tier 1 providers link. The route control product moved a large majority of routes to the alternate provider without human intervention. The next morning, the ISP noted the change over and upon recovery, the change back. Their senior network engineer noted that in cases like this, he would have been summoned in the middle of the night to fix the problem. By the time he analyzed the situation and moved to the alternate link, the event would have been over and the primary link would again have the preferred link requiring moving back to no benefit of the customer. As a result of the route control product, no trouble calls were received by customer care and the network engineer got a good night’s sleep.

Additional Uses of Route Control Software

A new, and unintended use for route control products applies to only the largest bandwidth users, but is interesting and instructive nonetheless. Web Content Producers (WCP) that use multiple providers of large links (multiple OC3 and above) are using route control products to manage their cost by watching usage, reporting on utilization in real time and automatically switching routes based on contract pricing. The WCPs are using this data and the automatic control to leverage better pricing from their providers and minimizing their cost at the same time. This also puts control of their Internet routing back into their hands as opposed to the engineers at the ISP.

A second interesting use of route control is to manage multiple links from the same provider. This is a very special case and requires that the links terminate in different points on a providers network typically called POPs (Points of Presence). Since the links out of two different POPs are usually managed independently and the loads on a POP are dependent on who and what is connected to the POP, this configuration appears almost like multihoming to a route control product. The route control product may only affect the first two or three hops into the provider’s backbone network, but sometimes, this is a sufficient gain to justify its expense. The real value here is that it tends to load balance the links better then can be done manually.

Yet another use for route control may be for the small business market where route control software could be placed on next generation residential gateways that host multiple Wide Area Network (WAN) ports. In this way, route control could manage a company’s redundant Internet connectivity options (Cable, DSL, Wireless, and Satellite) in such a way as to capture the best value for their Internet traffic. It would also allow these companies to seek more variable bandwidth connection contracts (such as pay by the bit) over today’s fixed flat rate data offerings.

Route control may also be used within ones network to manage/optimize internally routed customer or enterprise VPN traffic or potentially even Multiple Protocol Label Switching (MPLS) traffic. Its clear we have only begun to see the many uses for this product.

Return on Investment:

Banking a return on an investment (ROI) in route control software can vary depending on a number of things including: the type of business (content provider, broadband service provider, ISP, etc.), amount of traffic, and the number of local multihoming options available. For example, if you’re a captive audience for Internet connectivity with limited available local options the value of route control software will be diminished. Likewise if you don’t experience much traffic (10Mbps or less) the value of this software is also diminished. Withstanding these limitations, actual customers of route control software have seen any where from 3-12 month return on investment in practice. This translates into shorter return on investments for larger Internet companies - those carrying large amounts of traffic or supporting numerous (3 or more) Internet connections.

Conclusions:

However broadband service providers elect to cope with the increasing performance variation within the Internet service provider arena, two things are clear. First is that broadband service providers must seek multihoming to protect their service from unforeseen catastrophic failure. The second thing is that once multihomed, the sophistication of managing and optimizing these links goes beyond what most individuals can handle (no matter how skilled they are with BGP). This might be analogous to why one might consider using computers for accounting instead of a pencil and ledger. Both are sufficient to the task, but using computers allows you to maximize the potential of the accountant. Route control software provides the intelligence these multihomed networks require to capitalize on the performance available in an increasingly commoditized Internet connectivity.

It is important to understand that the true value of route control software is not cost savings and that realization of any cost savings could be difficult to observe in practice (even though vendors have gone to great lengths to show their products can save you money). The true value of route control software is performance improvement that translates into a higher quality user experience for your customers and in-house control of your Internet routing. If you need to cost justify quality service to your customers, route control software will always seem like an unnecessary expense unless you’re a large enough service provider make it pay rewards beyond performance gains.

Special thanks goes out to David McMahon for his contribution to this article!

Listing of some route control software companies:

Can Birds-Eye.Net help you or your Company?
Receive your Birds-Eye.Net articles and white papers hot off the presses by adding our RSS feed to your reader.

 

(C) Copyright Birds-Eye.Net, All rights reserved.
It is against the law to reproduce this content or any portion of it in any form without the explicit written permission of Birds-Eye Network Services, LLC. Federal copyright law (17 USC 504) makes it illegal, punishable with fines up to $100,000 per violation plus attorney's fees.