Birds-Eye.Net Oversubscribing Broadband Networks

Oversubscribing Broadband Networks
Where subscriber experience goes head to head with broadband operator capital costs

By: Bruce Bahlmann - Contributing Author (your feedback is important to us!)

Why is my broadband data connection getting slower? Broadband data subscribers increasingly find themselves asking this very question. Not so long ago the subscriber’s computer or even some Internet sites were largely to blame for a majority of broadband subscribers’ woes of slow service. However, those days are long gone, as bottlenecks are emerging in the access network as broadband service providers seek to rationalize the economics of offering flat rate, all-you-can eat service.

Broadband data service providers have raised the price for service, reduced the bandwidth available to subscribers, and implemented various measures of oversubscription in an attempt to keep capital costs in check. Broadband operators seek to balance these measures to provide a service that is within acceptable levels to their average subscriber rather than cater to the power users who were their early adopters.

To better understand oversubscription and what it means to operators and subscribers we’ll work through something called a “busy hour calculation.” A busy hour calculation seeks to determine how much bandwidth is being used at the most congested time on the network and whether the network is sufficiently sized to carry that traffic.

Regardless of whether you’re using Digital Subscriber Line (DSL) or cable broadband service, one encounters a number of traffic (bandwidth) bottlenecks. These can occur within your home network (if you have multiple devices connected), it can occur within the “last mile” of cable that connects your home to the service provider (note DSL is effected by this but in a different way), there can be a bottleneck between your service provider and the Internet, and between that link and your final destination.

Oversubscription is mainly to blame for the bottlenecks between you and the service provider as well as between your service provider and the Internet. In this article we are only going to look at the former of these bottlenecks.

The first bottleneck due to oversubscription is within the last mile of service. The last mile connects subscribers to their service provider. For DSL service, this relates to the distance limitations of DSL transmission technology in providing residential data service to customers – basically the further you are away from the central office the less bandwidth your service is capable of delivering. Over subscription does not really come into play for DSL’s last mile because there is a one-to-one allocation of resources (hardware and transmission lines) between the service provider and each customer – however bottlenecks are possible on the network that aggregates DSL last-mile traffic.

Cable modem services don’t suffer from distance limitations like DSL, but because cable is a shared pipe, bottlenecks can emerge from access network due to oversubscription. To offer advanced services like broadband Internet access, cable operators have segmented their cable systems into numerous small sections of cable feeding smaller numbers of homes. Each section of coaxial cable (or hybrid of fiber and coax called HFC) can facilitate connections for about 500 homes. If that stretch of cable can achieve 10-percent penetration for data services it would mean that about 50 of the 500 homes passed have become subscribers.

Most residential data services (including DSL and cable) operate asymmetrically, meaning they have less upstream than downstream capacity. Cable operators use a standard delivery mechanism called Data Over Cable Service Interface Specification (DOCSIS) that was initially designed to deliver around *27Mbps downstream and about *2.5Mbps upstream, which is shared among users on a network segment. However, only about 75-percent of this bandwidth is actually usable in practice due to inefficiencies in the cable plant and outside interference. If you take this inefficiency into account there is about 1.92 Mbps usable bandwidth on the upstream and 20.73 Mbps usable bandwidth on the downstream on a typical DOCSIS 1.0 network.

That still sounds like a lot of bandwidth in terms of allocating this to potentially 500 homes.Not so fast. Cable operators like any other data provider attempt to squeeze as much throughput out of their equipment as possible.Rather than allocating all of this bandwidth to a single node, operators combine several nodes together (sometimes 12 or more) to a single Cable Modem Termination System (CMTS) card, typically with one downstream and four, six, or eight upstream ports - See Figure 1.0.

*Note that DOCSIS version 1.0 offers the bandwidth described above. Newer versions of DOCSIS (such as 1.1 and 2.0) offer increased bandwidth in the upstream but are not widely deployed. DOCSIS 1.0 still exists in 99% of the marketplace.

Node combining allows cable operators to reduce the number of CMTS units they initially require to deploy the service, lowering their initial capital investment. From a subscriber perspective, combining means that more subscribers could potentially share the same available bandwidth. During initial deployment, broadband operators do not typically know what kind of take rate each community will generate. Therefore it is extremely difficult to appropriately size the service (using combining) beyond making an initial stab at providing basic service coverage for a reasonable and affordable capital expense. Thus capital budgets dictate how dense each CMTS is combined with nodes.

Upon this combining (which can be initially excessive) most (if not all) homes passed will be capable of supporting the service. However, the node density on the CMTS is such that any one combined area can quickly become saturated with subscribers if not properly managed. Proper management includes watching subscriber levels on each CMTS blade and then relieving the ones that have become oversubscribed by splitting up the initial combining across 2 or more CMTS blades. Since cable operators are not positive where their subscribers will be coming from it is less risky to maximize coverage for their capital dollars than purchase and deploy additional hardware upfront in an effort to decrease node density. This practice may be the least expensive way to offer the service but creates problems down the road. The problem comes after this technology has been installed and customers have been added. At this point, the initial capital investment is spent but further capital investments must be made to maintain prescribed service levels. These capital investments appear to be more difficult – perhaps because they represent an additional cost to maintain existing services as opposed to the initial investment that was merely a cost to offer the service.

Cable operators address this problem by either purchasing additional equipment to provide relief to the oversubscription or continue to add more subscribers to the over crowded CMTS. In an effort to keep additional capital expenditures low, the latter tends to be the road heavily traveled. Lets review the implications of this action by providing an example. If a cable operator were offering a broadband data service with 512 Kbps downstream and 128 Kbps upstream access speeds, the available upstream bandwidth on the CMTS can support 15 (1.92 Mbps divided by 128 Kbps) simultaneous active subscribers and the downstream could support up to 40 (20.73 Mbps divided by 512 Kbps).

Of the 50 actual subscribers of the service within a single node, one can assume that not all are online at the same time. Operators typically estimate that up to half of the subscribers are online at the same time during peak hours. Of these online subscribers, perhaps a quarter are actively using the service (uploading/downloading) at the same time. Therefore, there would only be 6.25 of subscribers online and simultaneously active using the service during peak hours (50 subscribers x 50% x 25%) per 500-home node. Together, these subscribers require 3.2 Mbps (512 Kbps x 6.25) of downstream peak bandwidth and 800 Kbps (128 Kbps x 6.25) of upstream peak bandwidth. This traffic represents 42-percent of the available upstream bandwidth and 15-percent of the available downstream bandwidth supported by a single CMTS upstream and downstream port.

Based on these numbers and using the example 512/128-Kbps service, the link would be on the verge of oversubscription when more than 320 subscribers were installed on the downstream and 120 on the upstream. When the technology was designed, it was believed that subscribers would download far more than they upload (thus the drastic differences in available bandwidth on the downstream versus upstream).In practice, however, this has not exactly held true as new applications increasingly push the limits both ways.

Current CMTSs port configurations allow a cable operator to balance upstream and downstream loading. The most common today is a 1x6 configuration, with 1 downstream per 6 upstream ports per CMTS card. Thus, in practice, an operator may take 12 nodes (500 homes each), combine them together and connect them to a single downstream port on the CMTS. In the upstream, they would then combine 2 nodes per port. The net result of this is that you no longer have 50 customers (using the example above) served by the available bandwidth. Instead you may have 12 times that number or more.

The increase in subscribers per CMTS port as a result of by combining creates a potential problem for the downstream as well as the upstream in terms of consistently delivering the 512/128-Kbps service. For example, if the cable operator has 12 nodes combined with 10-percent penetration, the downstream port would serve 600 (10% x 500 x 12) subscribers, yielding a downstream port that is 188-percent (600 divided by 320) oversubscribed on the downstream and 500 percent (600 divided by 120) oversubscribed on the upstream.In this example, the maximum bandwidth the cable operator can deliver to during peak times is 276 Kbps downstream and 25 Kbps upstream traffic to these subscribers.

From the subscriber’s perspective this adjusted bandwidth during peak usage is about 53-percent (276 Kbps divided by 512 Kbps) of their subscribed downstream bandwidth and 19% (25 Kbps divided by 128 Kbps) of their subscribed upstream traffic. Essentially the service is no better than half as fast as what the cable operator advertises during peak times. Fortunately, subscribers only notice a slight drop in their downstream experience when their usable bandwidth drops from 512 Kbps to 276 Kbps due to limitations on the Internet itself – the Internet rarely (if ever) delivers better than 300k. But if a subscriber is uploading something large they will visibly notice the delays caused by the reduced bandwidth. In practice, 250-300 Kbps downstream is the sweet spot for residential data services, whereas it is only 64-96 Kbps upstream. A cable operator that can deliver that speed to the subscriber consistently will have very happy customers. However, those subscribers whose service level falls out of that sweet spot will begin to visibly notice a reduction in their speed and start to complain.

Interestingly, in the real world, many MSOs further cut corners by provisioning more than 1,000 subscribers per downstream port, cutting available bandwidth to each subscriber by half again.

A number of studies have offered projections for subscriber demand for bandwidth. Of these studies one completed by Randy Nash of Motorola Inc. (a CMTS vendor) suggests that the current (2002) bandwidth demand per home passed is 60 Kbps downstream and 5 Kbps upstream.If we rationalize these numbers in terms homes passed in our first example of 500 homes passed we arrive at a bandwidth demand of 30 Mbps (60 Kbps times 500) versus our earlier calculation of 3.2 Mbps of bandwidth demand for 500 homes passed. If broadband operators were to build out their residential data service using Randy’s numbers they would essentially require a CMTS blade per node, where as if they used the busy hour calculations within this article the operator could combine up to 10 nodes per CMTS blade. While both approaches would work equally well, economics will dictate which method works best for each cable operator.

One interesting data point that one can extract from Randy’s study is that broadband operators can expect to see about a 10-12 percent increase in bandwidth demand per subscriber per year. The increase can be attributed to a combination of things including increased usage, new applications, etc. and represents something useful to keep in mind for future capacity planning.

The over subscription numbers can get worst if not properly managed by broadband operators. For example, if the average number of subscribers online surpasses 25-percent or penetration tops 10-percent (in practice broadband operators see anywhere from 10-24 percent penetration per node) the amount of over subscription could more than double.

Upstream	Downstream	Upstream	Downstream
Service	Service	Subscribers Supported	Subscribers Supported
64k	64k	240	2,592
64k	128k	240	1,296
128k	256k	120	648
128k	512k	120	320
256k	1,024k	60	162
256k	1,536k	60	108

Other factors will impact these numbers as well. For example, many cable operators have begun offering tiered services – different speed services for different prices (see Table 1 for how many subscribers could be placed on a CMTS without oversubscribing). Tiered services are the best overall value for broadband data subscribers because they allow them to only buy the bandwidth that meets their budget. However, in an over subscription situation tiered services can allow subscribers to migrate to the service tier that more closely represents the actual bandwidth that broadband operators are able to deliver when they most use the service. Why pay for a T-1 when 256 Kbps is the maximum bandwidth available (due to oversubscribing) during peak times or when subscribers most frequently use their broadband connection. Tiered services could have the opposite effect for some broadband operators as subscribers flock over to lower priced services that best reflect the actual bandwidth they can obtain during the times when they most use the service. However, while lowered bandwidth services can support more subscribers without oversubscribing so can the competition. Thus, operators seeing fewer and fewer subscribers paying for higher tiered services may find they are way oversubscribed. Over subscription reduces the value of the higher tiered services, lowers the quality of the experience for most of the subscribers affected, and creates opportunities for competing services to offer the same (if not better) quality service for a better price point.

A number of traffic studies have been conducted that tracked subscriber usage. Although efforts continue on this front some preliminary results seem to indicate the “busy hour” tends to be from 8 pm to midnight on Fridays. Other than that, weekday traffic seems to increase during the afternoon, peak around midnight, and then bottom out around 5:00 am. Interestingly, all these studies seem to conclude that usage patters are seasonal, unpredictable, and yet steadily increasing. Some studies have gone to the extremes of attempting to profile subscribers and model usage patterns to the point where they can yield some type of traffic simulators. Creating sophisticated models to predict bandwidth usage based on subscriber profiles, network history, installation rates, and actively polling the network may be useful at some point in the future, but in terms of providing answers to today’s problems it is more like rocket science – which is perhaps useful to the progress of human kind but not of much use to the average person on the street.

You may have noticed that nowhere in this article did I mention any type of monitoring or active polling as a tool to gather information about networks to help make combining or bandwidth allocation and planning decisions. Monitoring is a terrific tool for Network Operation Centers (NOCs) to watch the availability of service critical network hardware and applications in an effort to keep everything working and reliable. Beyond that, monitoring or active polling is of little significance in determining planning events that have future economic and customer experience implications. Bandwidth planning is not an exact science so having up to the minute information is not worth the expense of gathering it – having it is merely a luxury. Instead, having simple planning models that can project the maximum number of subscribers that can be supported by a CMTS is good enough. Occasionally, it may be useful to run some bandwidth consumption reports across all your CMTS to check the accuracy of your numbers, but not to make planning decisions. Such sanity checks need not occur more than once a year.

If you’re going to offer a broadband data service make every attempt to ensure that your subscribers receive what they are paying for no matter what hour. In this scenario everybody wins and your subscribers remain happy.Perhaps a good way to manage your peak traffic is through properly sizing Internet Protocol (IP) subnets across your CMTS blades. Managing the population of subscribers using IP address techniques is a very simple means of managing over subscription without having to actively poll devices in the field to determine your next course of action. A relatively safe hard point is a class C subnet per CMTS blade. This hard point allows for up to 253 subscribers on the network – well below the 320 supported on the DOCSIS downstream channel and twice that of the 120 supported on the DOCSIS upstream channel. Future versions of DOCSIS will provide relief to the upstream and further solidify this class C hard point.

Another recommendation is to only offer services that you can support effectively on a CMTS. Cable operators interested in offering tiered services can easily forget that each subscriber they provide with a T-1 service costs them three 512-Kbps subscribers or six 256-Kbps subscribers. Almost every cable operator initially began offering residential data services advertising a T-1 speed to lure subscribers, but few of these types of services remain because they are impossible to support from an economical standpoint unless cable operators fetch 3-4 times their normal subscription rates. In addition, these services become so over subscribed that subscribers rarely ever see much more than 256-512 Kbps anyway. So, if you offer tiered services make sure there is value in what your offer at the upper tier or subscribers will seek out the lowest cost for the service delivered.

Future versions of DOCSIS will include variations of Quality of Service (QoS) that allow cable operators to allocate bandwidth to sensitive services such as Voice over IP (VoIP). Of course, making these reservations impacts the available bandwidth to other services. If oversubscribing didn’t exist best effort would still work beautifully for all services including VoIP. Unfortunately, QoS further complicates the problem of handling oversubscription so voice services can traverse cable networks at the expense of best effort traffic.