Managing Networks Across HFC Nodes
A complete look at broadband’s renumbering operation

By: Bruce Bahlmann - Contributing Author (your feedback is important to us!)

Created: August 13, 2001

Note: For help designing your provisioning system or developing tools to help test, automate, and deploy your system contact Birds-Eye.Net.

Broadband networks offer unique challenges to providing Internet service. One of the more interesting of these challenges involves managing their rapid growth using renumbering. Renumbering is a means of providing relief to over crowded networks by the introduction of a new network. This white paper will examine the renumbering operation in its many forms, how it has evolved, and suggests various improvements that could be made to reduce associated downtime, user error, and overall complexity. 

CMTS Architecture History 

To understand renumbering, it is helpful to know how various parts of the broadband network go together and how they have evolved. Broadband networks are composed of a number of broadband transports tied together using a Cable Modem Termination System (CMTS) as shown in Figure 1.0. Each broadband transport resembles a combination of fiber and coaxial cable that reaches out to a portion of the broadband operator’s subscribers and is referred to as Hybrid Fiber Coax (HFC). These broadband transports (also known as fiber nodes or HFC nodes) span anywhere from 250 to 2000 homes depending on when it was put into service (newer transports span fewer homes). Each CMTS handles a number of broadband transports (typically 1-8) and can function as a bridge or a router.

Figure 1.0 Basic CMTS Connectivity 

When a CMTS functions as a router, the 1-8 broadband transports that are bundled together at the CMTS fall onto a single network (or subnet). This bundle is the smallest and the largest unit supported by a router-based CMTS. Meaning, once this bundle is wired to the CMTS, it cannot be made any larger or broken down any smaller without rewiring the CMTS. Because of this, broadband operators try to strike a balance when they deploy router-based CMTSs so that they are connected to a bundle of broadband transports that are initially large enough to be economical to deploy yet small enough to not require rewiring later as subscribers are added.  

When a CMTS works like a bridge (see Figure 2.0), the broadband transports that are bundled together by the CMTS fall onto a single Virtual Local Area Network (VLAN). Bridge-based CMTSs don’t have the same size restrictions as router-based CMTSs. This is because you can initially add as many CMTSs to the VLAN as you want (so long as the subnet or subnets you allocate to this VLAN is/are sufficient to support all the CMTSs). Bridge-based CMTSs do share a similar problem to that of router-based CMTSs in that they cannot serve a smaller bundle of broadband transports without rewiring.

Figure 2.0 Using a VLAN and Bridge-based CMTS 

However, the ability to easily grow a bundle provides bridge-based CMTSs with a significant advantage over router-based CMTSs. For example if you’re installing a new hub site and want to bring everything on line, you would only require one relatively small subnet. Compare this with using router-based CMTS that would require you to create a number of small subnets to activate all the CMTSs. Since each subnet wastes 3 IP addresses it doesn’t take long for you to use up a significant number of IP addresses just to deploy your hardware – especially since hubs can have anywhere from 10 to 400+ HFC nodes. Another disadvantage of the router-based CMTS is that, once deployed, you really have no control over where your subscribers are going to come from. If you’ve just deployed data services in a hub all these subnets are very small – so a flurry of installations in a particular neighborhood can easily deplete all the free addresses associated with these CMTS. Using the bridge-based CMTS, these installs would not deplete your free addresses as they are spread across a number of CMTS.  

Although you cannot further breakdown a single bridge-based CMTS, you can easily reduce the number of bridge-CMTSs associated with a VLAN. The ability to create initially large VLANs and then easily break them apart is a significant advantage of bridge-based CMTS architecture. This convenience easily made up for the additional requirement of an Ethernet switch (that permits the creation of VLANs) for bridge-based CMTSs. The router-based CMTS was initially advertised as not requiring the Ethernet switch thus saving money on additional hardware. However, what the router-based CMTS saved in additional hardware cost it more than made up for this in increased management and up front planning. 

Some initial router-based CMTSs offered limited VLAN support. These router-based CMTSs place the CMTS on card (or blade) and allow you to add anywhere from 1 to 8 of these cards into a single chassis. Once seated in the chassis, these cards could be bridged together to form a VLAN. However, this only enabled the CMTS to place 2 to perhaps 8 CMTS blades together. In a larger hub site, these CMTSs still require you to supply a number of subnets to bring all them on line but it reduces the overall number of networks required by previous router-based CMTSs by a factor of between 2 and 8. 

Going Operational 

Once CMTSs are networked and brought on line, your work has only begun. That is because you cannot entirely forget about them. This hardware requires constant monitoring to ensure they stay operational and you need to pay close attention to IP address consumption on the subnets that span these CMTSs. Router-based CMTSs require significantly more management because the IP address subnets on which they reside are usually smaller than that of bridge-based CMTSs. The larger the subnet the more subscribers you can install without worrying about running out of IP addresses. However IP addresses must be carefully rationed out, as their efficient use will enable you to more easily obtain additional addresses – inefficient use will require you to consolidate. The subscribers who connect to the broadband transports receive their IP address automatically through software called a Dynamic Host Configuration Protocol (DHCP) server. The DHCP server is configured with a range of addresses that are valid for a particular CMTS or collection of CMTSs (depending on the use of VLANs).  

The use of the word “automatically” here greatly relies on the implementation of a communications protocol that the subscriber’s computer & the DHCP server uses, the functionality of the DHCP server, and most importantly the knowledge and skill of the individual(s) overseeing the IP address allocations and the management of the DHCP server. In other words, “automatically” is a by product of the mature standards-based DHCP protocol, specific broadband friendly DHCP features, as well as hard working individuals who’s behind the scenes efforts to oversee IP address allocation, assignment, and utilization provide a highly reliable Internet service. A DHCP server is merely a software tool that can manage specific blocks of IP addresses with which it has been configured. In fact, the level of effort and skill required to properly/intelligently configure these blocks of IP addresses on the DHCP server is often a forgotten manual process, yet it is essential to successfully running a reliable network. 

Assignment and configuration of IP address blocks to DHCP servers closely follows configuration of network hardware in the field – this activity is coordinated as each is dependent on one another. When a new CMTS is installed in the field, it is either configured to join a VLAN or assigned subnet information depending on the type of CMTS explained previously (bridge or router based CMTS). These deployments are summarized below: 

Router-Based CMTS 

Router-based CMTS require a subnet to be deployed. The network engineer must first draw this subnet from the pool of available IP addresses managed by the network administrator. The network administrator will need to make an educated guess as to how large a subnet to dispense as well as seek any other efficiencies (e.g. route summarization). The subnet that is dispensed represents a first attempt to satisfy the operational needs of that subscriber area served by the CMTS. This information is then manually configured into the CMTS along with the chosen frequency pair that will be used by downstream Cable Modems (CMs). This frequency pair must not be used by any existing services being broadcasted over the broadband transports selected. Both the subnet and frequency information must be entered prior to the CMTS being connected to the broadband transports to prevent it from taking out existing services being broadcasted – some CMTS have default frequency settings. After the CMTS is configured it is then wired to some number of broadband transports. The broadband transports selected for each CMTS are based on existing wiring and combining that is available in the facility where the CMTS resides. Selecting broadband transports for a CMTS requires knowledge of how the facility was wired and the proper engineering design documentation kept on site. Bundling broadband transports on the CMTS falls outside the area of expertise for most network administrators and network engineers. This activity is usually done by broadband operator headend technicians or staff engineer and/or outside organizations that specialize in wiring headends. About that time, or perhaps after the CMTS is configured, the network administrator enters this subnet information into the DHCP server(s) on the network. The combination of the these two events along with connecting the CMTS to the broadband transports enables IP addresses to be dispensed over the area serviced by the CMTS. 

Bridge-Based CMTS 

Bridge-based CMTS require the same wiring procedure as router-based CMTS however they can generally be deployed across a smaller number of broadband transports. This is because they can easily be joined together using a VLAN so they are not limited by the availability of IP addresses or the need to keep IP address utilization high. As a result, the number CMTSs deployed in any given facility is likely to be higher for bridge-based CMTSs than router-based CMTSs. This permits the bridge-based CMTS to service a smaller population of subscribers as the demand for bandwidth increases. Deploying bridge-based CMTSs requires each CMTS to have a minimum of an IP address and frequency pair. The CMTS’s IP address is generally a private IP address that is carved out the subnet it joins to ensure it will not be dolled out to any other hardware. This reservation of sorts also creates a link between the CMTS’s IP address and the broadband transports it services. The fact that its IP address is private allows the CMTS to easily be made invisible or inaccessible to subscribers as well as the Internet – an important distinguishing characteristic of the bridge-based CMTS. Since the router-based CMTS represents the subscriber default route to the Internet it must be accessible by subscribers as well as the Internet. 

The actual hardware that supports these different deployment options come in different configurations and addresses certain operational needs. These different hardware configurations are discussed below: 

Stand-alone Configurations 

The simplest CMTS configuration is where each box serves as a single CMTS (similar to Figure 3.0). Here the router acts on a single bundle of broadband transports and is basically self-contained.

Figure 3.0 Stand Alone Configuration 

The self-contained model provides each bundle of broadband transports with its own dedicated CMTS that has its own power supply(s), memory, processor, etc. This model actually works especially well in broadband, as the broadband transports it serves are also single points of failure. Having each bundle of broadband transports terminate at a self-contained CMTS allows you to distribute the work of maintaining network connectivity with subscribers across multiple CMTS. Self-contained CMTS also enable the greatest service flexibility – for example if one goes down you only loose customers in that area (all others are unaffected) and it is generally easy to troubleshoot and replace these units. Another nice feature is it enables you to spread out a bit more (the footprint of these units may be small but you need a lot of them – so it forces you to spread out your interconnects rather than concentrating all of them into one location. Having all connections spread out makes service, upgrades, etc. easy without impacting any more subscribers than you have to.

Figure 4.0 Redundant Stand-Alone Configuration 

In some locations it may make sense to provide redundancy in the CMTS functionality (e.g. a broadband transports that service business customers). The redundant stand-alone configuration is able to deliver this by duplicating connections of the existing CMTS with an additional CMTS. This can be easily done when the CMTS is bridge-based. There is currently insufficient information available as to whether the router-based CMTS supports this configuration to this same extent. Bridge-based CMTS support this function by having each CMTS operate at a different frequency pair yet within the same subnet (VLAN). If the subscriber’s current CMTS were to fail, their CM would be forced into search mode where it would find the other CMTS and come back on line. Since this scenario does not involve a change in subnet for either the CM or the CPEs behind them, the resulting outage would be brief. This configuration is superior to other models as it allows the CM to learn which CMTS is active in the event of failure. Ideally, the best implementation of this configuration would also require that individual CMs be staggered across both CMTS so as to minimize any resulting sign-on storm in the event one CMTS fails. Staggering would also maximize on capacity & capability of both CMTS in delivering data services to their respective CPEs. The only problem with staggering is that in order for the CMs to stagger, they would need to require the OSS to inform them of their assigned frequency pair at the time of provisioning. This defeats/complicates the capability of redundancy in the CMTS because if the CMTS a CM uses fails, for CM to obtain a new CMTS, the OSS would need to assign it the frequencies being used by the remaining working CMTS. Since this would require the OSS to manage both CMTS and know which is working and which is not, it defeats the simplicity of the CM just finding the next available and continuing to work – in other words the CM would not receive its assigned frequency via the OSS but would learn it during its booting cycle. 

Consolidated CMTS Configuration

Figure 5.0 Consolidated CMTS Chassis 

Another hardware configuration of the CMTS involves an all-in-one chassis as pictured in Figure 5.0. Like the self-contained model previously described, this unit consolidates all CMTSs into a single chassis that houses several key redundant components and is able to support either router or bridge based CMTSs. This is a great solution for operators with limited space yet a large number of broadband transports to service – it also represents one of the most cost effective hardware solutions. However, there are drawbacks to this model. For example, you cannot easily service a single CMTS nor can you easily swap out cards, as first generation chassis required front panel coax connections. Headend coaxial cable is very stiff and once more than 20 or so of these cables are connected to the CMTS cards they become increasingly difficult to remove and service (each coax must be loosened with a wrench before the card can be removed). Some CMTS vendors address this by placing the coax interconnects on the back of the chassis – which helps speed card removal and enables some type of redundancy (typically 1 to N). Minus the power supply, the Mean Time Between Failure (MTBF) of the CMTS cards should be within the same range as the stand-alone CMTS thus the overall gain is merely a smaller footprint. Beware that the failure of a consolidated CMTS chassis results in a major down time. Although this is considered rare, it is well within the range of possibilities and will result in significant down times as consolidated coax cables take a long time to disconnect and reconnect. 

Whether you deploy a router or bridge based solution that is stand-alone or consolidated you still need to configure/distribute IP addresses for subscribers downstream from these CMTS. Your method of allocating these IP addresses must optimize your overall utilization of these IP addresses while making room for ongoing growth in your subscriber base. When you initially build out (deploy) your data delivery system it inherently has a kind of hard-wired capacity to support/sustain some specific number of subscribers. This number of subscribers is dictated by the size of the subnets you allocated to the CMTSs or VLANs as well as pools of IP addresses your DHCP server is configured to dispense. Upon deployment, subscribers are able to join your data services and as a result begin consuming the finite resources (IP addresses, bandwidth, etc.) that make up each subnet. Eventually, these finite resources can become increasingly burdened to the point where the service becomes “slow” or encumbered by over crowding. Another more probable result is that IP addresses will be recycled. Surprisingly, the latter of these results is the most threatening for broadband operators. This is because network administrators don’t have the luxury of allocating large enough subnets to CMTSs such that subscribers will consume all the bandwidth on a subnet before its IP pool on the DHCP server is exhausted.  

Recycling an IP pool on a DHCP server is the result of the DHCP server running out of unassigned (or free) IP addresses to provide to subscribers within any given subnet. When this occurs the DHCP server attempts to use any available expired leases that may exist with a subnet. The reuse of these expired leases is called recycling and although it sounds like a great idea, it can negatively impact the network. For example all of the following may result from recycling DHCP leases:  

·         Not all subscribers use DHCP, even though their CPEs were set up to do this initially. Instead, they configure their CPE with static IP address using their previous lease information. If you recycle their lease and try to give it to another subscriber and the original holder of the lease still has their CPE powered up, the recycled lease will fail to work – as it will detect a duplicate on the network and not become operational. DHCP servers have the ability to test to see if IP addresses are in use before they assign them to subscribers using ping. However, this is only a work around for handing out recycled leases – it does not fix the problem of customers configuring their CPEs as statics.

·         Not all CPEs use the DHCP communications protocol the same. For example, early Microsoft Windows CPEs would not release their IP address during power down. If a CPE fails to perform a lease renewal before its lease runs out the DHCP server would attempt to recycle this when according to the client it believes its lease is still good.

·         There is also the issue of time relative to the DHCP server and that of the CPE. If the CPEs run at a different time than the DHCP server this could also create instances of the DHCP server thinking the lease is expired and attempt to recycle it when according to the CPE the lease is still good.

·         Lastly, but potentially worst of all, recycling leases generally means shorter lease times. This translates into increased DHCP load handling of DHCP clients. Increasing the load on DHCP impacts the sizing and reliability factors for which it was designed or architected into the deployed system. If you change design parameters (like reducing lease times), you may well change the demands on the equipment in your system thus requiring either hardware/software upgrades or possibly even require additional software hardware to address the increased load. These factors must be weighed before making such changes – also consult your DHCP vendor’s experts before you proceed. A harmless change that seemingly solves an immediate problem may well create future scalability issues that don’t become apparent for some time after this change – thus not allowing you to associate the two related events. 

Once all the expired leases are used up, the DHCP server runs out of IP addresses and can no longer provide new clients with IP addresses. In the case of broadband operators this means all installs on affected subnets are halted until more IP address space can be made available. Running out of IP addresses causes installation crews to cancel installs or leave subscribers in an unfinished state – both of which are unacceptable to broadband operators. Rather than look for ways to solve this complex problem of attempting to recycle leases (eventually this does need to happen), broadband operators resort to renumbering. 

Renumbering:

Renumbering takes an existing subnet that is reaching the end of its free address pool (note we are not talking about expired leases) and splits its subscribers between a newly introduced subnet and the existing subnet. By dividing the subscribers between the two subnets, the over population on existing subnet is addressed by moving half of its subscribers onto a new subnet. Depending on the type of CMTS you have, there are certain ways you would go about renumbering. 

Router-Based CMTS 

Renumbering the router-based CMTS will impact all of the subscribers associated with it because it represents the smallest and largest possible unit of this form of cable modem termination. Fortunately there is one way to address the IP pool exhaustion before you have to resort to rewiring the coaxial cables connected to the CMTS. Since deployments force you to allocate very small subnets to router-based subnets (remember that router-based CMTS require a large number of subnets to deploy), the CMTS is far from being taxed in terms of traffic load. Thus, you can increase the size of the subnet serving the CMTS until such time as the subnet is too large and subscribers begin impacting each other’s performance. Unfortunately, there is no known hard number that establishes the ceiling of how big the subnet can grow until it each subscriber will impact one another’s performance. It just depends on the usage patterns of the subscribers within each subnet. For example, you could have a network as small as 64 IP addresses that is bandwidth limiting in one area yet have network as large as 256 IP addresses that is not bandwidth limiting in another area. Since broadband operators really don’t have good traffic modeling tools to tell them when they have truly reached the ceiling of what the subnet will reliably bear from a traffic throughput standpoint, they often resort to just observing crewed forms of bandwidth utilization while paying close attention to customer complaints. When it comes time to increase the size of the subnet the following summarizes the events that take place: 

·         Subnet is deemed overcrowded by network administrator or someone in the broadband operator’s Network Operations Center (NOC). The result of this kind of overcrowded state is simply that the free (unassigned) pool of IP addresses available for the DHCP server to doll out to subscribers is reaching a critical level (typically 80-90% utilized).

·         Depending on the install rate in the area and the number of IP addresses actually left in the pool, a network-renumbering task is scheduled for a service window (typically 3-5 am Tuesday through Thursday). The NOC maintains ownership of this task through its completion and ensures that any and all resources are readied for the task to be completed successfully and on time.

·         When the scheduled service window arrives, the network administrator has already ear marked a subnet for the renumbering event. This information has been passed on the network engineer who is now in the field entering this information into the CMTS – during this time; all the CMs off this CMTS (or blade) have lost connectivity with the CMTS and are in search mode. Likewise, all customers associated with this particular CMTS are off-line. About the same time, the network administrator creates a new subnet in the DHCP server with this new information and waits for the effected CMTS to begin passing traffic on the new subnet.

·         For fallback purposes, the existing subnet where all the subscribers reside still remains in the DHCP server. This subnet cannot be reclaimed until such time as the event successfully brings a sufficient number of subscribers CMs back on-line.

·         When the CMTS is activated with the new subnet, the CMs boot requests behind it are relayed on up to the DHCP server. Since these requests come in on a different subnet than before, the DHCP server must first release the lease held by each CM before it can answer it with a lease on the new subnet. As this happens, the new subnet begins filling up with leases from CMs and once the first couple CPEs* also happen to show up in the list of leases (note CPEs are generally slower to correct themselves as a result of a renumbering as they are not forced into a reboot like the CMs as a result of losing their CMTS) the renumbering event can be considered a success. 

* Note that CPEs represent the most impacted component in a network-renumbering event. This is because they are all running on information (an IP address lease) they thought was correct but all of the sudden it is not. Unlike the CMs, they don’t have the benefit of being forced into a state of reboot, so they must wait till they reach a state in their lease that forces them to recover. There are two states in every lease as defined by the DHCP protocol that would enable a CPE to recover from a renumbering event. The first possible state is called rebind. During this state (which is by default around 87% of the lease time but can be explicitly set to other times on some DHCP servers) the CPE is forced to ask all available DHCP servers for an update on its existing lease. This request is a broadcast and is forwarded by the CMTS to the DHCP server. The DHCP server will send a NAK back to the CPE saying that its existing lease is no longer valid. This forces the CPE to into a discover mode which will allow it to obtain a new working lease on its new subnet. The other state occurs when the CPE’s lease expires. This forces the CPE into a discover mode allowing it to recover from the renumbering event. Unfortunately, lease times are long so the time to rebind may be hours or even days depending on how the network administrator has configured their DHCP server. As a result, the CPEs behind CMs may take some time before they are forced to broadcast a message that will enable them to recover. In the mean time, they are completely off-line – isolated as a result of using an IP address lease that is invalid on the current network that they reside. 

·         Upon success, the previous subnet is emptied (leases are deleted) and the subnet is then removed from the server. It is important that the leases are first deleted before the subnet is removed, as leases are stored in the database where as the subnets are not. If you remove the subnet before you delete the leases, you hang (or strand) these leases in the database. In most cases, the only way to remove these leases is to recreate the subnet, delete the leases, and then remove the subnet.

·         If the event was a failure (the result of faulty equipment or just that you ran out of time to complete it properly), the network engineer merely reloads the saved configuration in the CMTS to recover its previous state and the renumbering event is rescheduled.

·         If the event is successful, the renumbering event is closed. 

Since installing a new CMTS and or CMTS blade is a big deal, the broadband operator will attempt to squeeze everything thing out of their existing hardware before they resort to adding more. So increasing the size of the subnet is always a better solution than pulling spare equipment out of the warehouse and installing into the production network to diffuse an overcrowded situation. 

Note that growing the subnet beyond the size that would force you to install additional equipment will complicate your management of IP addresses through a future renumbering event. This is because if you split the number of subscribers over two subnets (the existing large subnet and perhaps a new smaller subnet) you run the risk of stranding majority of the existing subnet. The best way to manage allocation of subnets is to always aim low and add as needed rather than have a large subnet assigned to a small number of subscribers. If your existing subnet is too large for the subscribers that you intend to leave associated with it, you may be better off renumbering both subnets over stranding your existing large subnet. As a result, attempting to save money in growing large subnets over forcing yourself to renumber (even if it requires you to add more hardware) will only further complicate your IP address management and likely result in a more complex renumbering event. Since these events take place in the early morning, the simpler you can make them the better. 

However, when the need arises, the following summarizes the events that take place when a new CMTS must be added as part of a renumbering: 

• Subnet is deemed overcrowded by network administrator or someone in the broadband operator’s NOC. The result of this kind of overcrowded state is that the bandwidth used by the existing number of subscribers has exceeded the capability of their subnet assigned to the CMTS. In other words, you’ve reached the end of what you can do as far as growing the subnet to diffuse overcrowding. Your only alternative is to add new hardware.
• Since this type of event is not normally driven by IP address pool exhaustion, it can be scheduled at any time. However, depending on the install rate in the area and the number of IP addresses actually left in the pool, a network-renumbering task may need to at least consider these in scheduled for a service window (typically 3-5 am Tuesday through Thursday). The NOC maintains ownership of this task through its completion and ensures that any and all resources are readied for the task to be completed successfully and on time.
• A majority of this work can be done during business hours. For example, racking the new equipment or installing the new card and configuring it with a new subnet. Likewise the DHCP server can also be configured with this new subnet and the desired coaxial cable can be tagged so as to identify their next location. In instances where the distance between CMTSs is significantly different new coaxial wire may need to be pulled – you should make sure everything reaches. Essentially, all tasks except the final movement of coaxial wires can be accomplished during non-service window hours.
• Moving these wires that represent broadband transports changes the mapping between subnets and the broadband transports they represent. For troubleshooting purposes it is important to track which subnets service which broadband transports and vice versa. For example if a CMTS goes down you will know which broadband transports are affected – thus how many subscribers are impacted. This information is also used in diagnostics and monitoring so it is important to keep these mappings up to date. Thus when your conducting this type of renumbering event you should make sure that someone updates your subnet to broadband transport mappings.
• When the scheduled service window arrives, the tagged coaxial cables are moved to the new CMTS (or blade). Some broadband facilities like to keep everything nice and neat, so moving a cable may cause it to fall out of spec with how the broadband operator wires equipment. If this is the case, moving cables may require you to shorten the cable and put new fittings on it. The movement of the cables will force the CMs at the end of these cables to go into search mode. When the cables are relocated onto the new CMTS, they should be able to obtain a new lease and configuration from the DHCP server. The network administrator or NOC personnel will closely watch these to make sure CMs are coming up and alert network engineers if they don’t see all the normal traffic coming from the CMs or the CMTS.
• In terms of fallback, it’s a good idea to first move the coaxial cables that you know will not need to be trimmed. In this way, you can get a sense from the network administrator that the event is moving in the right direction before you begin removing cables that you must shorten. If for any reason the new CMTS is not working properly, you can attempt to double check the configuration to provide the network administrator with what s/he expects as far as traffic from the CM or CMTS. If you can’t get this working, the easy fallback is to relocate the moved cable(s) back to their previous CMTS, call off the renumbering event, and then try to figure out what when wrong.
• If CMs are booting properly, your half way there. All that remains is the long wait for CPEs to correct themselves to the new network. The moment that several CPEs successfully interact with the DHCP server and obtain a working lease on the new subnet, the renumbering event is deemed a success.
• If the event is successful, the renumbering event is closed. 

Bridge-Based CMTS 

Renumbering a bridge-based CMTS is quite different from that of a router-base CMTS. Bridge-based CMTS are able to leverage the use of an Ethernet switch to greatly simplify network operations as well as network renumbering events. 

Figure 6.0 Bridge-Based CMTS Renumbering 

The Ethernet switch joins a number of bridge-based CMTS into one or more VLANs that in turn can then be associated with one or more routers as shown in Figure 6.0. The Ethernet switch offers broadband operators the opportunity to easily associate/move/switch bridge-based CMTS or routers between VLANs.  

Figure 7.0 Using Trunking to Save Switch Ports and Router Interfaces 

You can save money and complexity if you incorporate trunking between the router and the Ethernet switch as shown in Figure 7.0. Trunking is a means of sending multiple virtual circuits over a single link. Since most routers support sub interfaces, you can assign a different virtual circuit to each sub interface and trunk these to the switch. On the switch you can associate one or more VLANs to each virtual circuit in the trunk. As a result, a single link (sub interface on the router) can service multiple VLANs. Making all of this redundant gets complex and is perhaps beyond the scope of this paper but it can be done – bottom line is this configuration option limits your need to rewire. After that, you can easily increase or decrease the number of bridge-based CMTSs associated with each VLAN without standing in front of the equipment. The ability to increase and decrease the number of CMTSs on a given subnet is the key operational efficiency in effective network renumbering. 

When it comes time to renumber the same factors that impact router-based CMTS impact bridge-based CMTS. Thus we are looking at IP address utilization across subnets as well as bandwidth utilization for the subnet associated with each VLAN. If these factors tell us its time to renumber we have one of three choices available to diffuse the congestion.  

The first of these choices is exactly the same as the best-case router-based corrective measure and the result of high utilization of IP rather than high utilization of bandwidth. That is to increase the size of the existing subnet serving the VLAN that is congested. Operationally there is one additional step to performing this task. That is to assign each CMTS with an IP address off the new subnet. This can either be done automatically via DHCP if the CMTS supports this or manually through a telnet or some other session. Using DHCP eliminates manual configuration and also speeds field replacements of this hardware. 

The second choice involves diffusing a situation where you’re experiencing a high bandwidth utilization event on a VLAN. You can diffuse this situation by creating a new VLAN and then spreading the existing subscribers equally between them. This is accomplished by moving a certain number of CMTS over to the new VLAN while leaving the remaining CMTS on the existing VLAN. The result of this operation is that it only impacts half of the subscribers – those being the ones associated with the CMTS that are moved to the new VLAN. The remaining customers are not affected. The following steps are particular to this event (much of the detail of the common tasks has been removed):  

• Subnet is deemed overcrowded
• Since this type of event is not normally driven by IP address pool exhaustion, it can be scheduled at any time…
• A majority of this work can be done during business hours…
• When the scheduled service window arrives, the CMTS are moved off their present VLAN and onto their new VLAN. This VLAN is configured with a new router or has already joined a trunking group.
• Next the CMTS are given a new IP address on the subnet they now reside. This is either done automatically using DHCP or done manually through telnet. If it is done through telnet, you need to make this change happen while the CMTS is on its existing VLAN before moving it to its new VLAN or else when you move it to the new VLAN it will be unreachable.
• Once all the CMTS come up on the new VLAN you should begin to see CMs also come up, CPEs will follow later.
• For fallback purposes, all the running configurations on the switch, router, and CMTS will not be saved until such time as you have a number of CMs and CPEs up and running on the new VLAN.
• If everything looks good, you save the running configs and close the renumbering event.
• If not, you don’t save the running configs (or perhaps you just save them to your laptop) and then force a restart of all the equipment to get back to where you were when it all began and reschedule the renumbering event. Since no wires were connected during the renumbering event, everything can be recovered using the prior software configuration. 

The third choice is quite rare when using bridge-based CMTS and involves a case where you have reached bandwidth congestion on a VLAN with only one CMTS on it. When this occurs it’s handled similar to rewiring the router-based CMTS only with additional steps needed to configure the VLAN belonging to the CMTS to a new network/sub-interface and assign the CMTS a new IP address. Unless you’re broadband transports span a large number of subscribers your unlikely to experience this choice often. This is why having the CMTS act like a bridge provides you with more flexibility than when it acts as a router. 

Anyway you slice it, renumbering adds up to the introduction of a new subnet. When adding a new subnet it is important that some things are done beforehand to ensure the new subnet will not cause more problems than the one your trying to fix by adding it. There are a number of things that are required before a new subnet can be added to the mix in a broadband system. The following is a list of items that you should keep in mind when adding new subnets to your network:

• Each new subnet must be advertised on the Internet before it can be of use by subscriber CPEs. If a subnet is not advertised, it will not be valid and will therefore not work.
• Each new subnet must have a minimum of reverse DNS generated for each of the IP addresses provided to CPEs. Private IP addresses are the only exception. Reverse DNS is needed for these CPE IP addresses as many web sites on the Internet perform reverse DNS upon hitting their site. If your IP address does not have a reverse DNS name, it will seem like your Internet service is very slow as the web site your trying to talk to has to time out before you can view the content. What complicates this is that not every web site requires this so it will seem like the service speed is sporadic.
• There are other issues such as routing within your internal network and possibly making updates to your routing protocol (BGP, OSPF, etc.) as needed. These are site specific and may not apply to every broadband operator but it would be bad to find out you need to do this after it begins impacting subscribers. 

During events like network renumbering there certainly are many opportunities to make mistakes. Since a majority of these changes/additions are invoked by manual configuration at 3:00 am there is a need to provide more reliable and less manual operations. The following represent some suggested changes to the provisioning system that could invoke this automation. 

Real-time Additions 

DHCP servers in critical applications should be capable of continuously running. Most carrier-class DHCP servers do not require any periodic maintenance to continue operation however few can withstand operational changes without so much as a hiccup. When changes are loaded or applied to these DHCP servers (e.g. new network added) the server is stopped (all host transaction processing is halted) while the new configuration is loaded and the normal server functionality is restarted. 

In mission critical applications such as a DHCP server handling thousands of hosts, any momentary outage can cause a chain reaction. The resulting pent up demand for servicing of DHCP requests can further delay the startup process of the DHCP server and leading to a potential service outage. To avoid this type of disruption, DHCP servers should be able to withstand “at least” the following changes: 

• Addition of a network
• Addition of a policy
• Addition of a client entry
• Addition of a service class
• Modification of an existing network (everything but name, subnet, subnet mask)*
• Modification of an existing policy (everything but name)*
• Modification of a client entry
• Removal of a client entry

*Modification of these parameters could impact other networks, policies and thus the name (index) and size of the network should not support changes to reduce the possible impact this function could have on other “existing” configurations. 

Seamless Renumbering

A problem with DHCP servers that are used by a broadband operator is that they do not have sufficient available IP address space to operate in a rapidly expanding environment. In this case, an event called renumbering occurs quite often that redistributes IP addresses to a select portion of clients residing in a network that is highly utilized. The renumber candidates undergo a process where by their existing gateway is changed to a newly introduced network. In this new network, their current DHCP lease becomes invalid and they must invoke some surrender (DHCP release) and acquisition (DHCP discover) process to obtain a working lease from the DHCP server. Due to inefficiencies in the renumbering process (mainly the DHCP server as well as the DHCP protocol), this often causes clients to experience more than a momentary outage. In fact, the worst-case outage for a client to recover from the outage caused by a renumbering is the rebind time (or 87% of the lease time – if the rebind time is not explicitly set). During the rebind time, the client performs a broadcast request to the server that is properly handled by their new network’s relay agent to finally allow the client to communicate with the DHCP server. Previous renewals (a type of request that is often sent before a rebind), since they are uni-casts to the DHCP server, will not reach the DHCP server because the client does not have the correct gateway to reach the DHCP server directly. This delay in client reconfiguration can be significantly reduced if client leases were allowed to align themselves with a set renumbering time to permit a near-seamless (near zero outage time) renumbering event. In the future as the availability of DHCP services is rated by uptime, seamless renumbering will be one mechanism that administrators need to reduce customer downtime.

Hardware-Configuration 

Another desirable option could be to configure routers, switches, and CMTS as part of the network renumbering. Since a majority of these changes are configuration only and the fact that these configurations are in-line with those on the DHCP server it stands to reason that the same provisioning application also configure the routers, switches, and CMTS involved in the renumbering. This action would ensure the mapping of broadband transports to networks is maintained and also eliminate human error in making changes to switch and router configurations. Once more, it would provide an additional way to manage the changes on the router, as it all has to go through the provisioning system that often implements a higher level of security than the router, switch, or CMTS.

Can Birds-Eye.Net help you or your Company?
Receive your Birds-Eye.Net articles and white papers hot off the presses by adding our RSS feed to your reader.

(C) Copyright Birds-Eye.Net, All rights reserved.
It is against the law to reproduce this content or any portion of it in any form without the explicit written permission of Birds-Eye Network Services, LLC. Federal copyright law (17 USC 504) makes it illegal, punishable with fines up to $100,000 per violation plus attorney's fees.