Broadband Network Management Requirements for High Speed Data Internet Service

By: Bruce Bahlmann - Contributing Author (your feedback is important to us!)

Created: December 6, 1999

Note: For help designing your cable modem network management methods and procedures or developing tools to help you improve or implement such a program contact Birds-Eye.Net.

1.1 Overview:

The monitoring tools used today for high-speed data (HSD) Internet information service are nearly the same as they were 3 years ago. In addition, the deployment of Data over Cable Service Interface Specification (DOCSIS) introduced new monitoring requirements and complexity. The goal of this document is to develop a list of requirements which regions feel will enable them to effectively monitor the HSD service. These requirements are not meant to specify a full-fledged network management system (e.g. for video, telephony, etc.) – only requirements needed to monitor HSD services. For details regarding specific regional comments and/or HSD monitoring systems in use refer to the Notes from conversations with regional NOC staff document. Regional specific feedback included in these requirements are signified by the following:

1.2 Background:

Monitoring is a means of gathering and examining information about something in an effort to maintain service quality/reliability. The frequency and scope of information gathered depends greatly on what is being monitored and the way it is being monitored. There are several different ways of monitoring something including:

• Watching for change (improvement or deterioration) in something, perhaps beyond certain thresholds
• Watching for the absence or presence of something
• Watching for some specific event or sequence of events to take place

Determining what to monitor requires an understanding of the "system" and what collection of items (or network elements) provides the best visibility into the inter-workings (or operations) of the system. Fault, Configuration, Accounting, Performance, and Security (FCAPS) represent a network management model used industry-wide to provide good visibility into the operation of most services. The FCAPS model (whose elements are defined in Table 1.0) recommends particular network management areas of focus that would enable one to maintain good service quality/reliability.

Table 1.0 FCAPS Network Management Model

As a result of monitoring areas of focus (such as those in FCAPS) in the ways discussed previously, one might hope to trigger one of the following:

• Proactive response – Information that would enable something that is deteriorating to be detected thereby allowing repairs to be made prior to failing thus eliminating potential service interruption.
• Reactive response – Information that would enable something that is now broken to be detected thereby allowing repairs to commence independently of customer service calls.

The responses above are dependent on the proper configuration of all network elements within the network management visibility. If any network elements are not properly configured an erroneous response (or the lack there of) could result. The proper configuration of individual network elements enables higher-level network management to occur. Higher-level network management forms relationships between two or more triggered events allowing one to identify a probable cause – which may well be something that has not yet triggered. This level of analysis requires sophisticated network management software and, most importantly, extensive knowledge and experience of all network elements within the network management visibility.

The importance of FCAPS as it relates to HSD is that it provides the network management areas of focus required to sufficiently monitor the service.

1.3 HSD Monitoring Architecture:

There are several components of the HSD delivery system, which are critical to providing reliable service -- some of them are under our control (plant, headend) and some are not (customer’s home, servers, regional network, etc.). Before one can monitor HSD (or establish requirements for monitoring), they must evaluate various points in the system where service interruptions can occur and prioritize the effort that must be spent to effectively monitor it. The components of the HSD system (ranked in order of small to large in terms of the scale at which it may effect customers) are as follows:

1. Customer premise
2. Node segment
3. Node wide
4. Network
5. Server
6. Third party (not location specific)

Figure 1.0 describes the delivery system for HSD indicating possible locations of the service interruptions described above (for a more detailed explanation of HSD service interruptions refer to the Guide to HSD Service Interruptions document). The boxes drawn around groups of network elements represent their respective ownership.

Figure 1.0 High Speed Data Service System

If one were able to monitor in such a way that all these components were covered, the result would be comprehensive network management of the HSD service. However, today there does not exist any one-network management system that will provide this comprehensive network management. Therefore, we must evaluate the service interruptions above and continually focus our attention on progressively covering each component of the HSD service system until such time as all sources of service interruption are addressed (visible) by network management.

2.0 Goals:

The primary goal of this document is to derive a list of requirements needed to sufficiently monitor HSD services. Providing this visibility benefits other services such as analog video, pay-per-view (PPV), digital video, and telephony. This benefit results from all services sharing the same medium as well as the fact that what effects one service (on a shared medium) often effects other services. Requirements for monitoring HSD services have the following goals:

• Focus on areas that are single points of failure and could effect large numbers of customers -- Certain areas have built-in redundancy and may not require immediate attention.
• Monitoring requirements "should" be convergent services centric and inter-operate with existing network management system (don’t want a separate monitoring system for every service).
• Try to buy off the shelf – do not want any custom built system – too many canned applications available – keep it simple [Atl]
• Monitoring system should meet all the requirements of the customers of this system – These customers include Tier 1, Tier 2, field operations, plant operations, NOC, management, individual subscribers, and also corporate.

NOTE these goals may be satisfied in a phased fashion depending on relative priorities of the monitoring requirements and the availability of commercially software that meets a majority of these requirements. If a phased project is accepted, requirements with highest priority will be addressed first.

3.0 Requirements:

Requirements are organized in the table below in no particular order. Each requirement is stated in terms of its associated area, relative priority, phase in the project, and any assumptions that must be noted. Priorities noted from (0-critical, 1-high, 2-medium, 3-low). 

4.1 Customer premise monitoring:

The information gained from sampling customer cable modems in a specific area can provide us with information regarding the health of the plant in that area (see #1 in Figure 1.0). Unfortunately, unless this is done right it can also lead to inconclusive information. Attempts to use customer devices (i.e. customer cable modems or telephony NIUs) as a means of detecting partial node outages greatly depends on a high level of penetration of these services on fiber nodes. If we do wish to monitor the customer premise we should be aware of the following tradeoffs:

• Customer premise monitoring allows us to utilize customer purchased devices (no additional hardware) as well as the same network management station used to poll other HSD resources in exchange for basic node outage detection.
• Coverage for all nodes may be years away. Take a city of 30,000 with a penetration of 5% for HSD and 30% on telephony in an area where the max customers per node is 250. A total of 10,500 customers would have to be polled to determine partial node service interruptions on the city’s 120 nodes. This would mean that approximately 88 subscribers per node would provide sufficient information to enable us to detect partial node outages. While this is reasonable to expect good coverage across the whole node – most regions are no where near these numbers for both services (HSD and telephony).
• Correlating polling information with physical location so that one can draw conclusions from devices on similar stretches of plant is also a problem. Polling customer modems in an area is further complicated in the DOCSIS model due to combining. This is because customers with the same network address or CMTS interface may all be on separate fiber nodes. Thus any polling that is done among them "may" not be usable For example, two down subscriber devices could mean that two node segments are down or only one is down (depends on how the plant runs through the community).
• There is also an issue regarding the interval of polling. For example, if multiple modems were polled the period of time between modems geographically close to one another may vary. If this time varies too much the samples taken from the modems would have no relationship to one another.
• There is also the subject of scalability in terms of polling an increasing number of NIUs and cable modems to determine partial node outages. Since this number is directly proportional to the number of customers it will require an increasing amount of resources to manage this system.

4.2 Monitoring standardization, coordination, and information sharing:

Ideally it would be nice if one could proactively fix everything such that customers would never experience a service interruption. Experience with equipment is one of the best ways that one can increase the number of "potential" problems that can be detected and then proactively fixed. This requires time and information sharing such that as new information is learned about potential problems it can be distributed to others so all can benefit. However, lightning does strike, things fail, and unknown problems develop. In these cases one must resort to a reactive response where one must locate and fix the cause of the service interruption as quickly as possible. Reviewing events leading up to reactive responses (during post-mortem meetings) allows network management staff to begin to understand "potential" causes of these service interruptions. These causes can then either be modeled in a lab for further examination or added to the list of things monitored in an attempt to detect this event in the future (hopefully prior to its "projected" outcome – a service interruption). An MSO's network management organization could greatly benefit from standardization, coordination, and sharing of information. Regions should align themselves with a standardized set of items they monitor in terms of each service. Each of these standard items would be well tested and understood (i.e. reliable means of monitoring). Additional monitoring items should be coordinated via national NOC such that all "new" items under development (or in testing stages) are tracked for degree of success. When an item has proven some degree of success it should be shared with other regions to enable more extensive testing/refinement. Perhaps after some qualification period it is added to the standardized list of items. The emphasis should be to provide each region with the same visibility of service monitoring. If a region does not have the staff to explore additional monitoring items (via coordination from national NOC), it will at least maintain the standard (but growing) list of items maintained by the national NOC. A decision should be made to coordinate the deployment and on going maintenance of a standardized HSD monitoring system. If a standardized system is not selected the size and scope of the project will increase.

4.3 Exposure to fiber cuts:

Regardless of our ability to manage equipment failures, one area where we are completely exposed is fiber cuts (see #3 in Figure 1.0). Often, many runs of fiber from the hub to the subscriber community traverses areas inaccessible to standard equipped plant operations personnel (i.e. underground wire or bridge crossing) or inaccessible due to some other reason (i.e. busy intersection). This fiber is also a single point of failure – meaning it is not redundant. These matters may be further complicated if a cut (break) occurs at night, poor visibility (fog, rain, snow, etc.), or when there is limited staff available to help find the break (i.e. visual examination is the current means of finding breaks). During these times service interruptions can be quite lengthy, as there is no means to direct service personnel to the specific location of the break. One possible solution would be to place an Optical Time Domain Reflectometer (OTDR) on fiber runs where we are sufficiently exposed. If we combine OTDR with GPS tracking of the route taken by the fiber run, it would allow network operations personnel to dispatch service personnel to the exact location of the break and speed repairs by as much as 50% (or the time it takes to locate the break). This item is not currently recognized as a requirement for HSD monitoring and should be considered if any standardized monitoring strategy is adopted. If this issue is not addressed as part of seeking an HSD monitoring system, one should at least devote resources to a comprehensive study of its impact and possible worst case exposure. The study would seek to determine the costs associated with fiber cuts in terms of loss of customer confidence, negative publicity, etc. Perhaps results would impact the way new cable systems are designed to minimize this type of exposure.

4.4 End of line monitoring:

Because customer devices will never represent 100% of the end of line for each and every fiber node we need to seek a more reliable measure of this important operational aspect. Knowing the status of ALL end of lines replaces other incomplete monitoring systems and this should be one area we spend much effort. End of line monitoring also represents a fixed number of devices that can be modeled by location. In addition, end of line monitoring would force a rework of the way we label nodes which would allow us to reference specific portions of a node rather than the existing system which keys on various actives (i.e. amplifiers) along the node. A decision is required on whether to include end of line monitoring as part of HSD monitoring system effort. The inclusion of end of line monitoring would increase the size and scope of the monitoring project with the benefit of providing unprecedented HFC monitoring reliability. This increase takes into account additional development and hardware deployment.

4.5 Learning and Development (L&D) involvement:

Regions spend an increasing amount of resources training new and existing staff. This training may range from an orientation for new hires on network management basics and systems to updates on new techniques or processes and can be expensive. For example, the region may tie up their best people (or at least the ones most able to explain or teach the systems in use) and/or incur costs associated with sending employees to off site training. These costs could be offset if L&D was a stakeholder in the handoff process of deploying a network management system. L&D could then obtain all necessary materials and documentation for regions to train NOC staff within their own learning centers. L&D could also be responsible for updating materials and its training with that supplied by the vendor. Inclusion of L&D requires a commitment from regions to work with this organization to supply its training on network management. This decision should be made early on in the project to ensure that timelines will be met and necessary resources within the L&D organization are allocated.

4.6 Information Technology (IT) involvement:

Regional support for business critical computer hardware/software typically comes from IT. Regional IT is staffed with sufficient expertise to build and maintain business critical systems. HSD monitoring is one such system that demands IT involvement to ensure it is maintained properly (i.e. OS updates are installed, applications and data are backed up regularly, user accounts are administered, security is closely guarded, etc.). Regional NOC staff are ill equipped to deal with day-to-day regional network management operations along with administering their equipment. It is also difficult to hire and keep individuals with this kind of skill set. Thus a decision to work with regional IT is required early on. Any changes in the current set of applications already supported by regional IT needs to be approved. Regional IT may also place additional requirements on the hardware or operating system platform they prefer to support. Delays in regional involvement may lengthen deployment.

5.0 References:

1. "Guide to HSD Service Interruptions", Revision 0.20, Bruce Bahlmann, 01 December 1999.
2. "Notes from conversations with regional NOC staff", Bruce Bahlmann, 20 December 1999.
3. "Centralizing Network Management", a presentation by Don Williams, 12 December 1999.

  Can Birds-Eye.Net help you or your Company?
Receive your Birds-Eye.Net articles and white papers hot off the presses by adding our RSS feed to your reader.

(C) Copyright Birds-Eye.Net, All rights reserved.
It is against the law to reproduce this content or any portion of it in any form without the explicit written permission of Birds-Eye Network Services, LLC. Federal copyright law (17 USC 504) makes it illegal, punishable with fines up to $100,000 per violation plus attorney's fees.