|
Standardized VoIP Testing with GenTest
A database driven approach to regional independent testing
strategy
By: Bruce Bahlmann - Contributing Author (your
feedback
is important to us!)
Created: February 18, 2004
The rollout of PacketCable enabled technology requires
a number of complex components, each of which must be tested as part of a
complete system (as well as individually) in order to ensure flawless
delivery of this CableLabs technology. Because of the sheer number of
components involved with PacketCable as well as their nascent state, such
rollouts encounter numerous firmware upgrades that all must integrate
seamlessly with a growing array of end user devices. With each enhancement
and/or new end user device introduction, comes a need to re-establish and
confirm complete interoperability between the new device (or enhancement)
and the rest of the PacketCable system. This confirmation testing involves
hundreds of tests, is largely manual, and averages between 3-4 weeks to
complete depending on the scope of the change involved.
The GenTest Application Suite (GTAS) was developed
specifically for automating the testing of Voice over Internet Protocol (VoIP)
components involved with the delivery of PacketCable. The development of
GTAS is the direct result of observing manual efforts involved in conducting
various PacketCable lab testing and allowing for the variability that occurs
when conducting different PacketCable testing, at different physical sites,
and with different equipment.
GTAS Component Overview
Diagram 1.0 shows an overview of the major components
of the GTAS. The main drivers of the GTAS are the three databases to the
left. The tests database is a MySQL database housing the various tests
available to be executed. A MySQL database was used to allow new tests to be
added to the database from anywhere on a company’s intranet as well as to
allow different instances of GenTest.exe to always use the master Tests
database. Essentially all instances of GenTest.exe therefore share the same
Tests database. The other two databases are proprietary text-based databases
that are used to configure individual test sequences as well as define the
operational environment of GTAS.
Diagram 1.0
GenTest Application Suite (GTAS) Components
Text databases where selected for simplicity purposes
as well as to allow test sequences and environmental settings to be easily
copied from one instance of GTAS to another. While these databases are in
textual form, they cannot be hand edited. Hand editing either of these text
files (proprietary databases) will cause certain corruption within the
database that could result in partial or complete loss of all test settings.
Do not under any circumstances hand edit the
proprietary databases files.
The GenTest.exe component of the GTAS represents the
heart of the system where all configurations are created, modified, and
where all test sequences are defined. The GenTest.exe uses information from
the Tests database to execute individual tests per the test sequence being
executed. With each test executed, a callout script that is defined within
each test record (along with associated parameters) is executed based on the
test sequence. This callout performs the steps required to complete the test
being executed. Upon completion of the callout, certain parameters are
returned to GenTest.exe which then records the results in a delimited text
file before it proceeds to the next test defined within the test sequence.
The last two components of the system are the Perl
Interface as well as the Ameritec Interface Module. The purpose of the Perl
Interface is to allow GenTest.exe to communicate with other external (and or
third party) hardware or applications. The Ameritec Interface Module exposes
all the command line interface functions to any callout script via a simple
serial interface cable. This interface removes the complexity of
communicating with the Call Generator via its serial interface and allows
very easy execution of tests as well as gathering of associated test
results.
GTAS Test Sequence Review
Before the one can fully understand all the functions
of GenTest.exe an example of the flow of a test is provided to show how the
test is constructed, executed, and how results are generated. In our
example, lets say one wants to execute one test that involves dialing a
seven-digit number from one on-net phone number to another on-net phone
number. Since in reality, a test lab could be set up in a mired of different
ways, using any brand of cable modem with embedded Media Terminal Adapter (MTA),
and assigned any available telephone number – all tests must be able to
accommodate such variability. As such, each test within the Tests SQL
database does not define what phones or phone numbers will be used. Rather,
they just define what callout script to use, which arguments are required to
complete the test, and whether the test can be executed completely
automatically. In terms of our example, here is how this would be defined
within the SQL database:
Test SQL Database:
Test Number =>
G.1.1.1.1
… other test description fields …
Callout => sevendigittest.pl
Arguments => onnet1, onnet2
Test Type => auto
Theoretically, other test types might be executed such
as manual, semi-manual, or other variations of automated testing as desired.
While these test types are not supported within the current release of
GenTest, the capability for such a feature is definitely possible.
GenTest.exe is configured to obtain entries from the Tests database and
handle them in a way that can be uniquely defined. In fact, GenTest supports
any number of test sequences (or series) – each of which can uniquely define
their operational parameters. This demonstrates the power and flexibility of
this completely database driven PacketCable testing suite. An individual
testing sequence may be defined to perform a single test or a group of tests
depending on the specificity of the Test Number defined.
GenTest.exe Test Sequence Configuration
Test Sequence =>
G.1.1.1.1
onnet1 => 5551212
onnet2 => undef
Upon configuration, this test sequence would perform
all test numbers beginning with “G.1.1.1.1”, during which time, any
arguments within those tests matching onnet1 or onnet2 would be assigned the
phone numbers defined above. These arguments would be passed to the callout
script defined by the test record.
sevendigitcall.pl
testfor=call_complete+onnet1=5551212+onnet2=undef
Upon completion of each test, a series of test fields
would be assembled based on how the test sequence was configured. The
resulting output would then be appended to a file and delimited based on the
associated settings within the test sequence configuration.
GenTest.exe Output
G.1.1.1.1|sevendigitcall.pl|5|1|0|
The output of this file may look something like that
above if the test were to be executed successfully. Any argument that is
passed by an executed test record to GenTest.exe but does not have an
associated definition will be labeled as “undef”. If any errors occur during
the joining of the test definition and the test parameters (within
GenTest.exe) the callout will not be executed and GenTest.exe will proceed
to the next test. However in the unique case of a parameter pass, the
callout will not fail as the callout script always has the last word in
terms of what it will accept for valid input and what it will not. If it
does not accept the input, it can generate an error causing this test to be
skipped. Note that you can define many different phone number mappings
within any given test (to satisfy all the different tests to be run within
the set). Any additional parameters defined for a given test sequence but no
specifically used within any given test are simply ignored.
Now that we have described an example flow of a test
execution, we will discuss how to create and manipulate a test using the
GenTest application GUI. The GenTest GUI is essentially a Tk type interface
to the two text databases that drive the testing tool. The main display for
this interface is shown in figure 1.0. Through this interface tests may be
created, edited, and executed. You may also configure other aspects of the
tool like the database access information using the Tool Settings function
and of course Quit the application using a convenient button. Note that
exiting the application using the “X” in the upper right hand corner of the
interface also exits this application cleanly.
Figure 1.0
Main Display of GenTest
If you like the way a particular test is set up, you
can also copy this test easily by selecting Edit Test, changing the name of
the test, and than applying your changes. Such an operation effectively
duplicates a test and gives the copy a new name. Changing the name of a test
copies that test into a new test but leaves the old test untouched. There is
currently no way to remove or delete test sequences.
The MySQL database connection can be activated within
GenTest.exe by configuring its access information within the “Tool Settings”
as shown in Figure 2.0. Suggested names for the database as well as the main
table name where the test records are to be kept can also be defined. For
standardization it is recommended that you use the following:
Database name:
call_features
Table name: tests
Database field names for
table: test
num VARCHAR(100)
NOT NULL PRIMARY KEY
name VARCHAR(100)
objective MEDIUMBLOB
procedure MEDIUMBLOB
expected VARCHAR(100)
callout VARCHAR(100)
arg VARCHAR(100)
testfor VARCHAR(100)
testtype VARCHAR(100)
Note that some of these field lengths may be
excessive depending on the type of test documentation you have and could be
pruned to save database disk space. Use your best judgment to determine what
field lengths work best for your implementation but always give yourself
room to spare.
Apply will attempt to save these settings into the
gentestenv database. Optionally, you can click cancel to abort applying
changes to your previously saved settings.
Figure 2.0
GenTest.exe SQL Database Configurations
Figure 3.0 shows a test create/edit initial
configuration interface. To create a test sequence a number of things must
be known including what you want to name the test, what search string you
would within the database to obtain a listing of tests belonging to this
test sequence. A test sequence can consist of a single test or multiple
tests. Use the “Test series” and “Test type” fields to define the number of
test records that will match your series. For example if you use “G.1” for
Test series and “auto” this would request all tests from the tests database
whose test numbers begin with “G.1” and are of type “auto”. The more
definitive you get with the values of these fields the less tests this
sequence will represent. The less definitive you get with these values, the
more tests this sequence will represent.
The output filename and delimiter describe the name of
the output file as well as how it will be delimited. Currently the output
for GenTest.exe is primarily text but this information can easily be
imported into a spreadsheet for analysis.
Figure 3.0
GenTest.exe Test Modification Area
The “Associated” fields area defines certain important
relationships that have been configured within this particular test
sequence. The phones represent phones that have been created and associated
with this test sequence. Similarly, the test fields represent the particular
test fields associated with this test sequence. Through this interface you
can edit these test fields or phone associations or cancel the modification
or creation process. It is important to know that when creating or editing a
test, the changes will not be applied until you select “Apply Changes”. Any
other changes made prior to this final step will be lost if you forget this
step. If you have made any changes to a test sequence, the Cancel button
does not become an option – however if you still want to cancel at this
point, your only option is to quit the application (clicking the X in the
upper right hand corner of the interface window).
NOTE: The test fields, while fully implemented
within the graphical user interface, are not fully implemented within the
output.
If you select the “Test Fields” button on GenTest.exe
it will take you to the interface shown in Figure 4.0. Although this area is
not implemented within the output, it is fully configurable within each test
setup. The purpose of this area is to allow the user to configure various
fields that either needs to show up in the output or as part of a manual or
semi-manual interaction with the GenTest.exe application. While neither
manual nor semi-manual modes are currently supported, there is no question
that such modes will be a necessary part of any packetcable testing. Both
manual and semi-manual allow tests to include “subjective” results to be
captured, interpreted, and entered. To support this mode, fields can be
created, edited, associated, and unassociated with a particular test
sequence.
Figure 4.0
Test Fields Management Area
Note that once a field has been created, it becomes
available (but not associated) to all test sequences and can be easily
associated with each test sequence without having to create it again.
Creating or editing a field bring up the same interface. The only difference
is that when you create a field, it is done with the intention to make
something where as when you edit something it is done with the intention to
modify an existing parameter. If you create a field and associated with
multiple test sequences, any field properties are only defined ONCE.
Meaning, if you modify this field, the field’s behavior in all test
sequences will be modified.
Associating a field essentially attaches that field’s
properties to the test sequence. When this is fully implemented, the net
result would be that this field is either displayed or is prompted for user
input. UnAssociating a field simply removes that field from being affiliated
with the test sequence. Upon any changes to fields, you will need to select
Done. This function may pop up another dialog that allows you to order these
fields as they would be displayed on a manual or semi-manual interface.
Ordering these fields is optional but as this capability is fully supported
it may make sense to order them as you follow along with your testing.
Figure 5.0 shows an area allowing test fields to be
created or edited. This would only be required for new tests that require
manual or semi-manual interaction. More details about how all these fields
must be entered as well as how they will work will come in future releases
of GenTest.exe as this manual and semi-manual functionality is fully
supported. Until that time, this capability only exists within the
GenTest.exe GUI.
Figure 5.0
Test Field Create/Edit Area
Mapping phones is a very critical and fully implemented
function that exists within GenTest.exe and is shown in Figure 6.0. The
purpose of Map Phones area is to permit the tests defined within the SQL
database to be executed at different test sites. Since each test site has
its own hardware (e.g. Media Terminal Adapters) that is relatively fixed,
this feature allows phone numbers and MTA modem MAC addresses to be
addressed relative to the site as opposed to the central database.
As a result, when tests are run relative to a
particular site, they are able to take on the characteristics of that site
(phone numbers) without any changes to the test records or the test scripts.
This is the fundamental principle of what GenTest.exe does – it generacizes
test sites so that they can run common tests.
Using this interface, individual phones can be created,
and than mapped to specific test arguments related to the tests defined
within the test sequence. Currently each phone must be individually mapped
to each test argument defined within the test sequence. Although this is
cumbersome, it does work very well and so long as the test sequences do a
good job of re-using phones you should not be required to remap phones
(shuffle phone numbers among defined phone arguments). Rather, if such an
exercise is necessary, simply create a copy of this test and then perform a
new phone remapping exercise. Like the test fields, phone number records
that have been created once can be used by all active test sequences and any
changes made to an existing phone record will be applied to all test
sequences.
Figure 6.0
Phone Mapping Area
Associating a phone can be accomplished by selecting
the phone on the list to the left and than clicking Associated Phone. This
will require you to select which argument must be associated with that
phone. UnAssociating a phone simply removes this relationship within the
test sequence.
Note that you must create enough phones to handle
all the ports you plan to use on your call generator. If not, some of the
tests you define within your test sequence will fail due to a lack of
sufficient arguments.
To create or edit a phone record, simply click Create
or Edit Phone. This will take you to the interface shown in Figure 7.0. Use
this interface to define the name of a phone used within your lab and
associated a phone number and MAC address with that phone. Currently the MAC
address is not used but in the future it could be used to perhaps generate a
report of all phones used or perhaps look up certain records on external
applications or hardware.
It is recommended that you use very short phone record
names but something that describes where they reside (what state/city/lab)
and what kind of modem they are configured on. Once phones have been
defined, you click “Apply” to save the phone record or cancel to abort this
operation.
Figure 7.0
Phone Definition Area
Upon successful phone creation, mapping, and test
sequence saving, you are ready to run the test sequence you just created –
provided that your database is properly setup with tests. Test sequences are
executed from the Main Display (see Figure 1.0) by simply selecting the test
sequence from the list of previously created tests and than clicking
“Execute Test”.
Note to make sure that your PC is connected via
serial cable to your active call generator. GenTest.exe requires serial
communications with your call generator to perform tests defined within the
tests database.
When you execute a test, a number of things happen
locally on your PC. One noticeably thing that happens is a command window
gets created over the GenTest.exe GUI. Another thing that happens is that
GenTest.exe contacts the SQL database and returns a list of tests that match
the criteria defined within the test sequence. If this list is greater than
0 (zero), the GenTest.exe application proceeds with executing each test
based on the defined parameters within the SQL database. All tests that were
successfully executed (whether they pass or fail) have their results
recorded in the output file defined within the test sequence. This file is
appended be the completion of each test and will be further added on by
subsequent execution of other test sequences with similar output file names.
Upon the completion of any test it is recommended that you rename the output
file if the results are to be used for later analysis. If results are not
being used for later analysis, this file can just be deleted without causing
any problems for future runs of GenTest.exe – which will just create a new
file if one does not already exist.
Figure 8.0 shows the results of an executed test
sequence. At this point, all the tests retrieved from the SQL database have
been attempted and some general accounting of these executed tests have been
completed. The results of the tests are displayed on the Main Display to
indicate the overall success of the completed test sequence.
Figure 8.0
Summarized Test Results
For example, this summary information shows that two
tests were attempted as part of test sequence “comcast-1” and of those two
tests attempted one test passed and one test was not attempted for some
reason.
Additional guidance is needed regarding error messaging
when tests are for some reason not attempted or for some reason fail. For
this reason, the GenTest.exe application is deemed a prototype but for all
practical purposes it is a fully working application that can test any
number of defined tests.
The potential of this application within a large voice
over IP provider cannot be fully understood until the realities of
performing ongoing testing within a multi-vendor deployment are experienced
first hand. Execution of comprehensive call feature testing represents a 2-3
week effort for 2-3 seasoned engineers. This testing must be executed on all
new MTAs added to the deployed system. Additionally, operators may want to
perform confirmation testing on all MTA firmware upgrades just to be sure.
This level of effort represents an on-going need for automated testing and
call feature testing is only the beginning of the testing required of a
PacketCable deployment.
Can Birds-Eye.Net help you or your Company?
Receive your Birds-Eye.Net articles and white
papers hot off
the presses by adding our RSS feed to your reader.
|