"Triple play" is so yesterday: how to test voice, data, and video multiplay services

READ OTHER APRIL ARTICLES:  Contents, April 2007

Regardless of how many “plays” service providers offer, video is the most demanding service that Internet Protocol (IP) based networks must deliver. As IP video (TV and games) is deployed, service providers need to test their network configurations. That’s where Juniper Networks’ Westford lab plays a vital role. The lab consists of several test beds, one of which is an entire IPTV network containing everything from video sources to TV screens. Consulting engineer Chandra Shekhar Pandey administers this network, called the IPTV and Multiplay Center of Excellence.

Communications providers (Juniper’s customers) come to the lab to learn how to configure their networks to deploy IPTV. They rely on Pandey to help with network architecture, configuration, and deployment, because his network contains the entire video chain. Pandey also works with Juniper’s engineering teams who need access to the test bed in order to gain a better understanding of the user experience.

Chandra Shekhar Pandey works with service providers (customers) and Juniper engineering teams for design and validation in the IPTV and Multiplay Center of Excellence.

In contrast, most IPTV networks send only the channels to a residential gateway or STB to which a customer has subscribed, and they send just one channel at a time. When selecting a channel, the user essentially sends a command to the network, which responds by transmitting the requested channel. Thus, it’s challenging for IPTV providers to give customers a channel-surfing experience that emulates cable.

In addition, whenever an IPTV provider adds a new customer, the additional load on the network must not cause degradation of service to previous subscribers. Provider networks must also deliver uninterrupted services even when under denial-of-service attack or other security threats.

As Eric Jenkins, system test manager of the company’s Interface Group, observed, “Chandra’s lab is a representation of the real world.” Juniper’s Center of Excellence contains the company’s core routers, metro routers, security appliances, and edge routers, as well as network elements from other manufacturers, such as DSLAMs, fiber-to-the premises (FTTP) equipment, DSL modems, residential gateways, middleware, and video servers (Figure 1). The center also shares lab space with other test beds used to verify new and updated products.

Figure 1. Juniper’s IPTV and Multiplay Center of Excellence network represents a complete service-provider’s network, from video sources to TV screens.

When they evaluate their systems at the Center of Excellence, service providers can focus on all or part of the network. For example, they may concentrate on testing a Juniper core-network router or edge router only. Or, they may concentrate on the overall service delivery. They also look to the center’s staff for advice on how to design and configure their network to deliver next-generation services.

The Center of Excellence network also consists of live video sources such as a satellite receiver and a video server. The live feed goes to an encoder that encapsulates the feed into MPEG-2 and H.264 (MPEG-4 Part 10) over IP for transport. The network also contains video-on-demand (VoD) servers and “middleware” servers that contain software applications that run over the network. A session-initiation protocol (SIP) server adds voice calls to the network. A Juniper application accelerator accelerates HTTP (Web services) on the network.

Redundant Juniper video-provider edge routers focus the data into Gigabit Ethernet (GigE) streams that feed a redundant pair of Juniper core routers. The routers, in turn, transport 10-GigE streams to a pair of metro routers.

The redundant routers and links let engineers test the network should one router or interconnecting link fail. Service providers often place redundant routers in cities hundreds of miles apart so that if a natural disaster destroys on of the routers, customers won’t lose service.

Metro routers send IP packets to a Juniper edge router, which contains subscriber information such as the services to which the customer has subscribed. A GigE switch in front of the edge router can aggregate lightly loaded DSLAMs or FTTP equipment, or uplinks can directly connect to the edge router. Dual links to edge routers provide link-level redundancy.

Eric Jenkins leads a team of engineers that evaluate data forwarding and encapsulations in software.

Pandey’s network also contains test equipment that generates and monitors IPTV traffic. Several Agilent Technologies multi-services testers generate traffic equivalent to 500 channels (some are audio only), and the testers simulate the equivalent loads of thousands of subscribers. They also monitor bandwidth usage and can tell engineers when the network is reaching overload and needs more capacity.

The center’s engineers can monitor video quality with remote video over IP monitoring systems from IneoQuest. The monitoring systems provide a score called a media delivery index that represents video quality. The IPTV multicast traffic generators are located at the head end’s GigE switch.

The monitors let Pandey compare video quality and packet loss across the core, metro, and edge routers. The systems communicate with network-monitoring software from Hewlett-Packard. Data from the IPTV monitors help the monitoring software decide if a service provider is delivering the subscribed level of service.

The Westford lab also contains test beds that engineers use to stress routers beyond their design specifications. Engineers such as Chris Fernandes test routers and line cards for basic functionality. “IPTV,VoIP, and gaming are layered on top of data services,” said Jenkins of the company’s Interface Group. “There must be no degradation of service, even if a router is handling thousands of subscribers. Subscribers expect the same experience they’re used to even though the delivery mechanism has changed.”

During a test, Pandey and others will force an active router to fail. They observe forwarded traffic to check how the remaining network elements perform. “The network must recover from this failure by rerouting traffic within 300 ms, or it will affect the user’s experience,” said Pandey.

The Center of Excellence does more than simply let Juniper’s customers configure and test proposed deployments. It also provides Pandey with valuable information that he feeds to the engineering team for product enhancements and new product ideas.

When design engineers incorporate new features into a design, they hand the new product to the engineers in Jenkins’ group. These 11 engineers test the forwarding and encapsulation functionality of new designs—two essential functions of IP routers. They also run regression tests on new software to verify that it remains compatible with older revisions.

One important test covers channel-changing time. Juniper engineers use a “channel zapping” tool to initiate and time stamp channel changes. Because IP networks carry MPEG-encoded video streams, which periodically send I-frames followed by partial (B/P) frames, a change in channel doesn’t appear on the subscriber’s TV screen until it receives a full frame, called an I-frame.

When Juniper engineers began measuring channel change time, they used timing software on the Agilent multi-services testers. Now, the time stamping takes place in hardware. “We sometimes have to work closely with test-equipment manufacturers to get the test equipment designed right to meet the scalability and functionality we and our customers need,” said Pandey.

After engineers in Jenkins’ group test data forwarding and encapsulations, they hand products to a group headed by software quality assurance (SQA) manager Clyde Neville. This group consists of eight engineers in Westford and six in India who perform functional, system, scaling, negative, interoperability, and call-setup performance tests.

Clyde Neville leads a team that evaluates routers for software applications and protocols.

For interoperability tests, the lab contains a test bed with routers from Cisco Systems, Foundry Networks, Nortel Networks, and Unisphere Networks (Unisphere became part of Juniper in 2002, which is how Juniper entered the edge-router market). At the time of my visit, Neville’s group was focusing on interoperability testing for a new revision of Juniper’s JUNOS operating system for multiservice and core routers and its JUNOSe for broadband services routers or edge routers.

A typical interoperability test setup consists of two routers and a client generator made by Spirent Communications. “Customers want to make sure that a protocol they develop for another router will also work on a Juniper router,” commented Neville.

Neville’s group is responsible for proving that if any part of a router fails, the router will keep the data flowing. “We yank a line card out of a running router to verify that the system keeps running,” he explained. “We also perform boundary testing. That is, if a router is rated to handle 96,000 connections, we’ll exceed that number to ensure that the router won’t crash.”

Figure 2. Engineers test routers for performance while carrying applications encapsulated in Ethernet packets.

When engineers in Neville’s group don’t need the full test bed, they use a test bed that performed interoperability tests before they obtained the protocol tester. The test bed consists of a router set up to run as a client for the router under test. Windows and Linux-based servers provide content for the router. The test bed also contains a Dynamic Host Control Protocol (DHCP) server that lets engineers test routers using DHCP-v4 and DHCP-v6 protocols.

Neville’s group also runs scaling and integration tests on every release to verify current functionality doesn’t negatively affect any product. Using a test bed that’s a duplicate of one that Jenkins’ group uses, Neville’s group runs a series of automated scripts to test routers. Juniper has another identical test bed in India.

Regression tests typically run 24 hours a day. “We have over 100,000 scripts for running regression tests,” Neville explained. “The number of scripts we use grows with each software release. To test all of Juniper’s products, we have hundreds of thousands of scripts.”

Throughout my visit, I heard Juniper engineers talk about working with test-equipment manufacturers to get complete test tools. Today, test equipment can simulate more than 500 channels of content and can monitor transmissions for protocol violations. Video-quality testers can produce scores the engineers can use to rate video quality.

“Test equipment is coming into its own,” noted Pandey. “We’ve worked with test-equipment makers to get the features we need.” Juniper VP of engineering Chris Vaughn pointed out that Juniper’s test-equipment needs are constantly outstripping the capabilities of commercial test equipment.

Jenkins echoed that sentiment, noting that “not many test vendors can support 44 ports on one line card. Test equipment is trailing our needs to support 96,000 interfaces [subscribers] on one router.”

The testing that takes place at Juniper Networks gives service providers confidence that the networks they develop and deploy will deliver data, voice, video, and games at a quality level that subscribers expect. Testing will only increase in importance as service providers add HDTV to their offerings. HDTV’s higher bandwidth will further stress networks, but users won’t tolerate any degradation in service.