Waiting for the big gig: Ethernet at 10 Gbps

For years, people have cited users' insatiable appetite for more and more data as the primary driving factor for our ever-expanding networks. Each time Ethernet rates have been increased by an order of magnitude, users have quickly consumed the newly available bandwidth.

With 1-Gbps Ethernet NICs (network-interface cards) becoming standard on desktop and workstation computers, silicon vendors are poised to once again bring Ethernet to a new level by reaching for 10 Gbps.

Several new standards are under development that promise to make 10 GE (Gigabit Ethernet) mainstream. Even though Ethernet doesn't offer the best-of-class functions that high-speed transactions typically require, the hope is to leverage the economies of scale that Ethernet has always enjoyed because of its wide deployment across many diverse applications, making 10 GE more cost-effective than links such as Fibre Channel. Another cost-saving angle is that using Ethernet reduces design, installation, and maintenance costs associated with having to manage multiple network technologies.

In February, the IEEE complemented its 802.3ae 10 GE fiber-optic-cable-interface standard with the IEEE 802.3ak, or 10Gbase-CX4, standard for copper cables. The 802.3ae fiber standard tends to be expensive because it requires single-mode fiber, specialized connectors, and manual alignment of lasers during installation. The organization has just begun work on the 802.3aq standard for a multimode version of fiber-optic 10 GE targeted to operate at distances longer than 200m. This standard will be able to take advantage of fiber in data centers, avoiding the barrier to entry of having to install single-mode fiber.

Copper is a less expensive and, therefore, more desirable media than fiber. The 10GBase-CX4 standard, however, operates at distances of 10 to 20m, depending upon whom you talk to, limiting the applications that can take advantage of it. Much excitement is building around the 10GBase-T standard, which will define how to run 10 GE over copper cables. Two potential flavors of 10GBase-T target 100m over Category 7 cable and 50m over Category 6 cable. Disagreement exists over whether Category 5 cable can operate reliably transport signals as fast as 10 Gbps over useful distances; for companies that laid Category 5 instead of Categories 6 or 7, 10 GE may not even be an option. Given that the IEEE expects to complete the standard in 2006, you'll need to wait and see.

The technology behind 10 GE is central to network-core applications forming the backbone of networks. To achieve the high volumes that are supposed to decrease the cost of 10 GE, many vendors foresee 10 GE deployment in enterprise networks as 1 GE reaches the desktop, in consumer networks as fiber crosses the curb, and in data centers and storage networks as data consumption continues to rise unchecked.

If the past is any indication, 10 GE promises to change the face of the network as quickly as CDs changed the way you listen to music. This time around, however, is different; in the past, there was always another data format, larger than those in common use, that stressed the available bandwidth to its breaking point. To understand this fact, consider the cardinality of data. "Cardinality" is the number of elements in a given mathematical set. When modems ran at 300 baud, you could forget about sending large data sets, such as images, because text and voice data took up all the available bandwidth. As connection rates increased, so did the cardinality of data that users could send. Images and PowerPoint presentations became types of data that you could reasonably transfer, but streaming-audio files became the data set that bogged the network down. With DSL, audio became data that you can feasibly download. Video downloading is possible at 1 Mbps—although it takes longer to download than to view or consume—and more than feasible at wired 100 Mbps.

With 1 GE, high-quality video becomes more than feasible. Consider FTTH (fiber to the home). Passave, a PON (passive-optical-networking) silicon company, claims sales of 500,000 FTTH ports based on the IEEE 802.3ah EPON (Ethernet-PON) standard. Current deployments in Japan offer 1 GE links to each home at less than twice the cost of DSL. That bandwidth is more than 1000 times that for most users at less than two times the cost. OLT (optical-line terminals) split incoming fiber links using passive splitters; today, each OLT serves 16 homes, but Passave says its boxes can handle as many as 128 splits if you enhance them with FEC (forward-error correction). The PON equivalent of a DSLAM (digital-subscriber-line-access multiplexer) in a central office can deliver data to as many as 5000 homes.

Video currently represents the highest cardinality data—that is, having the highest bandwidth requirement—that home users employ; one movie's bandwidth needs dwarf those of audio data. Assuming that a DVD movie represents a typical video example, one movie equals 4.7 Gbytes, or 37.6 Gbps of data. If a 1 GE link runs at 10% efficiency, a user can download a two-hour movie in 376 seconds—a little more than six minutes—or in about 5% the time it takes to view the content. Broadcast video consumes the same downstream bandwidth regardless of how many nodes are involved. Add a 200-Gbyte hard drive at each home, and, in about four and a half hours, those users can store more than 40 movies with more than 80 hours' worth of content to watch whenever they want.

As downloading time decreases, user consumption of data will increase. Additionally, as latency drops, new real-time services become feasible. However, you simply cannot ignore some practical consumption limits: Even if you stream uncompressed CD-quality audio—at 650 Mbytes an hour, equaling 5.2 Gbps an hour or 124.8 Gbps a day, and at 10% efficiency—it takes only 20 minutes to download 24 hours' worth of content.

The key point is that we appear to have reached our limit of consumption. No common data set today consumes more bandwidth than video. All other common forms of data, including text, image, and audio, are at the level of noise in comparison. Even high-definition video is feasible; in the broadcast world, high-definition video requires only four times the bandwidth of standard video. As a consequence, EPON deployments can serve a great many people with a handful of 1 GE uplinks serving thousands of users. A 10 GE link isn't actually necessary until you get closer to the network core.

Many companies are banking on the belief that video will be the major driving factor for 10 GE. Video in the corporate world, however, has questionable value. Most corporations would prefer that their employees don't watch movies on their company computers, and non-commercial video—corporate announcements, for example—is not a driving factor. Corporations may only require noncommercial video infrequently—once a quarter, for example. Further, it is expensive to produce or is of value to only a small audience, and, thus, the bandwidth to transfer such video is no more than a brief anomaly compared with day-to-day data-traffic needs. For those applications requiring video, such as an online manual showing a mechanic how to install a part, AVI files have become compact and efficient enough to run effectively over 100-Mbps networks. High-quality, high-bandwidth video, then, is either such a rare case or unnecessary in an enterprise that it is not worth building an infrastructure to transport.

Even in commercial networks, video that is not broadcast to a wide audience has limited value. Consider the per-bit value of on-demand video with one receiving node. At $10 for a movie, you can generate about 25 cents in revenue per gigabit of actual data; this figure drops significantly when you consider the efficiency of the network. In addition, you have to build and maintain an extensive infrastructure to get the video to the user, either by having a high bandwidth—and expensive—core network or by placing a dedicated video server closer to the user, such as next to a DSLAM. Compare this scenario with text messaging, which has perhaps the best per-bit value of any data: 10 cents for a 50-character ASCII message yields $250 in revenue per gigabit of data or 1000 times the revenue generated by video. Video's main advantage is that it offers a means for users to more quickly spend more than they might have otherwise; it is easier to watch one movie than it is to send 100 text messages. This fact is one reason that so many companies are interested in networking commercial video.

But just how much data can a person consume? You can watch only so many movies a day. Even the marketing folks e-mailing unwieldy, 100-Mbyte PowerPoint presentations to each other make only a small dent in 1-Gbps of GE bandwidth. Only those people, such as film editors and a few engineers, working with huge data sets can actually actively use this bandwidth.

But even these users consume less bandwidth than you might think. One network manager, describing the network impact of engineers working with large system models, says that these engineers typically download only a component—500 Mbytes to 1 Gbyte—of a larger model, instead of the entire model comprising perhaps 4 Gbytes. These models are collections of hundreds or thousands of files. Taking into account the time it takes to query the model database, check out components, and download them, this transaction requires 30 to 45 minutes to complete. The engineer can then spend the rest of the day or even the week using this data. Even if the engineer is willing to expend half an hour to an hour backing up model data during the day instead of working with the model, the total typical data downloaded is 3 Gbytes. In any case, with the appropriate scripts, users can upload or back up files to the database each night and automatically download them the next day, removing any bandwidth load from the public network during peak hours.

Just because users have a 1 GE pipe doesn't mean they can more quickly consume data. Thus, 10 users with 1 GE links do not need a 10 GE uplink switch to manage their traffic. A handful of 1 GE pipes can serve hundreds to thousands of users, as the FTTH deployments demonstrate. The chief engineering challenge becomes how to create efficient oversubscription mechanisms, not finding more bandwidth. Again, 10 GE finds itself relegated to the network core.

So where might 10 GE come into play other than the core? Data centers and SANs (storage-area networks) provide two possibilities: passive data use and aggregation. An example of passive data use is making a simple change to a database. The change itself might take up only a few bytes, but the change entails revisions of the entire database, resulting in a need to back up the entire database. In cases in which the pipes between the SAN and the backup media are private or independent of the rest of the public-data network, the task of backing up the SAN has no impact on the user/public part of the network.

One truism about storage is that there is always more to store today than there was yesterday. Interestingly enough, the bottleneck for SAN backup is often not the pipe but the capacity of servers to fully use the pipe and the capacity of the backup media, such as a tape drive, to accept the data. If the pipe is the bottleneck because you have enough tape drives and servers, you might question why you are using most of your bandwidth to back up data that you haven't touched in five years. Granted, selective backing up of data is a challenging problem, but instead of continually backing up the same data, it pays dividends to more intelligently approach the problem by, for example, creating a hierarchical-backup schedule.

From an aggregation point of view, even if a person can't consume more than a relative blip of data, thousands of people can. With many users and sources of traffic, congestion becomes one of the most difficult network problems. As data demands approach bandwidth limits, you must implement more complex quality-of-service mechanisms to maintain the real-time characteristics of data and the efficiency of links. Much easier than implementing these mechanisms to handle congestion is to simply increase the bandwidth and eliminate the congestion. Arguably, adding bandwidth can be an expensive way to solve contention issues.

Rising in importance is another significant snag in the higher bandwidth approach to network management: convenience. Corporations are calling for their networks to support wireless users within the office and remote access for users who are telecommuting. This scenario means that users have to have access to the same applications they would in the wired office over much narrower links burdened with VPN (virtual-private-network) and other security overhead. Convenience requirements will actually reduce bandwidth requirements to make wireless and remote connectivity effective and possible. In some respects, enabling convenience is a far more pressing problem than worrying about whether 1 GE pipes will be oversubscribed.

Whenever a user experiences a delay, the network always gets the blame. However, consider a database application in which the user sends a small packet to open the database, perform a query, locate data, download data, and close the database. In object-oriented languages, the levels of abstraction can create tremendous amounts of overhead that can choke a pipe so that the data hardly moves. This situation points out the need not for a faster pipeline but for a new application.

In a similar way, insatiable appetite for bandwidth is no longer growing exponentially. In the past, it has taken data with a higher cardinality—that is, with a bandwidth requirement that is at least an order of magnitude higher than previous requirements—to justify the 10-times jump that Ethernet has traditionally offered with each generation. However, without a need for data with a higher cardinality than video, our consumption of data increases more slowly than we might need to justify 10-Gbps speeds outside the core. Currently, several 1 GE links cost less than one 10 GE link, and they can provide any needed congestion relief, further delaying the necessity for 10 GE links.

Certainly, 10 GE will find a niche as an interswitch connection or link between campuses, although 1 GE seems to be capably serving this need for all but the largest companies. However, if 10 GE is deployed primarily at the core, then there are no high-volume applications to leverage its cost to a more competitive level. One might still argue that a homogenous, single-protocol network is easier and less expensive to manage, but that's a much less compelling reason.

You can reach Contributing Technical Editor Nicholas Cravotta at editor@nicholascravotta.com.