Major challenges facing the scientific R&E community today involve insatiable needs for communications and collaborations at distance, as well as the ability to manage globally distributed computing power and data storage resources. Advances in telecommunications and networking technologies are up to the challenge of meeting these ever increasing demands for bandwidth.

Not long ago, it was common to find large disparities between the connection speeds of end systems to the local area network versus the speed of the shared wide area network. Fortunately, that is no longer common and instead we find end systems connected at speeds of 100 Mb/s to 1 Gb/s and wide area links operating at 1 Gb/s to multiples of 10 Gb/s. Many factors have contributed to the technology boom that resulted in wide area speeds becoming attainable. The dot com boom saw a huge build-out of infrastructure and technology only to have the bottom drop out of the marketplace. Many R&E organizations have taken advantage of the marketplace to acquire access to dark fiber and build customer owned and operated metropolitan, regional and wide-area infrastructures. This has put the fate of bandwidth availability squarely within the control of the end customers rather than the service providers and carriers. The result is a strategic opportunity for the R&E community.

Making multi-gigabit/second, end-to-end network performance achievable will lead to new models for how research and business are conducted. Scientists will be empowered to form virtual organizations on a global scale, sharing information and data in flexible ways, expanding their collective computing and data resources. These capabilities are vital for projects on the cutting edge of science and engineering, in data intensive fields such as particle physics, astronomy, bioinformatics, global climate modeling, geosciences, fusion, and neutron science.

This article addresses Ethernet advances leading to network convergence, end-to-end performance factors, and highlights examples of 10G early adoption, deployment, and wide-area infrastructure availability in the R&E community.

We are witnessing the convergence of LAN/MAN/WAN data rates at 10 Gb/s, resulting in common equipment and interfaces to access the enterprise, metro and wide area network. As a result, we are experiencing reductions in the cost for implementation, ownership, support, maintenance and in some cases, recurring charges in the WAN.

Ethernet technology is currently the most deployed technology for high-performance LAN environments. Enterprises around the world have invested in cabling, equipment, processes, and training in Ethernet. In addition, the ubiquity of Ethernet keeps its costs low, and with each deployment of next-generation Ethernet technology, deployment costs have trended downward. Ethernet has gained worldwide acceptance as an enterprise infrastructure technology due to its considerable advantages in interoperability, scalability, simplicity, consistency, service ubiquity, provisioning speed, and price/performance. With the rapid price decline in Gigabit Ethernet network interface cards, most servers come standard with Gigabit Ethernet network interface cards.

An Ethernet infrastructure supporting traditional networking applications, network storage, and clustering, enables greater compute density and physical consolidation of resources. Industry standardization of infrastructure components offers economies of scale that drive down deployment and management costs and leverage common training requirements. The simplified infrastructure reduces inventory and maintenance costs. Reconfigurable components offer the flexibility to make on-demand infrastructure changes.

A natural fit for 10G Ethernet technology is in the scalable uplink from the data center switches that connect server farms with 1G Ethernet interfaces. The price per gigabit for 10G Ethernet is projected to be 40% lower than for Gigabit Ethernet. As an open-standards-based, forward- and backward-compatible technology, Ethernet has been broadly adopted and understood by engineers worldwide. So instead of building networks of increasing complexity, and facing the increasingly more difficult task of finding engineers trained to manage them, enterprises and service providers can leverage the simple and familiar infrastructure that is second nature to a large majority of network engineers.

10G Ethernet also meets several criteria for efficient and effective high-speed network performance, which makes it a natural choice for expanding, extending, and upgrading existing Ethernet networks: A customer’s existing Ethernet infrastructure is easily interoperable with 10G. Existing Ethernet standards, such as 802.1q for virtual LANs, 802.1p for traffic prioritization and 802.3ad for link aggregation, also apply to 10G Ethernet, making the deployment of 10G Ethernet simply plug-and-play for most enterprises and service providers. The new technology provides lower cost of ownership including both acquisition and support costs versus current alternative technologies. Using processes, protocols, and management tools already deployed in the management infrastructure, 10G draws on familiar management tools and a common skills base. Multi-vendor sources of standards-based products provide proven interoperability.

In the metropolitan area network (MAN) the predominant leader continues to be 1Gb/s Ethernet; however, 10 Gb/s metro systems are beginning to emerge as prices of optical components continue to drop. 10 Gigabit Ethernet is on the roadmap to enable cost-effective, Gigabit-level connections between customer access gear and service provider POPs in native Ethernet format, simple, high-speed, low-cost access to the metropolitan optical infrastructure, metropolitan-based campus interconnection over dark fiber, targeting distances of 10 to 40 km, and end-to-end optical networks with common management systems. While there are pockets of new MANs, many locations are still waiting for better market conditions. MANs implemented using Ethernet technology are inexpensive and ideal for seamlessly interconnecting distributed Ethernet LANs because they require no protocol conversion.

The IEEE 802.3ae 10G standard, ratified in mid 2002, features an interface speed at 10 Gb/s at the media access layer, along with two families of Physical Layer Specifications (PHY): LAN PHY operating at 10 Gb/s and WAN PHY operating at 9.29 Gb/s compatible with the payload of OC-192c/SDH. The 10G standard not only increases the speed of Ethernet from 1 Gb/s to 10 Gb/s, but also extends its interconnectivity and its operating distance up to 40 km. Using 10G WAN PHY allows service providers to use the installed-base of SONET Layer 1 transport gear to provision 10G Ethernet traffic. Because the 10G Ethernet WAN PHY avoids the costly aspects of the traditional SONET, such as stringent grid laser specifications, jitter requirements and stratum clocking, it offers a compelling alternative to traditional SONET interfaces with better price/performance. The ability to send Ethernet directly from an Ethernet switch over a WAN PHY link eliminates the need for expensive Packet over SONET router interfaces.

It's important to note that 10G LAN systems offer fewer alarms and indicators than 10G WAN systems. Unlike on the WAN side, there are currently no explicit standards that prescribe techniques for carrying forward error correction (FEC) to extend the reach of 10G LAN PHY. As a result, equipment manufacturers are developing proprietary interfaces to carry 10G LAN PHY with FEC for metro applications. This lack of explicit standard raises the question of interoperability and third party testing. Efforts are under way to develop standards that will make Ethernet services “carrier class” by incorporating operations, administration, and maintenance capabilities.

Before 10G Ethernet will gain broad adoption, some technology barriers in end systems need to be addressed. For servers, a major issue is protocol processing. Using conventional network interface card (NIC) architecture simply scaled to 10G would result in the CPU’s processing power being the bottleneck. Ongoing efforts seek to offload some TCP processing (TOE) from the system CPU onto the NIC hardware.

Another source of processing overhead is data copying. In a conventional networking stack, incoming packets are stored in operating system memory and later copied to application memory. The copy function consumes CPU cycles and introduces delay. For parallel processing applications that use small buffers, data copying is a major performance hit. Commonly known as iWARP, the protocols for RDMA-over-IP will enable data to be written directly into application memory, eliminating costly copy operations. For applications which use small packets, 10G NICs that implement iWARP will provide lower latency by eliminating memory copies.

10G Ethernet performance has been constrained by the limits of end system interfaces and I/O interconnects. First-generation 10G NICs with partial TCP offloads and PCI-X system interface delivered peak performance of 6-8 Gb/s. Using large packet sizes, these NICs consume less than 100% of a typical server CPU. Second generation 10G NICs with TOE are available and achieve throughput similar to first generation NICs while lowering CPU utilization. Third-generation 10G NICs should achieve full line rate with large packets when combined with end systems with a 3GIO I/O interconnect such as PCI Express.

The smooth inter-working of 10G interfaces from multiple vendors, the ability to successfully fill 10 Gb/s paths both on local area networks, cross-continent and internationally, the ability to transmit greater than 10 Gb/s from a single host, and the ability of TCP offload engines to reduce CPU utilization all illustrate the maturity of the 10 Gb/s Ethernet market. The current performance limitations are not in the network but rather in the end systems.

The annual International Conference for High Performance Computing and Communications (SC)¹ is co-sponsored by ACM SIGARCH and the IEEE Computer Society in November each year. Networks are an integral piece of modern high performance computing. SCinet is the very high-performance network built to support the SC conference. SCinet features both a high-performance production-quality network and an extremely high performance experimental network connecting to all the major national scientific networks and supercomputer centers. 2001 was the first year SCinet deployed two pre-standard 10G LAN interfaces in the showfloor production LAN. In 2002, 10 10G LAN interfaces were deployed. In 2004, 48 10G LAN interfaces were used to satisfy bandwidth requirements.

The Bandwidth Challenge event held during SC invites participants to stress the SCinet network infrastructure while demonstrating innovative applications across the multiple research networks that connect to SCinet. The ability to maximize network throughput is an essential element to the success of high performance computation. The primary measure of performance is the verifiable network throughput. In the five year history of the Bandwidth Challenge during SC, the peak throughput achieved by the winning individual application are shown below-

2000:	1.7 Gb/s
2001:	3.3 Gb/s
2002:	16.8 Gb/s
2003:	23.2 Gb/s
2004:	101 Gb/s

Achievable bandwidth rates are directly related to the number and capacity of WAN circuits brought into the SC venue. You may ask “why is the bandwidth challenge significant?” The Bandwidth Challenge 1) offers an opportunity to test the next generation network capacity as early as 2 years before production; 2) provides the opportunity to test software ideas that will be required to make use of the next generation network 2 years in advance; 3) creates an opportunity to test future network engineers giving them a 2 year lead on the problems with future networks.

National Lamba Rail (NLR) Inc is a consortium of leading U.S. research universities and private sector technology companies "lighting" a national networking infrastructure to foster the concurrent advancement of networking research. Simultaneously, NLR will enable the next generation of network-based applications in science, engineering and medicine.²

NLR is the first national scale network to deploy transcontinental circuits based upon ubiquitous Ethernet technology end-to-end. The use of 10G LAN PHY standards-based facilities in NLR represents a generational shift in the nature, usability and cost of technologies in long-haul circuits. This is a powerful capability that enables the allocation of affordable, independent, dedicated, deterministic ultra-high performance network services for research projects.

Marking a new era in control over and accessibility to national-scale optical networking capabilities for the U.S. research community, the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC) has acquired a dedicated 10G LAN PHY circuit on the NLR infrastructure from Chicago to San Diego via Seattle. The 3,200-mile wavelength, known as the CAVEwave, will initially support the National Science Foundation-funded OptIPuter project shared between UIC and the University of California, San Diego.

“CAVEwave provides researchers with a deterministic network, with guaranteed bandwidth, schedulable times and known latency characteristics, in order to understand requirements for the real-time visualization, analysis and correlation of terabytes and petabytes of data from multiple storage sites,” explained EVL director Tom DeFanti. “All this bandwidth, supplements our existing network infrastructure, for less than the cost of a 32-node cluster at each end!”

The OptIPuter³, so named for its use of Optical networking, Internet Protocol, computer storage, processing and visualization technologies, is an envisioned infrastructure that will tightly couple computational resources over parallel optical networks using the IP communication mechanism. The OptIPuter exploits a new world in which the central architectural element is optical networking, not computers - creating “supernetworks”. This paradigm shift requires large-scale applications-driven, system experiments and a broad multidisciplinary team to understand and develop innovative solutions for a “LambdaGrid” world. The goal of this new architecture is to enable scientists who are generating terabytes and petabytes of data to interactively visualize, analyze, and correlate their data from multiple storage sites connected to optical networks.

Ethernet has withstood the test of time to become the most widely adopted networking technology in the world. Due to its proven low implementation cost, reliability, and relative simplicity of installation and maintenance, Ethernet’s popularity has grown to the point that nearly all traffic on the Internet originates or terminates with an Ethernet connection. As the demand for ever-faster network speeds has increased, the Ethernet standard has been adapted to handle these higher speeds. 10G Ethernet is the natural evolution of the well-established IEEE 802.3 standard in speed and distance. In addition to increasing the line speed, it extends Ethernet’s proven value set and economics to metropolitan and wide area networks by providing: lowest total cost-of-ownership; straight-forward migration to higher performance levels; proven multi-vendor interoperability; and a familiar network management interface.

The10G WAN PHY standard allows service providers to use the installed-base of SONET Layer 1 transport gear to provision 10G Ethernet traffic. For the customer, this eliminates the need for expensive Packet over SONET router interfaces, lowering the barrier to entry into the 10G WAN market.

Third generation 10G NICs combined with 3GIO I/O subsystems in end systems promise to deliver full line rate performance for servers. The maturing of the 10G Ethernet market is demonstrated by the smooth interoperability of 10G interfaces from multiple vendors, the ability to successfully fill 10 Gb/s paths both on local area networks, cross-continent and internationally, and the ability to transmit greater than 10 Gb/s from a single host.

Showcase events such as the SC conference have successfully demonstrated the interoperability of 10G technology as well as its capability of meeting the ever-increasing demand for bandwidth. Participants as well as attendees have an opportunity to witness the next generation network two years in advance

NLR is probably the most ambitious research and education networking initiative since the ARPANET and the NSFnet, both of which led to the commercialization of the Internet. In the spirit of these great success stories, NLR strives to stimulate and support innovative network research to go above and beyond the current incremental evolution of the Internet. The results of such endeavors are expected to facilitate further commercial development and creation of new technologies and markets, thereby stimulating economic development and contributing to U.S. national competitiveness.