Welcome to the second issue of the Cyberinfrastructure Technology Watch Quarterly. In this issue we focus on the state of one and 10 Gbps long-haul, optical circuits supporting the research community. It has been over a decade (1994) since the Very High-Speed Backbone Network Services (vBNS) connected major NSF centers and universities at OC-3 (155 Megabits/Sec Mbps). Soon thereafter, selected circuits were upgraded to OC-12 (622Mbps). In 1997, Internet2¹ was formed to connect a larger collection of universities at OC-12 and Gigabit speeds. Internet2 now operates with 10 Gigabit backbones. However, according to sites such as NASA’s ENSIGHT,² end–to-end file transport from major scientific data repositories to end users laboratories across the shared internet is less than 1% of that: typically 50-100 mbps.

In the late 1990s, the future looked promising for long-haul research networks, and in three years vBNS/Internet2 dramatically increased long-haul research network capacity. But in 2000, the “Tech Bubble” turned into the “Tech Meltdown.” During the Bubble phase, fiber-strands were being laid worldwide at the rate of over 8,000 km/hour (70 million kilometers in 1999)! Multiplying this build-out, Dense Wave-Division Multiplexing (DWDM) enabled carriers to run multiple 10-gigabit or even 40-gigabit channels (each termed a “lambda” for the wavelength bin in the infrared band it occupies) on a single fiber pair. This meant that the carriers had a functional way to multiply bandwidth without expensive trenching of new fiber. Current DWDM technology supports up to eighty 10-gigabit “waves” or “lambdas” on such networks (a mere 800 times the capacity of Internet2 backbone of eight years ago).

As a number of long-haul network companies went bankrupt, a new opportunity became available because of the bandwidth overbuild — telecom carriers were willing to discuss long term leases of fiber or lambdas to individuals. Foreign countries took the lead, with Canada’s CANARIE being in the vanguard. In the United States, NCSA, Argonne National Lab, EVL at UIC, and Northwestern convinced the state of Illinois in 1999 to extend the Illinois Century Network to construct a dark fiber state network, called I-WIRE,³ to support Illinois researchers’ needs for large amounts of bandwidth. In 2001, the Distributed TeraGrid Facility (DTF) proposed connecting large data nodes and computing clusters on a national scale using a 40 Gigabit dedicated backbone (four 10 Gb lambdas) among the four originating centers to form what we know now as the TeraGrid.⁴ Also in 2001, NSF funded the international “point of entry” for research networks STARTAP to become StarLight⁵ — a 1GigE and 10GigE switch/router facility for high-performance access to participating networks and a true optical switching facility for wavelengths.

We see these three state, national, and international initiatives as the catalyzing events. Researchers worldwide were convinced that large, dedicated optical circuit research networks were not only theoretically possible but were practically being put into service. A scant four years after the original TeraGrid award and three years after I-WIRE became operational, there are now eight sites connected to the extended TeraGrid, using the new formed National LambdaRail (NLR)⁶ to create extensions of the original four lambdas, and by now over two dozen state and regional dark fiber networks exist and are interconnecting to NLR.

In this issue, we are fortunate to have three articles whose authors have all played critical roles in this new age of long-haul research networks. They delve deeper into the details and provide critical insights:

Linda Winkler from Argonne National Laboratory has been in the trenches for both Teragrid and SciNet (the SCxy Conference’s big monster network that exists for 5 days every November). In her article, “Does 10G Measure Up?”, she describes the challenges of high-end research deployments and highlights heroic efforts in the Bandwidth Challenge where current state-of-the-art clocks in at over 100Gbps for an application using multiple 10G networks at SC2004. Linda also takes us through some of the Teragrid infrastructure.

In an article titled “The National LambdaRail” by Dave Farber and Tom West, the authors describe how regional research networks have taken advantage of the abundance of dark fiber to enable multiple, research-focused 10 Gigabit networks. Farber and West give a very nice condensed history of high-speed networking, how a variety of environmental factors made the NLR possible, how the NLR is being used today and what researchers might expect looking out 10 years.

Fast Research Networks are not a US-only concession. In fact, some would say that the US is a fast follower in the on-demand, lambda-based network. In “Translight, A Major US Component of GLIF”, Tom DeFanti, Maxine Brown, Joe Mambretti, John Silvester, and Ron Johnson describe the optical interconections available between U.S. and international researchers. Truly, research networks have gone global and big/fast research networks are becoming prevalent. International partners are critical in the world of Team Science.

Here at UCSD, we are building a campus-level OptIPuter⁷ interconnecting five laboratories and clusters of three functional types: compute, storage, and scalable tiled display walls. The total number of nodes in the OptIPuter fabric exceeds 500 and each lab has four fiber pairs that connect it to a central high-speed core switching complex that has both a Chiaro Enstara⁸ (a large router based on a unique optical core) and standard Cisco⁹ 6509 switch-router. The current instantiation supports both 10 gigabit and one gigabit signals running from each lab to the central core. In a follow-on NSF-funded proposal, Quartzite augments this structure by adding DWDM signaling on the established fiber plant, a transparent optical switch from Glimmerglass, and in 2006, a wavelength selective switch from Lucent. When complete, the Quartzite switching complex will be able to switch packets, wavelengths or entire fiber paths, allowing us to build different types of network layouts and capabilities to test OptIPuter research and other optical networking ideas. With reconfigurable networks and clusters, OptIPuter/Quartzite forms a campus-scale research instrument. At build-out, this instrument will support nearly half a Terabit of lambdas landing into a central, reconfigurable complex.

OptIPuter and Quartzite preview what campuses need to evolve to: immense bandwidth, optical circuits on demand, and reconfigurable endpoint systems. Of critical importance is the evolution of large and network-capable storage clusters that can be accessed with clear paths from research labs scattered around campus. Using cluster management systems (we use the Rocks Clustering Toolkit¹⁰), most scalable systems (compute clusters, tiled display clusters, application servers) can be thought of as soft-state. However, as science moves to the inevitable data-intensive modes, information storage is critical to the campus research enterprise. This coming generation of campus networks allows storage silos (critical state) to be remote from labs, then managed and operated on behalf of researchers without losing performance or adaptability for the research scientists themselves. In essence, soft-state systems can be put anywhere on campus (notably in labs), and critical-state systems are not required to be in physical proximity. “Unlimited” campus network capacity allows universities to co-optimize the preservation of critical data and the ability to rapidly change soft-state systems to meet research challenges.

We’d like to close with the following thought. Long haul, fast research networks are springing up everywhere and bandwidth is finally meeting the “it will be abundant” predictions that many of us have believed for nearly a decade. However, the missing link overall is the campus connectivity — some campuses are pioneering big networks, but most still operate on one gigabit backbones. It is a strange turn of events when the long-haul network is fatter and more capable than your campus network.

Arden Bement, the director of the National Science Foundation recently discussed this issue in the Chronicle of Higher Education.¹¹

“Research is being stalled by ‘information overload,’ Mr. Bement said, because data from digital instruments are piling up far faster than researchers can study. In particular, he said, campus networks need to be improved. High-speed data lines crossing the nation are the equivalent of six-lane superhighways, he said. But networks at colleges and universities are not so capable.“Those massive conduits are reduced to two-lane roads at most college and university campuses,” he said. Improving cyberinfrastructure, he said,“will transform the capabilities of campus-based scientists.”

Major challenges facing the scientific R&E community today involve insatiable needs for communications and collaborations at distance, as well as the ability to manage globally distributed computing power and data storage resources. Advances in telecommunications and networking technologies are up to the challenge of meeting these ever increasing demands for bandwidth.

Not long ago, it was common to find large disparities between the connection speeds of end systems to the local area network versus the speed of the shared wide area network. Fortunately, that is no longer common and instead we find end systems connected at speeds of 100 Mb/s to 1 Gb/s and wide area links operating at 1 Gb/s to multiples of 10 Gb/s. Many factors have contributed to the technology boom that resulted in wide area speeds becoming attainable. The dot com boom saw a huge build-out of infrastructure and technology only to have the bottom drop out of the marketplace. Many R&E organizations have taken advantage of the marketplace to acquire access to dark fiber and build customer owned and operated metropolitan, regional and wide-area infrastructures. This has put the fate of bandwidth availability squarely within the control of the end customers rather than the service providers and carriers. The result is a strategic opportunity for the R&E community.

Making multi-gigabit/second, end-to-end network performance achievable will lead to new models for how research and business are conducted. Scientists will be empowered to form virtual organizations on a global scale, sharing information and data in flexible ways, expanding their collective computing and data resources. These capabilities are vital for projects on the cutting edge of science and engineering, in data intensive fields such as particle physics, astronomy, bioinformatics, global climate modeling, geosciences, fusion, and neutron science.

This article addresses Ethernet advances leading to network convergence, end-to-end performance factors, and highlights examples of 10G early adoption, deployment, and wide-area infrastructure availability in the R&E community.

We are witnessing the convergence of LAN/MAN/WAN data rates at 10 Gb/s, resulting in common equipment and interfaces to access the enterprise, metro and wide area network. As a result, we are experiencing reductions in the cost for implementation, ownership, support, maintenance and in some cases, recurring charges in the WAN.

Ethernet technology is currently the most deployed technology for high-performance LAN environments. Enterprises around the world have invested in cabling, equipment, processes, and training in Ethernet. In addition, the ubiquity of Ethernet keeps its costs low, and with each deployment of next-generation Ethernet technology, deployment costs have trended downward. Ethernet has gained worldwide acceptance as an enterprise infrastructure technology due to its considerable advantages in interoperability, scalability, simplicity, consistency, service ubiquity, provisioning speed, and price/performance. With the rapid price decline in Gigabit Ethernet network interface cards, most servers come standard with Gigabit Ethernet network interface cards.

The US National Science Foundation (NSF) funds two complementary efforts through its International Research Connection Networks (IRNC) program — TransLight/StarLight and TransLight/Pacific Wave — that provide multi-gigabit links and supporting infrastructure to interconnect North American, European and Pacific Rim research & education networks, as well as to supplement available bandwidth that is provided by other countries.

TransLight/StarLight’s mission is to best serve established US/European production science, including support for scientists, engineers and educators who have persistent large-flow, real-time, and/or other advanced application requirements. Two OC-192 circuits are being implemented between the US and Europe. One circuit is a 10 Gb/s link that connects Internet2/Abilene and the pan-European GÉANT2 via a routed network connection. The second circuit is a 10 Gb/s link that connects US hybrid networks, which can provide high performance, dedicated optical channels, such as the National LambdaRail (NLR), to similar European networks at NetherLight (configured as either one 10 Gb or eight 1 Gb switched circuits, or lambdas). Considerations related to security and measurement/monitoring will carefully be addressed under this award for both circuits.¹

TransLight/Pacific Wave’s mission is the development of a distributed Open Exchange along the US west coast, from Seattle to San Diego, to interconnect North American, Asian, Australian and Mexican/South American links.²

Across North America, NLR, Canada’s CA*net4, and Internet2’s Abilene and Hybrid Optical and Packet Infrastructure (HOPI) projects connect the combined TransLights, from New York (Manhattan Landing, or MAN LAN), to Chicago (StarLight), to Seattle (Pacific Northwest GigaPoP). Pacific Wave carries the connection from Seattle down the US west coast to Los Angeles and on to San Diego and Tijuana (via CalREN - the California Research and Education Network, which is operated by CENIC - the Corporation for Educational Network Initiatives in California). These locations are the sites that support the vast majority of international connections to the US and form the fabric by which most international networks peer and exchange traffic with Abilene and the US Federal Research Networks. The TransLight team is the global community of people and groups who have most advanced the art, architecture, practice, and science of Open Exchange interconnectivity among high-performance networks. TransLight’s approach is based not just on backbone connectivity, but end-to-end connectivity and activism in advanced networking and applications, with a proven track record in attracting new technologies and stimulating collaborations, especially among leading domain scientists at end sites.

TransLight enables grid researchers and application developers to experiment with deterministic provisioning of dedicated circuits and then compare results with standard, aggregated “best-effort” Internet traffic. Multi-gigabit networks are referred to as “deterministic” networks, as they guarantee specific service attributes, such as bandwidth (for researchers who need to move large amounts of data), latency (to support real-time collaboration and visualization), and the time of usage (for those who need to schedule use of remote instrumentation or computers). Only through deployment of an integrated research and production infrastructure at network layers 1 through 3 will the various technical communities be able to address the major challenges of large-scale and complex systems research in peer-to-peer systems, Grids, collaboratories, peering, routing, network management, network monitoring, end-to-end QoS, adaptive and ad hoc networks, fault tolerance, high availability, and critical infrastructure to support advanced applications and Grids.³

The demands of leading-edge science increasingly require networking capabilities beyond those currently available from even the most advanced research and education networks. As network-enabled collaboration and access to remote resources become central to science and education, researchers often spend significant time and resources securing the specialized networking resources they need to conduct their research. As a result, there is less time and fewer resources available to conduct the research itself.

New technology holds the promise of providing more easily the networking capabilities researchers require. Increasingly, the best option for ensuring the technology and these capabilities are available seems to be for the research and education community to own and manage the underlying network infrastructure. This movement towards a facilities-owned approach is relatively unprecedented in the history of research and education networking, yet holds the promise for unique benefits for research and education. Ownership provides the control and flexibility, as well as the efficiency and effectiveness needed to meet research and education’s uniquely demanding networking requirements.

A new global network infrastructure owned and operated by the research and education community is being developed, deployed, and used. In the United States, a nationwide infrastructure is being built by the National LambdaRail (NLR) organization, in collaboration with scientists and network researchers, with leadership from the academic community, and in partnership with industry and the federal government. Furthermore, NLR both leverages, and provides leverage for, existing and new regional and local efforts to deploy academic-owned network infrastructure.

To understand how the most recent movement in research and education (R&E) networking differs from those of the past, and how unique the capabilities it provides are, let us take a look back at how R&E networking has developed in the United States over the past 35 years. In 1987, the initial NSFNET backbone provided just 56 kilobits per second of bandwidth. Even in 1991 only 1.5 megabits per second were available on the backbone, less than many current home broadband connections. Today, nationwide R&E networks have links of 10 Gigabits per second (Gbps), nearly 7000 times their capacity just 15 years ago. Yet it is increasingly apparent that even this is not enough capacity to meet emerging demands.

It is also important to realize that, tracing the development of today’s Internet back to the ARPANET of 1969, pioneers from the university community, with the support of government and industry, have provided leadership for network development to meet the needs of research and education. While many share in the development and evolution of the Internet as we know it today, university-based researchers played a key role both in developing fundamental Internet technologies and in providing large-scale testbeds that put those technologies to work and drove their further development.