Long-distance, high-speed and low-cost networking has encouraged the development of applications taking advantage of geographically distributed resources, opening up new research directions that were previously limited or unexplored for economic and practical reasons. The establishment of Cyberinfrastructure allows mature, scalable computing for different application communities.

Grid initiatives in Brazil were initially driven by international collaborations in several application areas and the pursuit of higher network bandwidth and larger computational facilities. In response to this demand, the National Laboratory for Scientific Computing – LNCC headed a proposal formulated together with representatives of application groups, computer and computational science, computer networking, high-performance computing, and federal government funding agencies belonging to the Ministry of Science and Technology (MCT). This early proposal was based on a number of existing international initiatives and focused on improving connectivity and communication performance for the coordinated use of existing regional HPC centers, as well as a number of academic and research institutions as potential users. At this time there were seven regional HPC centers and nine academic and research institutions involved, with connectivity provided by the Brazilian National Research and Education Network – RNP, and funding by the MCT funding agencies. The application areas included high-energy physics, bioinformatics, climate and weather forecasting and oil industry needs, among others.

In 2003, the legal framework was established for a National System for HPC (SINAPAD) with LNCC being designated the national coordinator (on behalf of MCT), and was recognized as part of the Federal Education, Science & Technology infrastructure. SINAPAD consisted of a network of regionally distributed, operational HPC centers aimed at providing computation on demand for education and scientific purposes, with a proposed operational structure based on mid-sized computer systems and clusters organized into a grid for the sharing of resources and reduction of idle time. Clusters of a few hundred processors were planned for each center, combining distributed and shared memory machines, facilities for data storage and handling, user friendly access through a web portal, security, accounting, and the option of several alternative architectures.

Brazil has a territorial extent of about 8.5 million km2, making it the fourth largest country in the world, with correspondingly large demands in order to reach all parts of the country. The federal constitutional order is similar to the USA, with 26 states and a Federal District around the national capital, Brasília. The history of academic HPC in Brazil began in 1990, based on the concept of government-funded regional centers open to universities and research institutes, as in Table 1, with seven such centers being established between 1992 and 1997. INPE formally joined the system only in 2002, contributing with its former production machine formerly dedicated to environmental sciences.

Internet technology had already been brought to Brazil by concerted initiatives by state governments in a number of the more populous states, such as São Paulo, Rio de Janeiro and Rio Grande do Sul, coordinated nationally by the federal government through RNP. The first national IP backbone network in Brazil was launched by RNP in 1992 to serve the national academic and research community, and connected 11 cities ¹. Since 1992, the national network run by RNP has evolved through four significant increases in capacity and usually also technology, where the fourth such change was initiated in November, 2005. Table 2 summarizes these changes. It is relevant to discuss here the Project GIGA testbed network (2004-) and the most recent national network upgrade initiated in 2005.

With the objective of developing and deploying new optical networking technologies, CPqD ² and RNP formed a partnership to build an optical testbed network, connecting seven cities in the adjoining states of Rio de Janeiro and São Paulo. This project, named GIGA ³, has been supported financially since December 2002 by the federal government's Fund for the Development of Telecommunications Technology (FUNTTEL) and has resulted in building a 700 km network using dark fiber provided without cost by four telcos and lit up using WDM optical equipment produced by PADTEC, a spin-off of CPqD. This network provides IP service over Gigabit Ethernet to end users in 17 research institutions and four telcos in seven cities (see Figure 1). Most users are R&D groups developing subprojects of Project GIGA, under contract to CPqD or RNP. For RNP, the establishment of this optical networking testbed was the first step in implementing its long-term strategy, known as the National Optical Initiative (ION), which emphasizes the provision of future networking services to end-users through a "facilities-based" infrastructure, such as dark fiber or WDM waves, rather than one based on renting telco-provided services, such as ATM or SDH/Sonet.

The testbed network built by Project GIGA has provided extensive hands-on experience in the design, deployment and operation of both metropolitan and long-distance optical networks, and thus has served as a basis for future development of networks to serve the wider research and higher education community. Within RNP, forward network planning has been heavily influenced by the experience gained in Project GIGA, particularly related to identifying and serving particular user groups, and also as regards the technologies employed. Attention has also been given to new directions in networking adopted in other countries during the last five years, highlighted by such initiatives as National Lambda Rail ⁴, CAnet4 ⁵ and SURFnet6 ⁶, among others, and widely reviewed in the excellent series of reports produced by the European SERENATE study ⁷. It was decided that RNP should accompany the global tendency to increase the link capacities of its national network to multiple Gbps. This was in fact undertaken in November, 2005, but was preceded by a migration to SDH/Sonet links starting in 2003, with the abandonment of ATM and Frame Relay (FR) as link technologies. The fact that this first migration could be achieved with an increase in aggregate link bandwidth of around six times (from 350 Mbps in 2003 to over 2 Gbps in 2005) with a 30% reduction in cost is a reflection of a combination of newer technologies and the introduction of competition in the telecommunications marketplace.

In 2005, RNP moved onto the next stage of its ION strategy, replacing the links between the 10 principal cities of its national network by a solution in the form of unprotected, transparent lambdas (waves) of 2.5 and 10 Gbps. The core of what is now known as the IPÊ ⁸ network (see Figure 2) was commissioned in November. The increase in aggregate link bandwidth for this IPÊ network core is from around 1.6 Gbps (SDH) to 60 Gbps (waves), or almost 40 times, at only three times the cost! In fact, the overall cost of the whole national network is now just 30% more than in 2003, for an increase in aggregate bandwidth of almost 180 times. As an essential complement to the deployment of the IPÊ network, RNP is also engaged in a nationwide project, known as Community Networks for Education and Research (Redecomep), to deploy optical metropolitan area networks in all 27 capital cities by December 2006. These networks are designed to provide gigabit access to the RNP point of presence (PoP), as well as interconnecting all the campi of RNP clients in these metro areas ¹. International connectivity for RNP's IPÊ network is based on links to commodity networks and also research and education connections (RE) directly connecting to other RE networks (REN), such as CLARA (the regional Latin American REN), Abilene (US NREN), CalREN (California REN), Géant (European REN), among others. A summary of these is shown in Table 3.

Figure 1. Location of the Project GIGA testbed network (2004)

Figure 2. IPË network in 4Q2005

At LNCC, early grid-related research and development activities started within Project ComCiDis (Distributed Scientific Computing) in 2002, with the purpose of providing guidelines and directions to the SINAPAD initiative, as well as to establish and strengthen national and international collaborations.

The research program of Project GIGA coordinated by RNP includes the thematic area of Large-Scale Distributed Applications, initially with twelve supported subprojects, allowing for the development and maturing of such technology between high-speed interconnected partners, targeting grid middleware and also some applications. The testbed traffic is restricted to supported subprojects. We focus on two of these subprojects, InteGridade ¹¹ and Sinergia, where NCSA ¹² participates as an international collaborator on cyberinfrastructures and cyberenvironments. These two subprojects are actually conducted as a single project, involving five with testbed access, and two more connected via Internet only. The institutions involved are: LNCC - ComCiDis (Distributed Scientific Computing Group), UFF - Institute of Computing, CBPF ¹³, UNICAMP - Institute of Computing, PUC-Rio ¹⁴ – Parallelism Lab, and UFRGS – Institute of Informatics, and UFES ¹⁵ - Dept. of Informatics. We also count on RNP and SINAPAD as natural partners for providing connectivity and computational resources, respectively. An additional international partner is Monash University (Melbourne, Australia) for access to Nimrod-G ¹⁶ based international testbeds and exploring applications based on Nimrod-O. As business partners, we have IBM participating through its academic software program and a local company (Taho) concentrating on wireless connectivity.

The objectives and goals of this joint project include: scalable computing infrastructure, tools and applications; tools and portals for monitoring, submission and scheduling of applications; development of some application-specific interfaces; development and integration of new services, integration of sensors and wireless resources; and application testbeds. The resulting activities distributed among the partners have been the following: implementation of a Grid with the inclusion of some of the partners' local clusters; implementation of a portal; establishment of policies; a monitoring service; scheduling policies and a scheduling service; a data integration service; aspects related to service orchestration; security comformance monitoring; network management services; automatic application transformation tools; accounting; and the porting of some applications.

The current grid infrastructure for the ComCiDis and InteGridade-Sinergia projects includes the use of Globus 2 ¹⁷, for interconnections, and of SGE ¹⁸, OpenPBS ¹⁹, and Condor ²⁰ as schedulers within clusters. The implemented portal is servlet-based and includes a monitoring service based on a LUA ²¹ script collecting information from Globus-MDS. Regarding some current activities we have been working on the development of a data integration service ²², automatic application transformation developments based on EasyGrid ^{23 24}, scheduling ²⁵, hierarchical submission ²⁶, dynamic adaptation ²⁷, and wireless grids ²⁸, among others. We are migrating to Globus 4 and in terms of security, we have been working with Globus certificates and are now moving to MyProxy ²⁹. OpenCA ³⁰ is being used for certification in the SINAPAD grid.

Together with NCSA, activities include building cyberenvironments and global infrastructure. With LNCC activities include becoming an external international node of NCSA resources and a part of the NCSA Condor pool. This involvement also includes development of grid middleware services, portals for specific applications, scheduling algorithms and service, security issues, support for service-oriented applications, the definition of common services for a certain set of applications and fine-grain authorization.

Applications have been developed to use the cyberinfrastructure facilities (at LNCC and also at some of the partners) for use in bioinformatics, haemodynamics of the human cardiovascular system, oceanography, climate studies and geoprocessing, for which reference groups have been set up at LNCC. Other application areas include molecular dynamics (in physics), astronomy, engineering with developments for the oil industry, and environmental modelling of the Amazon region hydrologic basins, with some spin-offs such as e-knowledge and e-government.

In SINAPAD, the goals are to increase the number of regional centers from seven to ten, in order to cover the whole country, and to deploy a total computational and storage resources of 4 Tflops and 30 TB, respectively, in 2006, with respective increases to 5 Tflops and 50 TB in 2007, targeting applications demanding large data set storage and management. The better use of machines also depends on software, including new programming paradigms, user interfaces, and distributed databases - therefore, these are also areas for research and development and of investments.

In terms of networking, through the deployment of the IPÊ network in 2005 and the ongoing deployment of optical metro networks in capital cities expected to be completed by the end of 2006, RNP is bringing about a significant change in the quantity and quality of communications resources at the disposal of the Brazilian research and higher education community, permitting the widespread use of advanced applications. Future efforts will be directed towards extending these facilities more widely, bringing multiple gigabit connectivity to the remaining 17 state capitals and also to population centers outside the metropolitan districts of the national and state capitals.

National Laboratory for Scientific Computing, ­ LNCC - http://www.lncc.br
National Research and Education Network, ­ RNP - http://www.rnp.br