CTWatch Quarterly » A National Grid Infrastructure for Australian Researchers

1. Introduction to APAC

The Australian Partnership for Advanced Computing (APAC) was established by the Australian Government to strengthen the advanced computing capabilities in Australia.

It is now a national partnership of eight organisations, one in each State as well as ANU ¹ and CSIRO². The State-based partners are joint ventures involving most of the Australian Universities and have received strong support from the State Governments and their members. All eight APAC partner organizations are listed at the end of the article.

APAC established the APAC National Facility in 2001 to provide a world-class peak computing facility for Australian researchers in higher education institutions. It also initiated programs to significantly increase the expertise and skills in partner organisations to support users of advanced computing systems.

In recent years, the Federal Government has supported APAC broadening its role to provide an advanced computing, information and grid infrastructure for the Australian research community. The APAC National Grid is allowing researchers to access distributed computation and information facilities as a single virtual system and is providing a new range of services to support research collaboration, nationally and internationally.

APAC’s vision is for Australian research teams to have seamless access to distributed computation and information facilities as part of the global research infrastructure. This vision is aligned with recent Government and institutional initiatives that focus on eResearch. For example, the Australian Government has established a National Collaborative Research Infrastructure Strategy (NCRIS)¹, which provides a coordinated approach to the deployment and support of Australia’s research infrastructure.

This paper outlines the concept and activities of the APAC National Grid. More information on APAC can be found at its website².

2. The Concept of the APAC National Grid

The APAC National Facility which is hosted at the ANU provides advanced computing services and specialist support to over 750 users around Australia ³. Most of these users are allocated resources by ‘merit-based’ granting schemes.

The peak system at the National Facility is an SGI Altix 3700 Bx2 cluster with 1680 processors - it was ranked number 35 in the Top500 list of November 2005. The facility also houses a mass data storage system based on Sun servers running SAM-QFS and a Storagetek tape silo with petabyte capacity. The system supports a number of ‘data intensive’ projects including some in linguistics and the social sciences.

In addition to this facility, the partners manage separate facilities and play a vital role in developing Australia’s capability in advanced computing, information and grid infrastructure. They provide operational advanced computing services to their users and are involved in research, development, education, training and outreach activities.

The aim of the APAC National Grid is to allow research teams to easily access the resources at the National Facility and the partner facilities as a single virtual system. These resources include

A variety of computing systems with a diverse range of applications software
Large-scale data storage systems providing services for information management and access
Portals and associated services to access the resources
Collaborative work environments including access grid rooms at each partner site and a range of visualisation and virtual reality facilities.

The APAC National Grid therefore provides a variety of resources to support distributed research teams in an operational (or ‘production level’ environment).

The design of the National Grid incorporates open interfaces to other resources such as institutional computing systems, on-line instruments, sensors and other data sources. It is also developing interfaces that will allow inter-operation with other grids at the institutional, national and international levels. The design aims to allow researchers to see the resources in the APAC National Grid as an extension of their personal work environments. In this sense, the National Grid is an infrastructure that is transparent to most of its users.

The concept of the National Grid as an integrated virtual system with interfaces to external systems and grids as illustrated in Figure 1 below. The resources of the APAC National Facility and the partner facilities are included in the cloud, unified through a common security system and portals to access the resources.

Figure 1. Conceptual diagram of the National Grid as an integrated virtual system

Most of the users of the National Grid will be the current users of the individual systems. The National Grid offers them an easier way to access multiple resources provided by APAC and its partners. As a result, the extent of the resources in the National Grid is being determined by those users that want to access resources through the National Grid interface mechanisms (portals, authentication, authorization). This approach avoids the need for partners to pre-determine a percentage of the resources to be allocated to users of the National Grid. In addition, the current processes to determine the resources provided to users can be continued.

The policies and agreements between the partners to support this approach are based on existing agreements that have been developed elsewhere for production grids. More effort is required to determine the agreements for external organizations in order to interface their systems to the National Grid. This will require an understanding of the services that these organizations demand from the National Grid.

3. The APAC National Grid Program

The APAC National Grid is being built through a program involving over 120 professionals in 24 organisations, with an effective full-time effort of around 50 people. The program is currently in a development phase, with an operational phase starting in early 2007.

The program consists of two kinds of projects that collectively are developing the National Grid infrastructure and delivering services to users. The application support projects involve working with specific research teams in the following areas: astronomy, high-energy physics, chemistry, bioinformatics, earth systems science and geosciences. Table 1 below lists the projects and their aims.

Virtual Observatory Data Warehousing
Efficient, standardised access to key Australian and international astronomical data collections within the Australian Virtual Observatory (Aus-VO).

Gravity Wave Research
Access to Australian computing and storage resources, along with environmental and interferometer detection equipment, and interfacing this grid infrastructure with those of international collaborators.

Virtual Observatory Theory Portals
Services for job configuration, job submission to available grids, and subsequent monitoring for a wide range of theoretical astrophysics codes with access to the Aus-VO.

Belle and ATLAS Experiments
On-line access to the Belle data (10 TBytes per year) produced by the KEK laboratory in Japan through a federation of SRB databases. Deployment of the LHC Computing Grid (LCG-2) toolkit to support implementation tests and physics simulations as part of the Atlas project.

International Lattice Data Grid
Implement a node of the International Lattice Data Grid (ILDG) in Australia to locate and access expensive lattice simulation data.

Computational Chemistry
A uniform web-based interface to computational molecular science codes allowing data exchange between different codes and unified operations on a set of molecules.

Genome Annotation
A distributed Genome Annotation System (using the Rice Genome as a model) involving a local Ensembl database and a grid-enabled Blast system.

Molecular Docking
A portal for molecular docking applications using a wide variety of chemical databases, with interfaces for automated analysis, visualization and storage of virtual screening results.

Earth Systems
Access to Australian data sets related to oceans, atmospheres, Antarctica and climate change using OpenDAP protocols and to computational models producing global data sets of land surface fluxes, state variables and related hydrologic quantities.

Geoscience Modelling
A workflow service to support mantle convection modeling accessing using open standards (based on XML) to access geological and geophysical data.

Earthbytes
Visualization and simulation of tectonic plate movements based on geological and geophysical observations.

Table 1. Application Support Projects in the APAC Grid Program

The infrastructure projects are developing and providing services to satisfy the requirements of the applications. These projects cover the areas of computing infrastructure, information infrastructure, user interfaces and visualisation infrastructure and collaboration infrastructure.

The core grid middleware for the National Grid is the Virtual Data Toolkit (VDT) being used in the Open Science Grid and deployed on Globus Toolkit (Version 2). This is being complemented with toolkits for virtual organisation management, resource discovery, job scheduling and job monitoring.

An APAC Certificate Authority (CA) has been established to provide an authentication service for users of the National Grid. The CA has recently been recognised as a production level service by the Asia-Pacific Grid Policy Management Authority (APGridPMA). This allows APAC certificates to be acknowledged by other grids around the world and therefore support international research collaboration.

The design of the National Grid involves a ‘gateway’ system at each partner site configured to support a range of grid services and to receive and process grid service requests. The aim of the design is to:

limit the number of systems that need grid components installed and managed within the APAC partnership thus reducing overall grid management overheads
enhance security as many grid protocols and associated ports only need to be open between the gatekeepers, as only the local gatekeeper needs to interact with site systems
support roll-out and control of production grid configuration through the implementation of standardised grid support across all APAC partner sites
support production and development grids and local experimentation without significant hardware investment through a Virtual Machine implementation where different services and different quality of services are provided on separate grid installations

Through the use of a ‘virtual machine’ tool, the gateways are supporting the following production and development grids simultaneously:

Virtual Machine 1 VDT based on Globus 2 (ng1)
Virtual Machine 2 VDT based on Globus 4 (ng2)
Virtual Machine 3 SRB services (ngdata)
Virtual Machine 4 Production portals
Virtual Machine 5 Development server (ngdev)
Virtual Machine 6 LCG grid

The architecture of the gateway system at one of the APAC partners (VPAC) is shown in Figure 2. The gateway system is connected to the external networks (GrangeNet and AARNet) and internally to the compute systems (Edda, Brecca, Wexstan) as well as the data control network. The scheduling of jobs on the compute systems is done through a version of PBS.

Figure 2. Architecture of an existing APAC partner gateway system

The information infrastructure is providing a variety of tools for autonomous data movement, replication and optimised access, data virtualisation and metadata support. An SRB federation has been implemented to support access to distributed data sets.

Portals for applications to access the National Grid are being developed using the GridSphere toolkit. Generic portlets have been developed for common procedures such as authentication, file transfer and job submission. Plans are being developed to provide a rendering engine for visualization and supporting simultaneous viewing across multiple sites.

Australian researchers have access to a number of Access Grid nodes and considerable effort has been contributed by the APAC partners and other groups to improve the robustness, management, usability and accessibility of these facilities. This effort is being extended in the APAC Grid program to enable these nodes to be used as instrument control rooms, distributed visualisation facilities and collaborative work environments.

Plans are being developed for the APAC partners to be connected by a high-speed private network. Currently most the partners are connected by the GrangeNet experimental network ⁴. This will be replaced by a network configured on the new national backbone network being installed by AARNet⁵.

4. The APAC National Grid in the International Context

APAC is participating in discussions with other ‘production grid’ organisations on the development of inter-operable grids. This will enable Australian researchers to collaborate using computing and information facilities around the world. Examples of collaboration are in accessing astronomical data sets, very long baseline interferometry, Belle and Atlas experiments, lattice data grids, and climate change modelling.

Such international collaboration is being underpinned by an advanced communications infrastructure involving a trans-Pacific gigabit research link provided by AARNet.

APAC is also participating in a number of international programs to promote the development and use of grid infrastructure. APAC is a Founding Institutional Member of PRAGMA which fosters cooperation on grid infrastructure and applications by Pacific Rim organisations. APAC is also a Silver Sponsor of the Global Grid Forum.

The APAC Partners

Australian Centre for Advanced Computing and Communications (ac3): http://www.ac3.com.au
Commonwealth Scientific and Industrial Research Organisation (CSIRO): http://www.hpsc.csiro.au
The Hub of Advanced Computing in Western Australia (iVEC): http://www.ivec.org
Queensland Parallel Supercomputing Foundation (QPSF): http://www.qpsf.edu.au
South Australian Partnership for Advanced Computing (SAPAC): http://www.sapac.edu.au
The Australian National University (ANU): http://www.anusf.anu.edu.au
University of Tasmania acting as host for the Tasmanian Partnership for Advanced Computing (TPAC): http://www.antcrc.utas.edu.au/tpac
Victorian Partnership for Advanced Computing (VPAC): http://www.vpac.org

¹ http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/ncris
² http://www.apac.edu.au
³ http://www.nationalfacility.apac.edu.au
⁴ http://www.grangenet.net
⁵ http://www.aarnet.edu.au