In addition to achieving high performance and being scalable to large numbers of processors, scientific codes must be carefully designed so that they can easily accommodate new mathematical models and algorithms as knowledge advances. If scientific codes can not evolve as new knowledge is gained, they will rapidly become outdated. It must also be possible to move the codes from one generation of computers to the next without undue difficulty as computer technology advances—the lifetime of scientific codes is measured in decades, the lifetime of computers in years.
For the above reasons, NWChem is best thought of as a framework or environment for chemical computation rather than a single, fully integrated application. The framework defines and supports a “virtual machine” model and mandates a certain structure for new modules. A well-defined virtual machine model hides details of the underlying hardware and encourages programmers to focus on the essentials; correctness, good sequential performance, expressing concurrency, and minimizing data motion. The high-level structure ensures correct operation, provides a consistent look-and-feel for users, and enables code reuse. New chemical functionality can then be developed using a well defined set of capabilities, which are described below. Testimony to the success of this framework (and perhaps to the inadequacy of our current programmer’s manual) is a recent comment from a developer new to NWChem – “my program is running correctly in parallel but I don’t know how.”
The key to achieving the above goals is a carefully designed architecture that emphasizes layering and modularity (Fig. 1). At each layer of NWChem, subroutine interfaces or styles were specified in order to control critical characteristics of the code, such as ease of restart, the interaction between different tasks in the same job, and reliable parallel execution. Object-oriented design concepts were used extensively within NWChem. Basis sets, molecular geometries, chunks of dynamically allocated local memory, and shared parallel arrays are all examples of “objects” within NWChem. NWChem is implemented in a mixture of C and Fortran-77, since neither C++ nor Fortran-90/95 were suitable at the start of the project. Since we did not employ a true object-oriented language, and in particular did not support inheritance, NWChem does not have “objects” in the strict sense of the word. However, careful design with consideration of both the data and the actions performed upon the data, and the use of data hiding and abstraction, permits us to realize many of the benefits of an object-oriented design.
In the very bottom layer of NWChem is the Software Development Toolkit. It includes the Memory Allocator (MA), Global Arrays (GA), and Parallel IO (ParIO). The “Software Development Toolkit” was (and still is) the responsibility of the computer scientists involved in NWChem. It essentially defines a “hardware abstraction” layer that provides a machine-independent interface to the upper layers of NWChem. When NWChem is ported from one computer system to another, nearly all changes occur in this layer, with most of the changes elsewhere being for tuning or to accommodate machine specific problems such as compiler flaws. The “Software Development Toolkit” contains only a small fraction of the code in NWChem, less than 2%, and only a small fraction of the code in the Toolkit is machine dependent (notably the address-translation and transport mechanisms for the one-sided memory operations).
The next layer, the “Molecular Modeling Toolkit,” provides the functionality commonly required by computational chemistry algorithms. This functionality is provided through “objects” and application programmer interfaces (APIs). Examples of objects include basis sets and geometries. Examples of the APIs include those for the integrals, quadratures, and a number of basic mathematical routines (e.g., linear algebra and Fock-matrix construction). Nearly everything that might be used by more than one type of computational method is exposed through a subroutine interface. Common blocks are not used for passing data across APIs, but are used to support data hiding behind APIs.
The runtime database (RTDB) is a key component of NWChem, tying together all of the layers of NWChem. Arrays of typed data are stored in the database using simple ASCI strings for keys (or names) and the database may be accessed either sequentially or in parallel.
The next layer within NWChem, the “Molecular Calculation Modules,” is comprised of independent modules that communicate with other modules only via the RTDB or other persistent forms of information. This design ensures that, when a module completes, all persistent information is in a consistent state. Some of the inputs and outputs of modules (via the database) are also prescribed. Thus, all modules that compute an energy store it in a consistently named database entry—in this case
The highest layer within NWChem is the “task” layer, sometimes called the “generic-task” layer. Functions at this level are also modules—all of their inputs and outputs are communicated via the RTDB, and they have prescribed inputs and outputs. However, these capabilities are no longer tied to specific types of wave functions or other computational details. Thus, regardless of the type of wave function requested by the user, the energy may always be computed by invoking task_energy() and retrieving the energy from the database entry named task:energy. This greatly simplifies the use of generic capabilities such as optimization, numeric differentiation of energies or gradients, and molecular dynamics. It is the responsibility of the “task”-layer routines to determine the appropriate module to invoke.
