Power consumption has become an increasingly important issue in HPC. Ignoring power consumption as a design constraint results in a HPC system with high operational costs and diminished reliability, which translates into lost productivity. Examples of such (capability) systems include ASCI White, ASC Q, and the recently unveiled ASC Purple.
Specifically, due to the exorbitant power consumption of ASC Purple, the facility that houses ASC Purple requires new air-handling designs and specifications to deal with ASC Purple’s gargantuan 7.5-MW appetite. With an average utility rate of $0.12/kWh, the electrical bill alone for this system would run nearly $8M/year. If we scale this architecture up to a petaflop machine, it would need approximately 75 MW to power up and cool down the machine. The power bill for such a system would then be on the order of $80M/year, assuming energy costs stay at $0.12/kWh. In addition, the expected mean time between failures for systems of this size is forecasted to be on the order of hours rather than days; further scaling of such capability supercomputers would result in HPC systems that would have several failures per hour by 2010.5
For the above reasons, this article presented a case for low-power (and power-aware) HPC in order to significantly improve reliability and efficiency, particularly with respect to operational costs. However, the main issue with low-power HPC is that it sacrifices too much raw performance in order to achieve its goals. Perhaps what the HPC community needs is an EnergyGuide sticker for HPC systems, like the one shown in Figure 5 for Green Destiny. Or more seriously, perhaps we should remember that our attitude towards energy contributed to the massive rolling blackouts in the summers of 2000, 2001, and 2003 and cost the U.S. billions of dollars and disrupted millions of lives, as noted this month by President George W. Bush when signing the 10-year, $12.3-billion Energy Policy Act of 2005.
As a compromise, there exists an emerging body of research in power-aware HPC. The basic idea is to start with a high-performance, high-power CPU that supports a mechanism called dynamic voltage and frequency scaling and then to create a power-aware algorithm that conserves power by scaling down the CPU supply voltage and frequency at appropriate times, as power draw is directly proportional to the CPU frequency and the square of the CPU supply voltage. Because the CPU consumes the largest percentage of power in a HPC node, this technique has been shown to be highly effective in reducing the overall power and energy consumption in an HPC system.
In the longer term, e.g., by 2020 when the failure rate is expected to reach several failures per minute,5 we will need the continued proactive approach towards power consumption espoused here in order to stave off the aforementioned forecast as well as reactive fault detection and fault handling in order to give the user the illusion of a fault-free machine.
2 www.sc-conference.org
3 M. Seager, "What Are The Future Trends in High-Performance Interconnects for Parallel Computers?" IEEE Symp. on High-Performance Interconnects Panel, August 2004.
4 W. Feng, "Making a Case for Efficient Supercomputing," ACM Queue, 1(7):54-64, October 2003.
5 S. Graham, M. Snir, and C. Patterson, eds., "Getting Up to Speed: The Future of Supercomputing," National Research Council, Committee on the Future of Supercomputing, National Academies Press, 2005.
6 D. Reed, "High-End Computing: The Challenge of Scale," Director's Colloquium, Los Alamos National Laboratory, May 2004.
7 J. Markoff and S. Lohr, "Intel's Huge Bet Turns Iffy," The New York Times, September 29, 2002.
8 W. Feng, M. Warren, and E. Weigle, "The Bladed Beowulf: A Cost-Effective Alternative to Traditional Beowulfs," 4th IEEE International Conference on Cluster Computing (IEEE Cluster), Chicago, IL, September 2002.
9 G. Johnson, "At Los Alamos, Two Visions of Supercomputing," The New York Times, June 25, 2002.
10 sss.lanl.gov ; At SC2001 in November, we demonstrated a small-scale 24-node prototype dubbed MetaBlade, running a simulation of a 10-million-body galaxy formation.
11 www.top500.org/list/2001/11
12 The original performance of Green Destiny on the Linpack benchmark was indeed "low performance" at about 68 Gflops. However, given that the Transmeta CPU was a hardware-software hybrid, we were able to optimize its floating-point performance (in system software) by 50%, resulting in a Linpack rating of 101 Gflops.
13 We note up-front that the comparison is an "apples-to-oranges" one given that the HPC systems are from different eras and have different architectures. The choice of HPC systems was motivated by the fact that we had complete configuration information of the systems and complete and unencumbered access to the systems to tune our n-body code.
14 www.para.tutics.tut.ac.jp/megascale/r_mproto.html
15 H. Nakashima, H. Nakamura, M. Sato, T. Boku, S. Matsuoka, D. Takahashi, and Y. Hotta, "MegaProto: A Low-Power and Compact Cluster for High-Performance Computing," IEEE Workshop on High-Performance, Power-Aware Computing (in conjunction with the IEEE Parallel & Distributed Processing Symposium), Denver, CO, April 2005.
16 www.orionmulti.com
17 IBM and Lawrence Livermore National Laboratory, "An Overview of the BlueGene/L Supercomputer," IEEE/ACM SC2002: High-Performance Networking & Computing Conference, Baltimore, MD, November 2002.
18 G. Almasi, R. Bellofatto, J. Brunheroto, C. Cascaval, J. G. Castanos, L. Ceze, P. Crumley, C. C. Erway, J. Gagliano, D. Lieber, X. Martorell, J. Moreira, A. Sanomiya, and K. Strauss, "An Overview of the Blue Gene/L System Software Organization," Euro-Par 2003 Conference, Klagenfurt, Austria, August 2003.
19 V. Bulatov, W. Cai, J. Fier, M. Hiratani, G. Hommes, T. Pierce, M. Tang, M. Rhee, K. Yates, and T. Arsenlis, "Scalable Line Dynamics in ParaDiS," IEEE/ACM SC2004: High-Performance Computing, Networking, and Storage Conference, Pittsburgh, PA, November 2004.
20 K. Davis, A. Hoisie, G. Johnson, D. Kerbyson, M. Lang, S. Pakin, and F. Petrini, "A Performance and Scalability Analysis of the BlueGene/L Architecture," IEEE/ACM SC2004: High-Performance Computing, Networking, and Storage Conference, Pittsburgh, PA, November 2004.
21 G. Almasi, S. Chatterjee, A. Gara, J. Gunnels, M. Gupta, A. Henning, J. Moreira, and B. Walkup, "Unlocking the Performance of the BlueGene/L Supercomputer," IEEE/ACM SC2004: High-Performance Computing, Networking, and Storage Conference, Pittsburgh, PA, November 2004.
22 We note that in addition to the differences in machine architectures and eras (which makes direct comparisons difficult) that power and space consumption do not scale linearly. So, the presented data should only be taken as ballpark figures.
23 None of the power numbers include the wattage needed for cooling. This means that for ASCI Red, ASCI White, and IBM Blue Gene/L that the power numbers would increase by a factor of 1.7 to 2.0 times. Furthermore, none of the space numbers include the extra floor(s) needed to cool the HPC systems.
24 Each Transmeta processor has a software layer, called code-morphing software, that dynamically morphs x86 instructions into VLIW instructions. This provides x86 software with the impression that it is being run on native x86 hardware.
25 X. Feng, R. Ge, and K. Cameron, "Power and Energy Profiling of Scientific Applications on Distributed Systems," 19th IEEE International Parallel & Distributed Processing Symposium, Denver, CO, April 2005.
26 V. Freeh, D. Lowenthal, F. Pan, and N. Kappiah, "Using Multiple Energy Gears in MPI Programs on a Power-Scalable Cluster," ACM Symposium on Principles and Practices of Parallel Programming (PPoPP�05), June 2005.
27 V. Freeh, D. Lowenthal, R. Springer, F. Pan, and N. Kappiah, "Exploring the Energy-Time Tradeoff in MPI Programs on a Power-Scalable Cluster," 19th IEEE International Parallel & Distributed Processing Symposium, Denver, CO, April 2005.
28 R. Ge, X. Feng, and K. Cameron, "Improvement of Power-Performance Efficiency for High-End Computing," 1st IEEE Workshop on High-Performance, Power-Aware Computing (in conjunction with the 19th IEEE International Parallel & Distributed Processing Symposium), Denver, CO, April 2005.
29 C. Hsu and U. Kremer, "The Design, Implementation, and Evaluation of a Compiler Algorithm for CPU Energy Reduction," ACM Conference on Programming Languages Design and Implementation (PLDI '03), June 2003.
30 W. Feng and C. Hsu, "The Origin and Evolution of Green Destiny," IEEE Cool Chips VII: An International Symposium on Low-Power and High-Speed Chips, Yokohama, Japan, April 2004.
31 W. Feng and C. Hsu, "Green Destiny and Its Evolving Parts," Innovative Supercomputer Architecture Award, 19th International Supercomputer Conference, Heidelberg, Germany, June 2004.
32 C. Hsu and W. Feng, "Effective Dynamic Voltage Scaling Through CPU-Boundedness Detection," 4th ACM Workshop on Power-Aware Computer Systems, Portland, OR, December 2004.
33 C. Hsu and W. Feng, "A Power-Aware Run-Time System for High-Performance Computing," ACM/IEEE SC2005: The International Conference on High-Performance Computing, Networking, and Storage, Seattle, WA, November 2005.
34 D. Grunwald, P. Levis, K. Farkas, C. Morrey, and M. Neufeld, "Policies for Dynamic Clock Scheduling," 4th Symposium on Operating System Design and Implementation (OSDI '00), Oct. 2000.
35 C. Hsu, W. Feng, and J. Archuleta, "Towards Efficient Supercomputing: A Quest for the Right Metric," 1st IEEE Workshop on High-Performance, Power-Aware Computing (in conjunction with the 19th International Parallel & Distributed Processing Symposium), Denver, CO, April 2005.
36 H. Nakashima, M. Sato, T. Boku, S. Matuoka, D. Takahashi, and Y. Hotta, "MegaProto: 1 Tflops/ 10kW Rack Is Feasible Even with Only Commodity Technology," ACM/IEEE SC2005: The International Conference on High-Performance Computing, Networking, and Storage, Seattle, WA, November 2005.
