PAPI provides tool designers and application engineers with a consistent interface and methodology for the use of low-level performance counter hardware found across the entire compute system (i.e. CPUs, GPUs, on/off-chip memory, interconnects, I/O system, energy/power, etc.). PAPI enables users to see, in near real time, the relationships between software performance and hardware events across the entire compute system.

Exa-PAPI builds on the latest PAPI project and we will extend it with:

  • Performance counter monitoring capabilities for new and advanced ECP hardware, and also software technologies (see 4.);
  • Fine-grained power management support;
  • Integration capabilities for exascale paradigms, such as task-based runtime systems that support dataflow programming models;
  • "Software-defined Events" that originate from the ECP software stack and are currently treated as black boxes (i.e., communication libraries, math libraries, task-based runtime systems, etc.).

The objective is to enable monitoring of both types of performance events---hardware- and software-related events---in a uniform way, through one consistent PAPI interface. That implies, 3rd-party tools and application developers have to handle only a single hook to PAPI in order to access all hardware performance counters in a system, including the new software-defined events.

Supported Architectures

  • AMD, power
  • ARM Cortex A8, A9, A15, ARM64
  • Cray: Gemini and Aries interconnects, power
  • IBM Blue Gene Series, Q: 5D-Torus, I/O system, EMON power/energy
  • IBM Power Series
  • Intel Westmere, Sandy|Ivy Bridge, Haswell, Broadwell, Skylake, KNC, Knights Landing
  • Intel KNC, Knights Landing power/energy
  • Intel RAPL (power/energy); power capping
  • InfiniBand
  • Lustre FS
  • NVIDIA Tesla, Kepler: CUDA support for multiple GPUs; PC Sampling
  • Virtual Environments: VMware, KVM

Sponsored By

Exascale Computing Project
National Nuclear Security Administration
The United States Department of Energy


Haidar, A., H. Jagode, A. YarKhan, P. Vaccaro, S. Tomov, and J. Dongarra, Power-aware Computing: Measurement, Control, and Performance Analysis for Intel Xeon Phi,” 2017 IEEE High Performance Extreme Computing Conference (HPEC'17), Best Paper Finalist, Waltham, MA, IEEE, September 2017.  (908.84 KB)
Haidar, A., H. Jagode, P. Vaccaro, S. Tomov, and J. Dongarra, Investigating Power Capping toward Energy-Efficient Scientific Applications,” Concurrency and Computation: Practice and Experience (CCPE): Special Issue on Power-Aware Computing 2017, Submitted.


  • Presented at the 2017 IEEE High Performance Extreme Computing Conference (HPEC '17) (September 12-14, 2017) in Waltham, MA, USA: Haidar, A., Jagode, H., YarKhan, A., Vaccaro, P., Tomov, S., Dongarra, J. "Power-aware Computing: Measurement, Control, and Performance Analysis for Intel Xeon Phi" (Best Paper Finalist)
  • Presented at the 11th Parallel Tools Workshop (September 11-12, 2017) in Dresden, Germany: Anthony Danalis, Heike Jagode, Hanumanth Hanumantharayappa, Jack Dongarra. "Counter Inspection Toolkit: Making Sense out of Hardware Performance Events"
  • Presented an update on the NVIDIA support in PAPI update at the CORAL Tools working group meeting on June 15, 2017.
  • Presented at the VI-HPS Tools Workshop (June 23, 2017): "EXA-PAPI: The exascale performance application programming interface"

ICL Team Members

Anthony Danalis

Jack Dongarra

Heike Jagode

Frank Winkler

Asim YarKhan

Exascale Computing Project

Exa-PAPI is part of ICL's involvment in the Exascale Computing Project (ECP). The ECP was established with the goals of maximizing the benefits of high-performance computing (HPC) for the United States and accelerating the development of a capable exascale computing ecosystem. Exascale refers to computing systems at least 50 times faster than the nation’s most powerful supercomputers in use today.

The ECP is a collaborative effort of two U.S. Department of Energy organizations – the Office of Science (DOE-SC) and the National Nuclear Security Administration (NNSA).