Innovative Computing Laboratory

Overview

The Performance Application Programming Interface (PAPI) offers a universal interface and methodology for gathering performance counter information from diverse hardware and software components. This includes major CPUs, GPUs, accelerators, interconnects, I/O systems, power interfaces, and even virtual cloud environments. Collaborations with industry leaders like AMD, Cray/HPE, IBM, Intel, NVIDIA, and others ensure seamless integration of PAPI with new architectures as they are introduced or come close to release. As the PAPI component architecture continues to expand, third-part performance tools interfacing with PAPI gain the automatic capability to measure data from these emerging architectures.

Two PAPI releases were shipped in 2023. PAPI 7.0.1 was officially announced on March 13th, 2023, and PAPI 7.1.0 on December 20th, 2023. These releases provide enhanced performance counter monitoring capabilities, extending support now to Intel Sapphire Rapids, AMD Zen4 CPUs, and ARM Neoverse V1 and V2 architectures. Notably, the ROCm component has been significantly refactored, enabling efficient monitoring of multiple GPUs in sampling mode and laying the foundation for future ROCm tools support. Additionally, the ROCm SMI component has been expanded to include GPU-to-GPU XGMI event support, offering a wider range of monitoring capabilities. Furthermore, the CUDA component has also undergone significant refactoring and now provides multi-thread and multi-GPU support.

Latest Releases

PAPI 7.2.0b
2024-08-30
PAPI 7.2.0b1 Release

PAPI 7.2.0b1 is now available as a beta release. This release introduces a new component, "rocp_sdk", which supports AMD GPUs/APUs through the ROCprofiler-SDK interface, currently still under development and testing. The release also includes general improvements to the PAPI code, enhancing both design and functionality, as well as various bug fixes.

Additional Major Changes are:

  • Preliminary support for AMD ROCprofiler-SDK events
  • AMD Zen5 L3 PMU support
  • AMD Zen5 core PMU support
  • Preset support for Zen5
  • Preset support for Ice Lake ICL
  • Basic support for the RISC-V architecture (no events yet)
  • Initial heterogenous CPU support: Alderlake and Raptorlake can now enumerate events for both Power and Efficiency cores on heterogeneous systems
  • Intel AlderLake Gracemont (E-Core) core PMU support
  • Intel AlderLake Goldencove (P-Core) core PMU support
  • Intel Raptorlake PMU support: Enables support for Raptorlake, Raptorlake P, Raptorlake S
  • Intel GraniteRapids core PMU support
  • Intel SapphireRapids uncore PMU support for:
    • Coherence and Home Agent (CHA)
    • Ultra Path Interconnect PMU (UPI)
    • memory controller PMU (IMC)
  • Intel IcelakeX uncore PMU support for:
    • Mesh to IIO PMU (M2PCIE)
    • UBOX PMU (UBOX)
    • Mesh to UPI PMU (M3UPI)
    • Ultra Path Interconnect PMU (UPI)
    • Power Control unit PMU (PCU)
    • Mesh to Memory PMU (M2M)
    • PCIe IIO Ring Port PMU (IRP)
    • PCIe I/O controller PMU (IIO)
    • memory controller PMU (IMC)
    • Coherency and Home Agent (CHA)
  • Sysdetect: support for ARM Neoverse V2
  • SDE: support for ntv_code_to_info functionality
  • Removed the obsolete bundled perfctr and libpfm-3.y code

Acknowledgements

This release is the result of efforts from many people. The PAPI team would like to express special Thanks to Vince Weaver, Stephane Eranian (for libpfm4), William Cohen, Steve Kaufmann, Peinan Zhang, Rashawn Knapp and Phil Mucci.

Download papi-7.2.0b1.tar.gz

To verify the integrity of the download, check the MD5 hash md5sum papi-7.2.0b1.tar.gz:

e81521450fab24e7d49c952bfc347935

PAPI 7.2.0b

Papers

Barry, D., H. Jagode, A. Danalis, and J. Dongarra, Memory Traffic and Complete Application Profiling with PAPI Multi-Component Measurements,” 2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), St. Petersburg, Florida, IEEE, August 2023.  (1.81 MB)
Danalis, A., and H. Jagode, Performance Application Programming Interface,” Accelerated Computing with HIP: Sun, Baruah and Kaeli, December 2022.
Dongarra, J., H. Jagode, A. Danalis, D. Barry, and V. Weaver, Performance Application Programming Interface for Extreme-Scale Environments (PAPI-EX) (Poster) , Seattle, WA, 2020 NSF Cyberinfrastructure for Sustained Scientific Innovation (CSSI) Principal Investigator Meeting, 20 2020.  (2.53 MB)
Barry, D., A. Danalis, and H. Jagode, Effortless Monitoring of Arithmetic Intensity with PAPI's Counter Analysis Toolkit,” 13th International Workshop on Parallel Tools for High Performance Computing, Dresden, Germany, Springer International Publishing, September 2020.  (738.47 KB)
Jagode, H., A. Danalis, and D. Genet, Roadmap for Refactoring Classic PAPI to PAPI++: Part II: Formulation of Roadmap Based on Survey Results,” PAPI++ Working Notes, no. 2, ICL-UT-20-09: Innovative Computing Laboratory, University of Tennessee, July 2020.  (763.75 KB)
Jagode, H., A. Danalis, and J. Dongarra, Exa-PAPI: The Exascale Performance API with Modern C++ , Houston, TX, 2020 Exascale Computing Project Annual Meeting, February 2020.  (556.78 KB)
Jagode, H., and A. Danalis, PULSE: PAPI Unifying Layer for Software-Defined Events (Poster) , Seattle, WA, 2020 NSF Cyberinfrastructure for Sustained Scientific Innovation (CSSI) Principal Investigator Meeting, February 2020.  (1.86 MB)
Winkler, F., Redesigning PAPI's High-Level API,” Innovative Computing Laboratory Technical Report, no. ICL-UT-20-03: University of Tennessee, February 2020.  (356.41 KB)
Jagode, H., A. Danalis, and J. Dongarra, Formulation of Requirements for New PAPI++ Software Package: Part I: Survey Results,” PAPI++ Working Notes, no. 1, ICL-UT-20-02: Innovative Computing Laboratory, University of Tennessee Knoxville, January 2020.  (1.49 MB)
Jagode, H., A. Danalis, H. Anzt, and J. Dongarra, PAPI Software-Defined Events for in-Depth Performance Analysis,” The International Journal of High Performance Computing Applications, vol. 33, issue 6, pp. 1113-1127, November 2019.  (442.39 KB)
Davis, J., T. Gao, S. Chandrasekaran, H. Jagode, A. Danalis, P. Balaji, J. Dongarra, and M. Taufer, Characterization of Power Usage and Performance in Data-Intensive Applications using MapReduce over MPI,” 2019 International Conference on Parallel Computing (ParCo2019), Prague, Czech Republic, September 2019.
Jagode, H., A. Danalis, and J. Dongarra, What it Takes to keep PAPI Instrumental for the HPC Community,” 1st Workshop on Sustainable Scientific Software (CW3S19), Collegeville, Minnesota, July 2019.  (50.57 KB)
Danalis, A., H. Jagode, T. Herault, P. Luszczek, and J. Dongarra, Software-Defined Events through PAPI,” 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Rio de Janeiro, Brazil, IEEE, May 2019.  (446.41 KB)
Danalis, A., H. Jagode, H. Hanumantharayappa, S. Ragate, and J. Dongarra, Counter Inspection Toolkit: Making Sense out of Hardware Performance Events,” 11th International Workshop on Parallel Tools for High Performance Computing, Dresden, Germany, Cham, Switzerland: Springer, February 2019.  (216.39 KB)
Haidar, A., H. Jagode, P. Vaccaro, A. YarKhan, S. Tomov, and J. Dongarra, Investigating Power Capping toward Energy-Efficient Scientific Applications,” Concurrency Computation: Practice and Experience, vol. 2018, issue e4485, pp. 1-14, April 2018.  (1.2 MB)
Parker, S., J. Mellor-Crummey, D. H. Ahn, H. Jagode, H. Brunst, S. Shende, A. D. Malony, D. DelSignore, R. Tschuter, R. Castain, et al., Performance Analysis and Debugging Tools at Scale,” Exascale Scientific Applications: Scalability and Performance Portability: Chapman & Hall / CRC Press, pp. 17-50, November 2017.
Haidar, A., H. Jagode, A. YarKhan, P. Vaccaro, S. Tomov, and J. Dongarra, Power-aware Computing: Measurement, Control, and Performance Analysis for Intel Xeon Phi,” 2017 IEEE High Performance Extreme Computing Conference (HPEC'17), Best Paper Finalist, Waltham, MA, IEEE, September 2017.  (908.84 KB)
Jagode, H., A. YarKhan, A. Danalis, and J. Dongarra, Power Management and Event Verification in PAPI,” Tools for High Performance Computing 2015: Proceedings of the 9th International Workshop on Parallel Tools for High Performance Computing, September 2015, Dresden, Germany, Dresden, Germany, Springer International Publishing, pp. pp. 41-51, 2016.  (565.14 KB)
McCraw, H., J. Ralph, A. Danalis, and J. Dongarra, Power Monitoring with PAPI for Extreme Scale Architectures and Dataflow-based Programming Models,” 2014 IEEE International Conference on Cluster Computing, no. ICL-UT-14-04, Madrid, Spain, IEEE, September 2014.  (3.45 MB)
Nelson, J., Analyzing PAPI Performance on Virtual Machines,” VMWare Technical Journal, vol. Winter 2013, January 2014.
Nelson, J., Analyzing PAPI Performance on Virtual Machines,” ICL Technical Report, no. ICL-UT-13-02, August 2013.  (437.37 KB)
McCraw, H., D. Terpstra, J. Dongarra, K. Davis, and R. Musselman, Beyond the CPU: Hardware Performance Counter Monitoring on Blue Gene/Q,” International Supercomputing Conference 2013 (ISC'13), Leipzig, Germany, Springer, June 2013.  (624.58 KB)
Weaver, V., D. Terpstra, and S. Moore, Non-Determinism and Overcount on Modern Hardware Performance Counter Implementations,” 2013 IEEE International Symposium on Performance Analysis of Systems and Software, Austin, TX, IEEE, April 2013.  (307.24 KB)
Weaver, V., D. Terpstra, H. McCraw, M. Johnson, K. Kasichayanula, J. Ralph, J. Nelson, P. Mucci, T. Mohan, and S. Moore, PAPI 5: Measuring Power, Energy, and the Cloud , Austin, TX, 2013 IEEE International Symposium on Performance Analysis of Systems and Software, April 2013.  (78.39 KB)
Weaver, V. M., M. Johnson, K. Kasichayanula, J. Ralph, P. Luszczek, D. Terpstra, and S. Moore, Measuring Energy and Power with PAPI,” International Workshop on Power-Aware Systems and Architectures, Pittsburgh, PA, September 2012.  (146.79 KB)
Johnson, M., H. McCraw, S. Moore, P. Mucci, J. Nelson, D. Terpstra, V. M. Weaver, and T. Mohan, PAPI-V: Performance Monitoring for Virtual Machines,” CloudTech-HPC 2012, Pittsburgh, PA, September 2012.  (2.69 MB)
Kasichayanula, K., D. Terpstra, P. Luszczek, S. Tomov, S. Moore, and G. D. Peterson, Power Aware Computing on GPUs,” SAAHPC '12 (Best Paper Award), Argonne, IL, July 2012.  (658.06 KB)
Malony, A. D., S. Biersdorff, S. Shende, H. Jagode, S. Tomov, G. Juckeland, R. Dietrich, D. Poole, and C. Lamb, Parallel Performance Measurement of Heterogeneous Parallel Systems with GPUs,” International Conference on Parallel Processing (ICPP'11), Taipei, Taiwan, ACM, September 2011.  (1.41 MB)
Kasichayanula, K., H. You, S. Moore, S. Tomov, H. Jagode, and M. Johnson, Power-aware Computing on GPGPUs , Gatlinburg, TN, Fall Creek Falls Conference, Poster, September 2011.  (2.89 MB)
Moore, S., and J. Ralph, User-Defined Events for Hardware Performance Monitoring,” Procedia Computer Science, vol. 4: Elsevier, pp. 2096-2104, May 2011.  (361.76 KB)
Weaver, V. M., and J. Dongarra, Can Hardware Performance Counters Produce Expected, Deterministic Results?,” 3rd Workshop on Functionality of Hardware Performance Monitoring, Atlanta, GA, December 2010.  (392.71 KB)
Terpstra, D., H. Jagode, H. You, and J. Dongarra, Collecting Performance Data with PAPI-C,” Tools for High Performance Computing 2009, 3rd Parallel Tools Workshop, Dresden, Germany, Springer Berlin / Heidelberg, pp. 157-173, May 2010.  (4.45 MB)
Mucci, P., D. Ahlin, J. Danielsson, P. Ekman, and L. Malinowski, PerfMiner: Cluster-Wide Collection, Storage and Presentation of Application Level Hardware Performance Data,” European Conference on Parallel Processing (Euro-Par 2005), Monte de Caparica, Portugal, Springer, September 2005.  (205.45 KB)
Moore, S., D. Cronk, F. Wolf, A. Purkayastha, P. J. Teller, R. Araiza, G. Aguilera, and J. Nava, Performance Profiling and Analysis of DoD Applications using PAPI and TAU,” Proceedings of DoD HPCMP UGC 2005, Nashville, TN, IEEE, June 2005.  (322.56 KB)
Andersson, U., and P. Mucci, Analysis and Optimization of Yee_Bench using Hardware Performance Counters,” Proceedings of Parallel Computing 2005 (ParCo), Malaga, Spain, January 2005.  (72.27 KB)
Dongarra, J., S. Moore, P. Mucci, K. Seymour, and H. You, Accurate Cache and TLB Characterization Using Hardware Counters,” International Conference on Computational Science (ICCS 2004), Krakow, Poland, Springer, June 2004.  (167.1 KB)
Yi, Q., K. Kennedy, H. You, K. Seymour, and J. Dongarra, Automatic Blocking of QR and LU Factorizations for Locality,” 2nd ACM SIGPLAN Workshop on Memory System Performance (MSP 2004), Washington, DC, ACM, June 2004.  (212.77 KB)
Mucci, P., J. Dongarra, R. Kufrin, S. Moore, F. Song, and F. Wolf, Automating the Large-Scale Collection and Analysis of Performance,” 5th LCI International Conference on Linux Clusters: The HPC Revolution, Austin, Texas, May 2004.  (511.6 KB)
Wolf, F., and B. Mohr, Hardware-Counter Based Automatic Performance Analysis of Parallel Programs,” Advances in Parallel Computing, vol. 13, Dresden, Germany, Elsevier, pp. 753-760, January 2004, 2003.
Dongarra, J., A. D. Malony, S. Moore, P. Mucci, and S. Shende, Performance Instrumentation and Measurement for Terascale Systems,” ICCS 2003 Terascale Workshop, Melbourne, Australia, Springer, Berlin, Heidelberg, June 2003.  (5.36 MB)
Dongarra, J., K. London, S. Moore, P. Mucci, D. Terpstra, H. You, and M. Zhou, Experiences and Lessons Learned with a Portable Interface to Hardware Performance Counters,” PADTAD Workshop, IPDPS 2003, Nice, France, IEEE, April 2003.  (432.57 KB)
Moore, S., A Comparison of Counting and Sampling Modes of Using Performance Monitoring Hardware,” International Conference on Computational Science (ICCS 2002), Amsterdam, Netherlands, Springer, April 2002.  (122 KB)
Moore, S., D. Cronk, K. London, and J. Dongarra, Review of Performance Analysis Tools for MPI Parallel Programs,” European Parallel Virtual Machine / Message Passing Interface Users’ Group Meeting, Lecture Notes in Computer Science 2131, Greece, Springer Verlag, Berlin, pp. 241-248, September 2001.  (39.61 KB)
London, K., J. Dongarra, S. Moore, P. Mucci, K. Seymour, and T.. Spencer, End-user Tools for Application Performance Analysis, Using Hardware Counters,” International Conference on Parallel and Distributed Computing Systems, Dallas, TX, August 2001.  (306.54 KB)
London, K., S. Moore, P. Mucci, K. Seymour, and R. Luczak, The PAPI Cross-Platform Interface to Hardware Performance Counters,” Department of Defense Users' Group Conference Proceedings, Biloxi, Mississippi, June 2001.  (328.56 KB)
Dongarra, J., K. London, S. Moore, P. Mucci, and D. Terpstra, Using PAPI for Hardware Performance Monitoring on Linux Systems,” Conference on Linux Clusters: The HPC Revolution, Urbana, Illinois, Linux Clusters Institute, June 2001.  (422.35 KB)

Presentations

Dongarra, J., H. Jagode, A. Danalis, D. Barry, and V. Weaver, Performance Application Programming Interface for Extreme-Scale Environments (PAPI-EX) (Poster) , Seattle, WA, 2020 NSF Cyberinfrastructure for Sustained Scientific Innovation (CSSI) Principal Investigator Meeting, 20 2020.  (2.53 MB)
Jagode, H., A. Danalis, and J. Dongarra, Exa-PAPI: The Exascale Performance API with Modern C++ , Houston, TX, 2020 Exascale Computing Project Annual Meeting, February 2020.  (556.78 KB)
Jagode, H., and A. Danalis, PULSE: PAPI Unifying Layer for Software-Defined Events (Poster) , Seattle, WA, 2020 NSF Cyberinfrastructure for Sustained Scientific Innovation (CSSI) Principal Investigator Meeting, February 2020.  (1.86 MB)
Danalis, A., H. Jagode, and J. Dongarra, PAPI's new Software-Defined Events for in-depth Performance Analysis , Dresden, Germany, 13th Parallel Tools Workshop, September 2019.  (3.14 MB)
Danalis, A., H. Jagode, and J. Dongarra, Does your tool support PAPI SDEs yet? , Tahoe City, CA, 13th Scalable Tools Workshop, July 2019.  (3.09 MB)
Jagode, H., A. Danalis, and J. Dongarra, What it Takes to keep PAPI Instrumental for the HPC Community , Collegeville, MN, The 2019 Collegeville Workshop on Sustainable Scientific Software (CW3S19), July 2019.  (3.29 MB)
Danalis, A., H. Jagode, and J. Dongarra, Is your scheduling good? How would you know? , Bordeaux, France, 14th Scheduling for Large Scale Systems Workshop, June 2019.  (2.5 MB)
Danalis, A., H. Jagode, D. Barry, and J. Dongarra, Understanding Native Event Semantics , Knoxville, TN, 9th JLESC Workshop, April 2019.  (2.33 MB)
Jagode, H., A. Danalis, and J. Dongarra, PAPI's New Software-Defined Events for In-Depth Performance Analysis , Lyon, France, CCDSC 2018: Workshop on Clusters, Clouds, and Data for Scientific Computing, September 2018.
Danalis, A., H. Jagode, and J. Dongarra, Software-Defined Events through PAPI for In-Depth Analysis of Application Performance , Basel, Switzerland, 5th Platform for Advanced Scientific Computing Conference (PASC18), July 2018.
Danalis, A., H. Jagode, and J. Dongarra, PAPI: Counting outside the Box , Barcelona, Spain, 8th JLESC Meeting, April 2018.
Weaver, V., D. Terpstra, H. McCraw, M. Johnson, K. Kasichayanula, J. Ralph, J. Nelson, P. Mucci, T. Mohan, and S. Moore, PAPI 5: Measuring Power, Energy, and the Cloud , Austin, TX, 2013 IEEE International Symposium on Performance Analysis of Systems and Software, April 2013.  (78.39 KB)
Kasichayanula, K., H. You, S. Moore, S. Tomov, H. Jagode, and M. Johnson, Power-aware Computing on GPGPUs , Gatlinburg, TN, Fall Creek Falls Conference, Poster, September 2011.  (2.89 MB)

ICL Team Members

Daniel Barry
Graduate Research Assistant
Treece Burgess
Research Associate II
Giuseppe Congiu
Visiting Scholar
Anthony Danalis
Research Assistant Professor
Heike Jagode
Research Associate Professor
Dong Jun Woun
Graduate Research Assistant