News and Announcements

HPC Learning Webinar

On Tuesday, February 7, at 12pm Eastern Standard Time, Jack Dongarra will be speaker for the Association for Computing Machinery (ACM) Learning Center live one-hour webinar titled “Current Trends in High Performance Computing and Challenges for the Future.”

Jack will discuss the ways in which HPC has changed during the last decade and where it appears to be headed. He will also delve into how the effective and reliable use of dynamic, distributed, and parallel computing environments will call for the development of a new generation of software libraries and algorithms.

Registration is required to attend the webinar.

Conference Reports

ECP PI Meeting

Principal investigators for the US Department of Energy’s Exascale Computing Project (ECP) met on November 29 at Argonne National Laboratory (Lemont, IL) to make presentations and consider the ECP’s big picture.

ICL’s George Bosilca, Jack Dongarra, and Heike Jagode attended this gathering, which was limited to one representative per project.

Quad charts for each of the funded projects aided the identification of topics and individuals relative to potential collaborations, interdependencies, and networking. Following the presentation of the charts, multiple parallel group discussions during days two and three allowed the participants to explore ways in which their projects connect to other portions of the ECP.

PEEKS Kick-off Meeting

ICL’s Ichitaro Yamazaki and Hartwig Anzt traveled to Albuquerque, NM, for a visit with collaborators at Sandia National Laboratories (SNL) on December 5 to discuss milestones and the initial steps involved in the Production-ready, Exascale-enabled Krylov Solvers (PEEKS) project, which is part of the US Department of Energy’s Exascale Computing Project (ECP).

Main SNL collaborators Mike Heroux, Erik Boman, Mark Hoemmen, and Siva Rajamanickam, as well as other SNL personnel, also attended the meeting. Ichi and Hartwig’s presentations provided both introductory and update information concerning PEEKS.

As the trip drew to a close, the ICLers had Friday morning off, so they took the opportunity to visit some local attractions. Hartwig toured the National Museum of Nuclear Science & History, while Ichi rode the Sandia Peak tramway to the top of the Sandia Mountains.

TESSE Workgroup Meeting

Members of the Task-based Environment for Scientific Simulation at Extreme Scale (TESSE) project workgroup met in the Stony Brook University of New York (SUNY) Global Center in New York City on December 12 to discuss progress.

TESSE uses an application-driven design to create a general-purpose, production-quality software framework that attacks the twin challenges of programmer productivity and portable performance for advanced scientific applications on the massively parallel, hybrid, manycore systems existing today and those anticipated for the future.

George Bosilca, Thomas Herault, and Damien Genet represented ICL at the meeting. Attending from the SUNY Institute for Advanced Computational Science (IACS) were Robert J. Harrison, W. Scott Thornton, and Mohammed Mahdi Javanmard; and from Virginia Tech, Department of Chemistry, Edward F. Valeev. In addition, invited to speak on the first day were Barbara Chapman of SUNY IACS and Brookhaven National Laboratory, and Dmitry Lyakh of Oak Ridge National Laboratory (ORNL).

Barbara discussed OpenMP and programming models. Dmitry presented on his work at ORNL, the agnostic design for performance portability.

Also during day one of the meeting, George, Thomas, and Damien gave a presentation on the Parallel Runtime Scheduling and Execution Controller (PaRSEC) project. PaRSEC is a generic framework for architecture-aware scheduling and management of microtasks on distributed manycore heterogeneous architectures. The presentation focused on the programming model, interfaces, how to interact with the runtime, and the integration of what is known as the SUMMA operation.

Robert gave a talk on the overall TESSE project, and Edward discussed his code TiledArray, a modern C++ library that is scalable to tens of thousands of processor cores.

Open discussions were organized around two topics: the scientific application community and the interoperability of the programming model and runtime systems.

Day two involved a discussion of the low-level aspects of the project.

IPDPS 2017 Review

dsc01804

Lead reviewers, including ICL’s Piotr Luszczek, and the organization committee of the IEEE International Parallel and Distributed Processing Symposium (IPDPS) met on January 5 in Rosemont, IL, near Chicago, to select papers for the conference, which is set to take place May 29 to June 2, 2017, in Buena Vista Palace Hotel, Orlando, FL.

Three tiers of reviewers overall were involved in the preparatory activities this year for IPDPS, with the lead review tier making the final decisions. The lead review team was diverse, encompassing academia, industry, and the national laboratories. Piotr participated in the application and software tracks.

ECP CEED Meeting

12a97ef8-d88e-11e6-8498-dad5ff5b4e27

31759D, CEED Workshop group at Argonne

On January 10 through 12, the kick-off meeting of the Exascale Computing Project (ECP) Center for Efficient Exascale Discretizations (CEED) convened at Argonne National Laboratory to familiarize the participants from CEED’s collaborating institutions with relevant software and technologies.

The ECP CEED project will develop the next-generation discretization software and algorithms that will enable a wide range of US Department of Energy and National Nuclear Security Administration applications to run on future hardware.

Twenty-eight people from all the institutions involved in CEED attended the kick-off meeting. The CEED team encompasses Lawrence Livermore National Laboratory, which is leading the project, and Argonne National Laboratory, five universities, and outside contributors. Azzam Haidar and Stan Tomov represented ICL.

Stan described the meeting as a very useful jumpstart for the project—from getting to know the variety of software products and technologies already developed, to learning more about the applications that will be targeted during the first year.

Among the already-developed software and technologies were PUMI, Holmes, Nek, MFEM, MAGMA, OCCA, PETSc, Hypre, and Spack. Receiving attention the first year will be ExaSMR and MARBL.

Stan said the meeting also afforded the opportunity to synchronize and start the work on initial milestones, collaborations, and interactions.

As a collaborator in CEED, ICL will be instrumental in identifying, developing, and optimizing tensor contractions that are essential building blocks for the applications. We will also take an active role in co-designing APIs with the LLNL scientists, external partners on the team, and vendors, and deliver a high-performance tensor contractions package through our MAGMA library.

At the meeting, Stan and Azzam presented the MAGMA library and its batched linear algebra capabilities. ICL will build on the MAGMA batched capabilities and be instrumental in developing a MAGMA tensor contractions engine to power main computational components of the high-order methods targeted in CEED. In addition, we will lead the hardware thrust in CEED, which will entail handling interactions with the exascale hardware vendors.

The types of applications that the CEED project will enhance include global climate modeling, turbulent combustion in internal combustion engines, nuclear reactor modeling, additive manufacturing, subsurface flow, and national security.

MPHC Program Meeting

dsc01816

Trieste, a beautiful city on the Adriatic coast of Italy, was the location January 10–13 for the Master in High Performance Computing (MPHC) program meeting, which involved extensive teaching and labs with hands-on exercises for six hours each day.

The activities covered computer architecture and hardware for HPC and upcoming exascale systems. ICL’s Piotr Luszczek taught asynchronous CUDA programming and gave a full rundown of overview, design, and implementation details of all the linear algebra libraries we produce and release at ICL.

The MHPC program is for graduate students from the International School for Advanced Studies (Scuola Internazionale Superiore di Studi Avanzati) and the International Centre for Theoretical Physics. Program links and materials are available here.

Piotr notes that Trieste is only a stone’s throw away from Slovenia and the Republic of Croatia.

ISTC for Big Data Tech Meeting

On January 25 at the Intel Science and Technology Center (ISTC) for Big Data technical meeting in Boston, MA, Piotr Luszczek and Thomas Herault discussed the integration of the linear algebra algorithms developed at ICL into the SciDB computational database management system. SciDB allows the mining of insights from diverse data sources such as genomics, clinical, and financial markets.

The meeting also featured Michael Stonebraker of the Computer Science and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute of Technology (MIT), Tim Mattson and Jeff Pankhurst of Intel, and Jason Kitchen and James McQuestion of the Paradigm4 company.

Inaugural ECP Annual Meeting

The first “All Hands” meeting to launch the US Department of Energy’s Exascale Computing Project (ECP) occurred at the Knoxville Convention Center from January 31 to February 2. It brought together a significant portion of the ECP team—between 450 and 500 people. ICL sent fifteen people to represent its seven ECP subprojects.

The goal of the meeting was to establish a shared understanding of the ECP’s overarching vision, with particular attention to fostering the ECP community, making progress on subproject integration, and promoting an ethos of holistic codesign among the different ECP groups—science applications, software technologies, hardware technologies, and facility operations.

The ICL participants had three objectives in mind as they fanned out across the breakout sessions led by the ECP groups: determine the needs of different science applications for ICL software technology, ascertain how best to integrate ICL software into the ECP software ecosystem, and understand how ICL would need to coordinate with the different DOE laboratories in the testing and deployment of the software its ECP projects will produce.

Recent Releases

2017 ICL Annual Report

icl-report-2017

ICL’s tradition of presenting an annual report continues. For brief research project summaries—including new endeavors on the horizon for 2017—an overview of ICL’s history, some of the highlights of the year that just ended, and a glimpse at the people and external associates behind the organization’s success, please download and review a copy.

LAPACK 3.7.0 Released

LAPACK 3.7.0 is now available. The Linear Algebra PACKage (LAPACK) is a widely used library for efficiently solving dense linear algebra problems. ICL has been a major contributor to the development and maintenance of the package since its inception. LAPACK is sequential, relies on the BLAS library, and benefits from the multicore BLAS library.

LAPACK 3.7.0 has TSQR for least-square, communication-avoiding symmetric-indefinite factorization with Aasen’s triangular tridiagonalization, Level 3 BLAS use by the rook pivoting form of LDL, the reduction to tridiagonal routine using the two-stage algorithm, and the reduction to bidiagonal routine using the two-stage algorithm.

Since 2011, LAPACK has included LAPACKE, a native C interface for LAPACK developed in collaboration with INTEL, which provides NAN check and automatic workspace allocation.

PLASMA 17.01 Released

PLASMA 17.01 has been released. This is the first release of PLASMA based solely on OpenMP multithreading directives.

This release focuses on linear systems and least squares and includes the following:

  • Linear systems solvers (LU, Cholesky)
  • Mixed precision iterative refinement (LU, Cholesky)
  • Matrix norms (fast, multithreaded)
  • A full set of multithreaded BLAS 3
  • Matrix inversion (LU, Cholesky)
  • Least squares with QR/LQ (including fast tall and skinny routines)
  • Band linear solvers (LU, Cholesky)

This release includes “support” for the Knights Landing Xeon Phi
(in reality, PLASMA works out of the box on the KNL).

In all four precisions, the current PLASMA contains the following:

  • 115,000 lines of code, plus another 37,000 lines of testing code
  • 400 API functions

Besides porting PLASMA from QUARK to OpenMP, the teams at UTK and Manchester also developed the following:

  • New LU factorization with internally blocked and multithreaded panel factorization
  • New QR/LQ factorization routines offering a couple of different tree reduction patterns
  • New mixed precision iterative refinement routines—reimplemented to follow LAPACK’s procedure
  • New matrix norms routines—reimplemented to allow for increased multithreading
  • Band linear systems routines for LU and Cholesky factorizations and solves
  • A new testing/timing harness—to allow for easy inner/outer iteration over the parameter space
  • A new build system—heavily borrowing from MAGMA
  • A new precision generation system—heavily borrowing from MAGMA

Thanks to all the people from UTK and Manchester for their hard work. Special thanks to those who went beyond the call of duty to deliver new exciting capabilities.

Get the code by cloning the Mercurial repository on Bitbucket:
hg clone ssh://hg@bitbucket.org/icl/plasma
or by downloading the tarball.

Stay in touch by joining the plasma-user forum. (Apply to join the group.)

Open MPI 2.0.2 Released

The Open MPI team, representing a consortium of research, academic, and industry partners, is pleased to announce the release of Open MPI version 2.0.2., which includes a variety of bug fixes and some performance fixes.

All users are encouraged to upgrade to version 2.0.2 when possible. It can be downloaded from the main Open MPI website.

Interview

Stephen Wood Then

Stephen Wood

Where are you from originally?

I was born and raised in Miami, FL.

Can you summarize your educational background?

For high school I was fortunate to have attended the Maritime, Atmospheric Science, and Technology Academy, where my interests in physics, fluid dynamics, and mathematics flourished. I graduated from Florida International University (FIU) cum laude with a bachelor of science degree in mechanical engineering (ME) in 2010 and a master of science degree in ME in 2011. While at FIU, my research was supported by the Ronald E. McNair Post-baccalaureate Achievement Program and US Department of Energy/FIU Science & Technology WorkForce Development Initiative (DOE Fellows Program). Through the DOE Fellows Program, I participated in two internships (Higher Education Research Experiences [HERE] and Research Alliance in Math and Science [RAMS]) at Oak Ridge National Laboratory (ORNL) in the Computer Science and Mathematics Directorate. The world-class interactions of these internships prompted me to seek opportunities to pursue a doctorate program closely associated with ORNL. I was fortunate to be a Tennessee State Collaborative on Reforming Education (SCORE) Energy Scholar in the inaugural cohort of Bredesen Center students at the University of Tennessee, Knoxville. I earned a doctor of philosophy in energy science and engineering (ESE) with an Interdisciplinary Graduate Minor in Computer Science (IGMCS) in 2016. The interdisciplinary ESE degree program offered at the Bredesen Center and the IGMCS offered by ICL have enabled me to be a productive and collaborative postdoc at the Joint Institute for Computational Sciences.

Tell us how you first learned about ICL.

In my undergraduate courses at FIU, professors discussed BLAS and LAPACK with sincere appreciation as they imparted the importance of verification and benchmarking efforts in engineering software development.

What made you want to work for ICL?

The opportunity to “pay it forward” by contributing to the development of software libraries that current and future engineers will utilize.

What are you working on while at ICL?

I’m collaborating with Hartwig Anzt to continue the development of MAGMA Sparse for computational engineering applications deployed on modern architectures.

If you weren’t working at ICL, where would you like to be working and why?

I’d like to be working at one of the national laboratories or NASA as part of a multidisciplinary team tackling hard problems in the national and global interest.

What are your interests/hobbies outside of work?

I enjoy the outdoors, trail running, rock climbing, swimming, biking, rowing, sailing, canoeing, and hiking. Facilitating my two-and-half-year-old daughter’s early experiences of these activities is one of the greatest joys I’ve ever had.

Tell us something about yourself that might surprise people.

I earned a first-degree black belt in Shorin-Ryu Karate in 2010.

Recent Papers

  1. Jagode, H., A. Danalis, and J. Dongarra, Accelerating NWChem Coupled Cluster through Dataflow-Based Execution,” The International Journal of High Performance Computing Applications, pp. 1–13, January 2017. DOI: 10.1177/1094342016672543  (4.07 MB)
  2. Anzt, H., J. Dongarra, G. Flegar, and E. S. Quintana-Orti, Batched Gauss-Jordan Elimination for Block-Jacobi Preconditioner Generation on GPUs,” Proceedings of the 8th International Workshop on Programming Models and Applications for Multicores and Manycores, New York, NY, USA, ACM, pp. 1–10, February 2017. DOI: 10.1145/3026937.3026940  (552.62 KB)
  3. Haidar, A., A. Abdelfattah, S. Tomov, and J. Dongarra, High-performance Cholesky Factorization for GPU-only Execution,” Proceedings of the General Purpose GPUs (GPGPU-10), Austin, TX, ACM, February 2017. DOI: 10.1145/3038228.3038237  (872.18 KB)

Recent Conferences

  1. JAN
    Piotr Luszczek
    Piotr
    Piotr Luszczek
  2. JAN
    ECP CEED Meeting Argonne, Illinois
    Azzam Haidar
    Azzam
    Stanimire Tomov
    Stan
    Azzam Haidar, Stanimire Tomov
  3. JAN
    -
    Open MPI Developers Meeting San Jose, California
    George Bosilca
    George
    George Bosilca
  4. JAN
    -
    ECP All-Hands Meeting Knoxville, Tennessee
    Anthony Danalis
    Anthony
    Asim YarKhan
    Asim
    Aurelien Bouteiller
    Aurelien
    Azzam Haidar
    Azzam
    Damien Genet
    Damien
    George Bosilca
    George
    Hartwig Anzt
    Hartwig
    Heike Jagode
    Heike
    Ichitaro Yamazaki
    Ichitaro
    Jack Dongarra
    Jack
    Jakub Kurzak
    Jakub
    Mark Gates
    Mark
    Piotr Luszczek
    Piotr
    Stanimire Tomov
    Stan
    Terry Moore
    Terry
    Thomas Herault
    Thomas
    Anthony Danalis, Asim YarKhan, Aurelien Bouteiller, Azzam Haidar, Damien Genet, George Bosilca, Hartwig Anzt, Heike Jagode, Ichitaro Yamazaki, Jack Dongarra, Jakub Kurzak, Mark Gates, Piotr Luszczek, Stanimire Tomov, Terry Moore, Thomas Herault
  5. FEB
    Hartwig Anzt
    Hartwig
    Hartwig Anzt
  6. FEB
    -
    GPGPU-10 Workshop Austin, Texas
    Ahmad Abdelfattah Ahmad
    Ahmad
    Ahmad Abdelfattah Ahmad
  7. FEB
    -
    SIAM CSE + Batched BLAS Atlanta, Georgia
    Ahmad Abdelfattah Ahmad
    Ahmad
    Anthony Danalis
    Anthony
    Aurelien Bouteiller
    Aurelien
    Azzam Haidar
    Azzam
    Dongarra
    Dongarra
    George Bosilca
    George
    Hartwig Anzt
    Hartwig
    Ichitaro Yamazaki
    Ichitaro
    Jack Dongarra
    Jack
    Jakub Kurzak
    Jakub
    Mark Gates
    Mark
    Piotr Luszczek
    Piotr
    Stanimire Tomov
    Stan
    Stephen Wood
    Stephen
    Thomas Herault
    Thomas
    Yaohung  Tsai
    Mike
    Yves Robert
    Yves
    Ahmad Abdelfattah Ahmad, Anthony Danalis, Aurelien Bouteiller, Azzam Haidar, Dongarra, George Bosilca, Hartwig Anzt, Ichitaro Yamazaki, Jack Dongarra, Jakub Kurzak, Mark Gates, Piotr Luszczek, Stanimire Tomov, Stephen Wood, Thomas Herault, Yaohung Tsai, Yves Robert
  8. FEB
    -
    Damien Genet
    Damien
    George Bosilca
    George
    Damien Genet, George Bosilca
  9. FEB
    -
    Damien Genet
    Damien
    George Bosilca
    George
    Damien Genet, George Bosilca
  10. FEB
    -
    MPI Forum Portlan, Oregon
    Aurelien Bouteiller
    Aurelien
    Aurelien Bouteiller

Upcoming Conferences

  1. MAR
    BDEC Wuxi, China
    Jack Dongarra
    Jack
    Piotr Luszczek
    Piotr
    Terry Moore
    Terry
    Tracy Rafferty
    Tracy
    Jack Dongarra, Piotr Luszczek, Terry Moore, Tracy Rafferty
  2. MAR
    -
    TESSE Workgroup Meeting Roanoke, Virginia
    Damien Genet
    Damien
    George Bosilca
    George
    Jeffrey Steill
    Jeffrey
    Thomas Herault
    Thomas
    Damien Genet, George Bosilca, Jeffrey Steill, Thomas Herault

Recent Lunch Talks

  1. JAN
    6
    Kwai Wong
    Kwai Wong
    JICS
    An Interoperable Workflow Platform for Multidisciplinary Simulations—openDIEL PDF
  2. JAN
    13
    David Eberius
    David Eberius
    Profiling PaRSEC and Using Software Events in PAPI PDF
  3. JAN
    20
    Yves Robert
    Yves Robert
    Bidiagonalization with Parallel Tiled Algorithms PDF
  4. JAN
    27
    Ryan Glasby
    Ryan Glasby
    JICS
    Results from HPCMP CREATE™-AV Kestrel Component COFFE for 3-D Aircraft Configurations with Higher-Order Meshes Generated by Pointwise, Inc. PDF
  5. FEB
    3
    George Ostrouchov
    George Ostrouchov
    ORNL
    Bringing Modern Statistical Science to HPC PDF
  6. FEB
    10
    Scott Klasky
    Scott Klasky
    ORNL
    Creating ADIOS-2 for scientific exascale data PDF
  7. FEB
    17
    Arvind Ramanathan
    Arvind Ramanathan
    ORNL
    Computer to Cure Cancer: Building Exascale Deep Learning Tools for Effective Cancer Surveillance PDF

Upcoming Lunch Talks

  1. MAR
    10
    Nuria Losada
    Nuria Losada
    Stop&Restart vs Resilient MPI applications PDF
  2. MAR
    17
    Stephen Wood
    Stephen Wood
    Multilevel Parallelism for CFD: Preconditioning via Parallel ILU Factorization PDF
  3. MAR
    31
    Piotr Luszczek
    Piotr Luszczek
    Implementation Techniques for Point Set Registration on Multicore and Heterogeneous Hardware PDF

Visitors

  1. Mathieu Faverge
    Mathieu Faverge from Bordeaux INP will be visiting from February 23 through February 24. Mathieu is working with the Distributed Computing and Numerical Linear Algebra groups.
  2. Sticks Mabakane
    Sticks Mabakane from Council for Scientific and Industrial Research (South Africa) will be visiting from February 25 through March 12. Sticks is working with the PAPI group on performance visualizations.

People

  1. Matt Bachstein
    ICL welcomes Matt Bachstein as a graduate research assistant.
  2. Jeff Steill
    ICL welcomes Jeff Steill as a graduate research assistant.
  3. Stephen Wood
    ICL welcomes Stephen Wood as a postdoctoral research associate.

Visitors

  1. Mathieu Faverge
    Mathieu Faverge from Bordeaux INP will be visiting from February 23 through February 24. Mathieu is working with the Distributed Computing and Numerical Linear Algebra groups.
  2. Sticks Mabakane
    Sticks Mabakane from Council for Scientific and Industrial Research (South Africa) will be visiting from February 25 through March 12. Sticks is working with the PAPI group on performance visualizations.

congratulations

Christmas Gift

baby-alan-haidarAlan Haidar was born to Dina and Azzam Haidar on December 17, 2016, at 9:38am EST. He entered the world at 20 inches in length and weighing 7 pounds and 1 ounce. Both mom and baby are doing well. Congratulations, Dina and Azzam!

Dates to Remember

ICL Winter Reception

Current ICL employees and their guest are invited to the annual ICL Winter Reception. This year’s event will be on Friday, February 17, from 5:30pm to 8pm, at Calhoun’s on the River in Knoxville.