News and Announcements
HPC Learning Webinar
On Tuesday, February 7, at 12pm Eastern Standard Time, Jack Dongarra will be speaker for the Association for Computing Machinery (ACM) Learning Center live one-hour webinar titled “Current Trends in High Performance Computing and Challenges for the Future.”
Jack will discuss the ways in which HPC has changed during the last decade and where it appears to be headed. He will also delve into how the effective and reliable use of dynamic, distributed, and parallel computing environments will call for the development of a new generation of software libraries and algorithms.
Registration is required to attend the webinar.
Conference Reports
ECP PI Meeting
Principal investigators for the US Department of Energy’s Exascale Computing Project (ECP) met on November 29 at Argonne National Laboratory (Lemont, IL) to make presentations and consider the ECP’s big picture.
ICL’s George Bosilca, Jack Dongarra, and Heike Jagode attended this gathering, which was limited to one representative per project.
Quad charts for each of the funded projects aided the identification of topics and individuals relative to potential collaborations, interdependencies, and networking. Following the presentation of the charts, multiple parallel group discussions during days two and three allowed the participants to explore ways in which their projects connect to other portions of the ECP.
PEEKS Kick-off Meeting
ICL’s Ichitaro Yamazaki and Hartwig Anzt traveled to Albuquerque, NM, for a visit with collaborators at Sandia National Laboratories (SNL) on December 5 to discuss milestones and the initial steps involved in the Production-ready, Exascale-enabled Krylov Solvers (PEEKS) project, which is part of the US Department of Energy’s Exascale Computing Project (ECP).
Main SNL collaborators Mike Heroux, Erik Boman, Mark Hoemmen, and Siva Rajamanickam, as well as other SNL personnel, also attended the meeting. Ichi and Hartwig’s presentations provided both introductory and update information concerning PEEKS.
As the trip drew to a close, the ICLers had Friday morning off, so they took the opportunity to visit some local attractions. Hartwig toured the National Museum of Nuclear Science & History, while Ichi rode the Sandia Peak tramway to the top of the Sandia Mountains.
TESSE Workgroup Meeting
Members of the Task-based Environment for Scientific Simulation at Extreme Scale (TESSE) project workgroup met in the Stony Brook University of New York (SUNY) Global Center in New York City on December 12 to discuss progress.
TESSE uses an application-driven design to create a general-purpose, production-quality software framework that attacks the twin challenges of programmer productivity and portable performance for advanced scientific applications on the massively parallel, hybrid, manycore systems existing today and those anticipated for the future.
George Bosilca, Thomas Herault, and Damien Genet represented ICL at the meeting. Attending from the SUNY Institute for Advanced Computational Science (IACS) were Robert J. Harrison, W. Scott Thornton, and Mohammed Mahdi Javanmard; and from Virginia Tech, Department of Chemistry, Edward F. Valeev. In addition, invited to speak on the first day were Barbara Chapman of SUNY IACS and Brookhaven National Laboratory, and Dmitry Lyakh of Oak Ridge National Laboratory (ORNL).
Barbara discussed OpenMP and programming models. Dmitry presented on his work at ORNL, the agnostic design for performance portability.
Also during day one of the meeting, George, Thomas, and Damien gave a presentation on the Parallel Runtime Scheduling and Execution Controller (PaRSEC) project. PaRSEC is a generic framework for architecture-aware scheduling and management of microtasks on distributed manycore heterogeneous architectures. The presentation focused on the programming model, interfaces, how to interact with the runtime, and the integration of what is known as the SUMMA operation.
Robert gave a talk on the overall TESSE project, and Edward discussed his code TiledArray, a modern C++ library that is scalable to tens of thousands of processor cores.
Open discussions were organized around two topics: the scientific application community and the interoperability of the programming model and runtime systems.
Day two involved a discussion of the low-level aspects of the project.
IPDPS 2017 Review

Lead reviewers, including ICL’s Piotr Luszczek, and the organization committee of the IEEE International Parallel and Distributed Processing Symposium (IPDPS) met on January 5 in Rosemont, IL, near Chicago, to select papers for the conference, which is set to take place May 29 to June 2, 2017, in Buena Vista Palace Hotel, Orlando, FL.
Three tiers of reviewers overall were involved in the preparatory activities this year for IPDPS, with the lead review tier making the final decisions. The lead review team was diverse, encompassing academia, industry, and the national laboratories. Piotr participated in the application and software tracks.
ECP CEED Meeting


On January 10 through 12, the kick-off meeting of the Exascale Computing Project (ECP) Center for Efficient Exascale Discretizations (CEED) convened at Argonne National Laboratory to familiarize the participants from CEED’s collaborating institutions with relevant software and technologies.
The ECP CEED project will develop the next-generation discretization software and algorithms that will enable a wide range of US Department of Energy and National Nuclear Security Administration applications to run on future hardware.
Twenty-eight people from all the institutions involved in CEED attended the kick-off meeting. The CEED team encompasses Lawrence Livermore National Laboratory, which is leading the project, and Argonne National Laboratory, five universities, and outside contributors. Azzam Haidar and Stan Tomov represented ICL.
Stan described the meeting as a very useful jumpstart for the project—from getting to know the variety of software products and technologies already developed, to learning more about the applications that will be targeted during the first year.
Among the already-developed software and technologies were PUMI, Holmes, Nek, MFEM, MAGMA, OCCA, PETSc, Hypre, and Spack. Receiving attention the first year will be ExaSMR and MARBL.
Stan said the meeting also afforded the opportunity to synchronize and start the work on initial milestones, collaborations, and interactions.
As a collaborator in CEED, ICL will be instrumental in identifying, developing, and optimizing tensor contractions that are essential building blocks for the applications. We will also take an active role in co-designing APIs with the LLNL scientists, external partners on the team, and vendors, and deliver a high-performance tensor contractions package through our MAGMA library.
At the meeting, Stan and Azzam presented the MAGMA library and its batched linear algebra capabilities. ICL will build on the MAGMA batched capabilities and be instrumental in developing a MAGMA tensor contractions engine to power main computational components of the high-order methods targeted in CEED. In addition, we will lead the hardware thrust in CEED, which will entail handling interactions with the exascale hardware vendors.
The types of applications that the CEED project will enhance include global climate modeling, turbulent combustion in internal combustion engines, nuclear reactor modeling, additive manufacturing, subsurface flow, and national security.
MPHC Program Meeting

Trieste, a beautiful city on the Adriatic coast of Italy, was the location January 10–13 for the Master in High Performance Computing (MPHC) program meeting, which involved extensive teaching and labs with hands-on exercises for six hours each day.
The activities covered computer architecture and hardware for HPC and upcoming exascale systems. ICL’s Piotr Luszczek taught asynchronous CUDA programming and gave a full rundown of overview, design, and implementation details of all the linear algebra libraries we produce and release at ICL.
The MHPC program is for graduate students from the International School for Advanced Studies (Scuola Internazionale Superiore di Studi Avanzati) and the International Centre for Theoretical Physics. Program links and materials are available here.
Piotr notes that Trieste is only a stone’s throw away from Slovenia and the Republic of Croatia.
ISTC for Big Data Tech Meeting
On January 25 at the Intel Science and Technology Center (ISTC) for Big Data technical meeting in Boston, MA, Piotr Luszczek and Thomas Herault discussed the integration of the linear algebra algorithms developed at ICL into the SciDB computational database management system. SciDB allows the mining of insights from diverse data sources such as genomics, clinical, and financial markets.
The meeting also featured Michael Stonebraker of the Computer Science and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute of Technology (MIT), Tim Mattson and Jeff Pankhurst of Intel, and Jason Kitchen and James McQuestion of the Paradigm4 company.
Inaugural ECP Annual Meeting
The first “All Hands” meeting to launch the US Department of Energy’s Exascale Computing Project (ECP) occurred at the Knoxville Convention Center from January 31 to February 2. It brought together a significant portion of the ECP team—between 450 and 500 people. ICL sent fifteen people to represent its seven ECP subprojects.
The goal of the meeting was to establish a shared understanding of the ECP’s overarching vision, with particular attention to fostering the ECP community, making progress on subproject integration, and promoting an ethos of holistic codesign among the different ECP groups—science applications, software technologies, hardware technologies, and facility operations.
The ICL participants had three objectives in mind as they fanned out across the breakout sessions led by the ECP groups: determine the needs of different science applications for ICL software technology, ascertain how best to integrate ICL software into the ECP software ecosystem, and understand how ICL would need to coordinate with the different DOE laboratories in the testing and deployment of the software its ECP projects will produce.
Recent Releases
2017 ICL Annual Report

ICL’s tradition of presenting an annual report continues. For brief research project summaries—including new endeavors on the horizon for 2017—an overview of ICL’s history, some of the highlights of the year that just ended, and a glimpse at the people and external associates behind the organization’s success, please download and review a copy.
LAPACK 3.7.0 Released
LAPACK 3.7.0 is now available. The Linear Algebra PACKage (LAPACK) is a widely used library for efficiently solving dense linear algebra problems. ICL has been a major contributor to the development and maintenance of the package since its inception. LAPACK is sequential, relies on the BLAS library, and benefits from the multicore BLAS library.
LAPACK 3.7.0 has TSQR for least-square, communication-avoiding symmetric-indefinite factorization with Aasen’s triangular tridiagonalization, Level 3 BLAS use by the rook pivoting form of LDL, the reduction to tridiagonal routine using the two-stage algorithm, and the reduction to bidiagonal routine using the two-stage algorithm.
Since 2011, LAPACK has included LAPACKE, a native C interface for LAPACK developed in collaboration with INTEL, which provides NAN check and automatic workspace allocation.
PLASMA 17.01 Released
PLASMA 17.01 has been released. This is the first release of PLASMA based solely on OpenMP multithreading directives.
This release focuses on linear systems and least squares and includes the following:
- Linear systems solvers (LU, Cholesky)
- Mixed precision iterative refinement (LU, Cholesky)
- Matrix norms (fast, multithreaded)
- A full set of multithreaded BLAS 3
- Matrix inversion (LU, Cholesky)
- Least squares with QR/LQ (including fast tall and skinny routines)
- Band linear solvers (LU, Cholesky)
This release includes “support” for the Knights Landing Xeon Phi
(in reality, PLASMA works out of the box on the KNL).
In all four precisions, the current PLASMA contains the following:
- 115,000 lines of code, plus another 37,000 lines of testing code
- 400 API functions
Besides porting PLASMA from QUARK to OpenMP, the teams at UTK and Manchester also developed the following:
- New LU factorization with internally blocked and multithreaded panel factorization
- New QR/LQ factorization routines offering a couple of different tree reduction patterns
- New mixed precision iterative refinement routines—reimplemented to follow LAPACK’s procedure
- New matrix norms routines—reimplemented to allow for increased multithreading
- Band linear systems routines for LU and Cholesky factorizations and solves
- A new testing/timing harness—to allow for easy inner/outer iteration over the parameter space
- A new build system—heavily borrowing from MAGMA
- A new precision generation system—heavily borrowing from MAGMA
Thanks to all the people from UTK and Manchester for their hard work. Special thanks to those who went beyond the call of duty to deliver new exciting capabilities.
Get the code by cloning the Mercurial repository on Bitbucket:
hg clone ssh://hg@bitbucket.org/icl/plasma
or by downloading the tarball.
Stay in touch by joining the plasma-user forum. (Apply to join the group.)
Open MPI 2.0.2 Released
The Open MPI team, representing a consortium of research, academic, and industry partners, is pleased to announce the release of Open MPI version 2.0.2., which includes a variety of bug fixes and some performance fixes.
All users are encouraged to upgrade to version 2.0.2 when possible. It can be downloaded from the main Open MPI website.
Interview

Stephen Wood
Where are you from originally?
I was born and raised in Miami, FL.
Can you summarize your educational background?
For high school I was fortunate to have attended the Maritime, Atmospheric Science, and Technology Academy, where my interests in physics, fluid dynamics, and mathematics flourished. I graduated from Florida International University (FIU) cum laude with a bachelor of science degree in mechanical engineering (ME) in 2010 and a master of science degree in ME in 2011. While at FIU, my research was supported by the Ronald E. McNair Post-baccalaureate Achievement Program and US Department of Energy/FIU Science & Technology WorkForce Development Initiative (DOE Fellows Program). Through the DOE Fellows Program, I participated in two internships (Higher Education Research Experiences [HERE] and Research Alliance in Math and Science [RAMS]) at Oak Ridge National Laboratory (ORNL) in the Computer Science and Mathematics Directorate. The world-class interactions of these internships prompted me to seek opportunities to pursue a doctorate program closely associated with ORNL. I was fortunate to be a Tennessee State Collaborative on Reforming Education (SCORE) Energy Scholar in the inaugural cohort of Bredesen Center students at the University of Tennessee, Knoxville. I earned a doctor of philosophy in energy science and engineering (ESE) with an Interdisciplinary Graduate Minor in Computer Science (IGMCS) in 2016. The interdisciplinary ESE degree program offered at the Bredesen Center and the IGMCS offered by ICL have enabled me to be a productive and collaborative postdoc at the Joint Institute for Computational Sciences.
Tell us how you first learned about ICL.
In my undergraduate courses at FIU, professors discussed BLAS and LAPACK with sincere appreciation as they imparted the importance of verification and benchmarking efforts in engineering software development.
What made you want to work for ICL?
The opportunity to “pay it forward” by contributing to the development of software libraries that current and future engineers will utilize.
What are you working on while at ICL?
I’m collaborating with Hartwig Anzt to continue the development of MAGMA Sparse for computational engineering applications deployed on modern architectures.
If you weren’t working at ICL, where would you like to be working and why?
I’d like to be working at one of the national laboratories or NASA as part of a multidisciplinary team tackling hard problems in the national and global interest.
What are your interests/hobbies outside of work?
I enjoy the outdoors, trail running, rock climbing, swimming, biking, rowing, sailing, canoeing, and hiking. Facilitating my two-and-half-year-old daughter’s early experiences of these activities is one of the greatest joys I’ve ever had.
Tell us something about yourself that might surprise people.
I earned a first-degree black belt in Shorin-Ryu Karate in 2010.





































Alan Haidar was born to Dina and Azzam Haidar on December 17, 2016, at 9:38am EST. He entered the world at 20 inches in length and weighing 7 pounds and 1 ounce. Both mom and baby are doing well. Congratulations, Dina and Azzam!