Optimization of Scalapack Libs

[gortiz]

Dear Forum,

Recently I get using scalapack_installer Blacs and Lapack libs and
I can build Scalapack Libs. All seems to be OK. But, what I can expect from optimization point of view? I mean, what efficiency I can expect related to performance of microprocessors, cache size, bus, etc. I suppose that it is strongly dependent if I build executable from source code.
But it is not for Blacs and Lapack libs in my case, because I use the sentence:

/setup.py --mpibindir=/usr/local/bin --mpiincdir=/usr/local/include --mpicc=/usr/local/bin/mpicc --mpif77=/usr/local/bin/mpif77 --blaslib=/usr/lib64/libblas.a --downblacs --downlapack

OK, thanks a lot!

g.

[buttari]

Dear Forum,

Recently I get using scalapack_installer Blacs and Lapack libs and
I can build Scalapack Libs. All seems to be OK. But, what I can expect from optimization point of view? I mean, what efficiency I can expect related to performance of microprocessors, cache size, bus, etc. I suppose that it is strongly dependent if I build executable from source code.
But it is not for Blacs and Lapack libs in my case, because I use the sentence:

/setup.py --mpibindir=/usr/local/bin --mpiincdir=/usr/local/include --mpicc=/usr/local/bin/mpicc --mpif77=/usr/local/bin/mpif77 --blaslib=/usr/lib64/libblas.a --downblacs --downlapack

OK, thanks a lot!

g.

Gortiz,
the efficiency of the scalapck code mostly depends on the blas library used for doing the local computations. Thus, if your /usr/lib64/libblas.a is a fast blas you're ok. There are a number of freely available opensource blas (ATLAS, GotoBLAS) or freely available vendor blas (ACML, MKL) or even non free, closed source ones (ESSL). You should use one of these.
Moreover, you may get some improvements if you use other implementations of the whole scalapack library: for example the intel mkl library contains its own scalapack implementation. However I didn't benchmark it so I don't know if it would be faster and how much.

alfredo

[Danesh_D]

Gortiz,

As Alferedo mentioned the most efficiency factors of your parallel code when you use ScaLAPACK relies on BLAS. ATLAS and MKL are usually known to be best implementations of BLAS libraries. I have worked with both of them and they are well-suited for high performance computing purposes. The reason is that BLAS libraries are pre-compiled libraries that provide maximum speed and efficiency. I think they have been written in Fortran 90 but callable in C by using appropriate wrappers. Anyway, what you meant by efficiency is usually done at compiling time. For example ICC (Intel C Compiler and many other compilers) provides loop fusion techniques which makes your code much faster if this flag is on when you compile your code. However, compiling time would increase dramatically. You can also define optimization level (-O) when you compile your code using gcc or mpiCC which can improve efficiency. Based on which compiler you use, you can consult its notes for optimization to get better clues.

Danesh

[PHinker]

buttari has it right in that the majority of the performance you'll get will be from an optimized BLAS implementation (although, there are some optimizations in the LAPACK and SCALAPACK code which can also help).

I do not think that MKL is a freely available library, however. The last time I checked it was still a purchased, licensed product from Intel.

ACML (from Amd), ATLAS (from source forge), GotoBLAS (from utexas), and Sunperf (from Sun) are all freely available and are all considerably faster than the reference implementation (from netlib.org) on the majority of problems.