SVD performance SMP vs. Cluster/Scalapack vs. Lapack

[thombos]

Hello again,

I have been benchmarking new compute equipment for a while now and it I found a number of interesting issues that I don't completely understand. One example benchmark is the svd of a 1000x200000 matrix, and as discussed previously on the forum I implemented the scalapack variant by performing an LQ decomposition first, then performing the SVD of L using pcgesvd and finally calculating VT using pcunmlq. The last part is the slowest in general, but I have been getting values as low as 25-30 seconds for this problem (all three steps together) with this code on a small cluster with dual socket quadcore blades, using a total of 64 cores.
Interestingly the best SMP machine is close to 60 seconds and a bluegene/P is a bit behind that.
I'm disappointed about the performance of the SMP systems, but the bottleneck seems to be somewhere with MPI or using MPI on a shared memory SMP system.
For comparison I also turned to just using plain Lapack cgesdd and multithreaded BLAS (GotoBLAS). Using 16 threads on my 16 core SMP machine in the office it takes 1060 seconds to solve this problem, while my scalapack program described above using 16 mpi ranks takes 128 seconds for the same problem on the same machine. Is there any way to get more performance out of SMP systems ?

Thanks, Thomas

[Julien Langou]

For comparison I also turned to just using plain Lapack cgesdd and multithreaded BLAS (GotoBLAS). Using 16 threads on my 16 core SMP machine in the office it takes 1060 seconds to solve this problem, while my scalapack program described above using 16 mpi ranks takes 128 seconds for the same problem on the same machine. Is there any way to get more performance out of SMP systems ?

Looks like you already get pretty good performance out of ScaLAPACK, no?

If you have some time one thing that would be really interesting for us to
have is a breakdown per operation to compare the LAPACK and the
ScaLAPACK code. LAPACK should automatically perform first an LQ facto,
then a SVD, then MLQ as you code does.

LAPACK is a sequential code, unless your using a really tricky vendor
implementation tailored for SMP (and I do not know of many), then the
implementations based on the reference LAPACK (from netlib) perform
poorly on SMP, in particular when there is more than say 2/4/8 cores ...
ScaLAPACK has been constructed with parallelism in mind. So, at 16
cores, it's not a big surprise to see it win over LAPACK.

Julien.

[thombos]

Well in fact, ScaLapack with 2 cores beats Lapack with 16 cores. I guess my main question was why ScaLapack on a cluster easily beats ScaLapack on a SMP machine, using same FSB and CPU in both configurations. This is interesting from the standpoint that clusters are cheaper than SMP machines especially when it comes to high core counts.
I guess what I'm fishing for would be something that uses parallel algorithms like scalapack, maybe even using blockcyclic matrix representation, but doesn't have MPI overhead, or somehow more efficient use of SMP machines. The only diagnostic we have so far is that the shared memory MPIs seem to hit bandwidth limits in the SMP machines, which doesn't seem to happen on clusters, although the interconnect bandwith in clusters is much lower than in SMP machines...

[thombos]

Okay, here come the numbers:

All numbers are from the same 8 x 2 core opteron machine, matrix size 1000 x 200000 (single complex)

Scalapack (16 MPI ranks)
LQ = 53 seconds, SVD(L) = 1 second, QVT = 57 seconds
Lapack (GotoBlas, 16 threads)
LQ = 525 seconds, SVD(L) = 3 seconds, QVT = 542 seconds.

Thomas

[Julien Langou]

Hello Thomas,
Thanks for the update. Well, that's impressive.
So long for the LAPACK model.
Cheers,
Julien.

[thombos]

BTW, the equipment that I'm looking to buy can do the LQ decomposition of this matrix in 12 seconds using 32 cores. Thats both the cluster and the SMP machine. The SMP machine was slower before because it had wrong memory modules during testing, now for my benchmarks the cluster and the SMP machine are always within a couple of seconds.
As for LAPACK I had a look at cgelqf. I don't have much time right now, but it seems to me, if it was slightly modified to assume blockcyclic storage of the input matrix it could be easily parallelized using OpenMP. I looked a little into this for fun, the nastiest bit would be to deal with WORK, which of course cannot/should not be shared, and the scheduling of threads because there is an operation that takes data from a neighboring block.

Thomas