Multiple MPI processes simultaneously accessing the GPU

[cafaro]

Hi,

in a code I have parallelized using MPI, I have in each distinct process a loop executed for millions of iterations. In each iteration, I need to call magma_dsygvd() to determine eigenvalues/eigenvectors. My question is, will the access to the GPU device (there is only one in my debug environment, I will modify the code using a different MAGMA function for the production environment with 2 Tesla K20 devices) be serialized among the MPI processes (competing to access the GPU, making the code slower) or is this automatically handled by MAGMA through streams ? If not, how should I handle this using MAGMA ?

Thank you in advance.
With kind regards,

Massimo Cafaro

[mgates3]

I believe MPI processes appear as different processes from CUDA's perspective, so they should be able to run simultaneously on the same GPU. I've done this with other MPI-enabled GPU codes, though not with MAGMA. However, if there are say 8 MPI processes and only 2 GPUs in one node, obviously there will be resource contention for the GPUs.
-mark

[cafaro]

Mark,

unfortunately I can not infer from your answer if accesses to the GPU are serialized or concurrent: in your example, 8 MPI processes could be all running on the two GPUs, if their resource requirements (in terms of GPU threads and GPU memory needed) can be simultaneously satisfied. In general indeed, many kernels can be running simultaneously on a GPU. What I need to understand is if MAGMA allows concurrent kernels launched by multiple MPI processes or not. If yes, does MAGMA take care automatically of concurrent kernel launches or should I modify the code in some way to achieve this goal ?

Thank you,
Massimo

[mgates3]

This is really a CUDA question. MAGMA does nothing special to deal with this. But in general, a single CUBLAS call will utilize the entire GPU, unless the matrix is small (say, N < 1000).
-mark