magma_ssyevd for very large matrix

[Menchael]

Hi everybody! I have a problem using magma_ssyevd for large matrices, for example 40000 * 40000. When I run the test program testing_ssyevd with command line "./testing_ssyevd -N 40000 -JV -L", the program ended up with GPU memory allocation failure. I am using magma-1.5.0, and as I know this version supports some out-of-GPU memory algorhithms. Does anyone know if magma_ssyevd support a matrix bigger than 40000, or is there any other function does ? BTW, my GPU is Tesla C2070, and I am using CUDA 5.0 toolkit. Thanks in advance.

[mgates3]

syevd does not support an out-of-GPU memory algorithm. Only the linear solvers (e.g., getrf, potrf) have out-of-GPU memory algorithms in MAGMA.

Other than physical memory limits, around 46000 you will also run into integer overflow with 32 bit ints. This happens because for an N * N matrix, 46341 * 46341 > 2^31. You can avoid that problem by using ILP64 with 64 bit ints, as shown in the make.inc.mkl-ilp64 example file.

-mark

[Menchael]

syevd does not support an out-of-GPU memory algorithm. Only the linear solvers (e.g., getrf, potrf) have out-of-GPU memory algorithms in MAGMA.

Other than physical memory limits, around 46000 you will also run into integer overflow with 32 bit ints. This happens because for an N * N matrix, 46341 * 46341 > 2^31. You can avoid that problem by using ILP64 with 64 bit ints, as shown in the make.inc.mkl-ilp64 example file.

-mark

Thanks for replying. So currently, MAGMA only supports out-of-GPU memory algorithms for linear solvers. I am now using MKL for large matrix EVD, and it's very slow. Do you know any library which support GPU version EVD for large matrix ? Also thank you for reminding the problem of 32 bit integers. I encontered this problem before, and solved it as you mentioned.

[haidar]

the current Magma Eigenvalue routine does not support out of GPU memory.
if your matrices are large we encourage using more than 1 GPU.
Otherwise I can recommend to install the PLASMA library for CPU's only but you can get higher performance than the existing CPU library 1.5 to 2X speedup on 16 SandyBridge.
The latest release of PLASMA (expected to be around December) have more improvement and performance can reach more than 2X speedup over the MKL dsyevd on 16 Sandy Bridge machine.
if you really need this as soon as possible I can provide the patch to the latest improvement but you can also try to install PLASMA and use ssyevd routine.
Azzam

[Menchael]

the current Magma Eigenvalue routine does not support out of GPU memory.
if your matrices are large we encourage using more than 1 GPU.
Otherwise I can recommend to install the PLASMA library for CPU's only but you can get higher performance than the existing CPU library 1.5 to 2X speedup on 16 SandyBridge.
The latest release of PLASMA (expected to be around December) have more improvement and performance can reach more than 2X speedup over the MKL dsyevd on 16 Sandy Bridge machine.
if you really need this as soon as possible I can provide the patch to the latest improvement but you can also try to install PLASMA and use ssyevd routine.
Azzam

Thanks for your reply. So generally, using magma_ssyevd_m with 2 GPUs can solve twice the size of that of magma_ssyevd ?
When I changed magma_ssyevd to magma_ssyevd_m, I faced this problem: magmablas_ssymv_mgpu_32_offset not supported on CUDA arch 1.x.
This is because on my server, besides 4 Tesla GPUs, there is one GeForce 210 GPU with compute capability 1.x. So can I appoint the GPUs to use before invoking magma_ssyevd_m to avoid this problem ?

[mgates3]

Yes, twice the memory size, which is about sqrt( 2 ) times the number of unknowns, N, since the matrix is N-by-N in size. E.g., it could go from say 1000 x 1000 (1e6 matrix entries) to 1414 x 1414 (2e6 matrix entries).

For limiting what GPUs to use, at the moment the best solution is probably the CUDA_VISIBLE_DEVICES environment variable.
http://devblogs.nvidia.com/parallelfora ... e_devices/

-mark

[Menchael]

Yes, twice the memory size, which is about sqrt( 2 ) times the number of unknowns, N, since the matrix is N-by-N in size. E.g., it could go from say 1000 x 1000 (1e6 matrix entries) to 1414 x 1414 (2e6 matrix entries).

For limiting what GPUs to use, at the moment the best solution is probably the CUDA_VISIBLE_DEVICES environment variable.
http://devblogs.nvidia.com/parallelfora ... e_devices/

-mark

I have tested magma_ssyevd_m after setting CUDA_VISIBLE_DEVICES environment variable, and finally the program ran successfully. One problem that is noteworthy is the GPU IDs. On my machine, GeForce 210's ID is 1, and the other Tesla GPUs' IDs are 0, 2, 3, 4. So if 1 is in the CUDA_VISIBLE_DEVICES variable, the program will crash.