magma_init() takes up to 2 seconds to init

[thanasis_giannis]

Is it normal for magma_init() to take up to 2 seconds to init? And if no, what could cause this?

Thank you in advance!

[mgates3]

No, I wouldn't expect it to take that long. It queries the CUDA devices for their architecture (Fermi, Kepler, ...) and possibly makes a queue (stream) on each GPU. Assuming that you include magma_v2.h, instead of magma.h, you can try adding -DMAGMA_NO_V1 to CFLAGS in make.inc and recompiling MAGMA. That would disable creating any streams in magma_init.

What GPUs do you have? E.g., what is the output of a MAGMA tester (as below)?

Code: Select all

bunsen magma/testing> ./testing_dgetrf -n 100
% MAGMA 2.1.0 svn compiled for CUDA capability >= 3.5, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 16. MKL 11.3.0, MKL threads 16. 
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% device 1: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 09:31:53 2017
% Usage: ./testing_dgetrf [options] [-h|--help]

% ngpu 1, version 1
%   M     N   CPU Gflop/s (sec)   GPU Gflop/s (sec)   |PA-LU|/(N*|A|)
%========================================================================
  100   100     ---   (  ---  )      0.05 (   0.01)     ---   

-mark

[thanasis_giannis]

Code: Select all

thanasis@cuda:/usr/local/magma-2.2.0/testing$ ./testing_dgetrf -n 100
% MAGMA 2.2.0  compiled for CUDA capability >= 3.0, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 1. MKL 11.3.0, MKL threads 1. 
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 19:05:22 2017
% Usage: ./testing_dgetrf [options] [-h|--help]

% ngpu 1, version 1
%   M     N   CPU Gflop/s (sec)   GPU Gflop/s (sec)   |PA-LU|/(N*|A|)
%========================================================================
  100   100     ---   (  ---  )      2.36 (   0.00)     ---   
thanasis@cuda:/usr/local/magma-2.2.0/testing$ 

The output is this. I did put the -DMAGMA_NO_V1 yet the time is still the same.

[mgates3]

On my laptop, magma_init is reasonably quick, about 0.1 sec. On some of our servers, I found mixed results.

Code: Select all

k20:    1.35 sec
p100:   0.88 sec
k40:    6.21 sec
c2050:  1.15 sec

I was surprised these were that large. On any particular machine, the results varied a bit from run to run, perhaps 20%. Looking deeper, nearly all the time in magma_init is taken by cudaGetDeviceCount. If I put a different CUDA call, like cudaGetDeviceProperties, prior to cudaGetDeviceCount, then cudaGetDeviceProperties takes all the time and cudaGetDeviceCount takes negligible time. I surmise it is overhead in loading and initializing CUDA that happens on the first CUDA call. You can probably use CUDA's profiler to see this overhead.

Perhaps ask NVIDIA why the first call takes so long.

-mark

[thanasis_giannis]

Maybe this will be helpful

https://devtalk.nvidia.com/default/topi ... much-time/