Page 1 of 1
magma_init() takes up to 2 seconds to init
Posted: Thu Aug 24, 2017 7:40 am
by thanasis_giannis
Is it normal for magma_init() to take up to 2 seconds to init? And if no, what could cause this?
Thank you in advance!
Re: magma_init() takes up to 2 seconds to init
Posted: Thu Aug 24, 2017 9:35 am
by mgates3
No, I wouldn't expect it to take that long. It queries the CUDA devices for their architecture (Fermi, Kepler, ...) and possibly makes a queue (stream) on each GPU. Assuming that you include magma_v2.h, instead of magma.h, you can try adding -DMAGMA_NO_V1 to CFLAGS in make.inc and recompiling MAGMA. That would disable creating any streams in magma_init.
What GPUs do you have? E.g., what is the output of a MAGMA tester (as below)?
Code: Select all
bunsen magma/testing> ./testing_dgetrf -n 100
% MAGMA 2.1.0 svn compiled for CUDA capability >= 3.5, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 16. MKL 11.3.0, MKL threads 16.
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% device 1: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 09:31:53 2017
% Usage: ./testing_dgetrf [options] [-h|--help]
% ngpu 1, version 1
% M N CPU Gflop/s (sec) GPU Gflop/s (sec) |PA-LU|/(N*|A|)
%========================================================================
100 100 --- ( --- ) 0.05 ( 0.01) ---
-mark
Re: magma_init() takes up to 2 seconds to init
Posted: Thu Aug 24, 2017 5:56 pm
by thanasis_giannis
Code: Select all
thanasis@cuda:/usr/local/magma-2.2.0/testing$ ./testing_dgetrf -n 100
% MAGMA 2.2.0 compiled for CUDA capability >= 3.0, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 1. MKL 11.3.0, MKL threads 1.
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 19:05:22 2017
% Usage: ./testing_dgetrf [options] [-h|--help]
% ngpu 1, version 1
% M N CPU Gflop/s (sec) GPU Gflop/s (sec) |PA-LU|/(N*|A|)
%========================================================================
100 100 --- ( --- ) 2.36 ( 0.00) ---
thanasis@cuda:/usr/local/magma-2.2.0/testing$
The output is this. I did put the -DMAGMA_NO_V1 yet the time is still the same.
Re: magma_init() takes up to 2 seconds to init
Posted: Fri Aug 25, 2017 3:34 pm
by mgates3
On my laptop, magma_init is reasonably quick, about 0.1 sec. On some of our servers, I found mixed results.
Code: Select all
k20: 1.35 sec
p100: 0.88 sec
k40: 6.21 sec
c2050: 1.15 sec
I was surprised these were that large. On any particular machine, the results varied a bit from run to run, perhaps 20%. Looking deeper, nearly all the time in magma_init is taken by cudaGetDeviceCount. If I put a different CUDA call, like cudaGetDeviceProperties, prior to cudaGetDeviceCount, then cudaGetDeviceProperties takes all the time and cudaGetDeviceCount takes negligible time. I surmise it is overhead in loading and initializing CUDA that happens on the first CUDA call. You can probably use CUDA's profiler to see this overhead.
Perhaps ask NVIDIA why the first call takes so long.
-mark
Re: magma_init() takes up to 2 seconds to init
Posted: Sun Aug 27, 2017 9:55 am
by thanasis_giannis