Difficulty Compiling Magma in Fortran

[KoonST]

I am trying to replace LAPACK dgesv with magma_dsgesv_gpu but I am having difficulty getting the magma version to compile. I am using a pgf90 compiler.
I have loaded a module containing magma/0.2 and put the appropriate links to the magma library in my program's Makefile.

When I try just replacing dgesv with magma_dsgesv_gpu the compiler says I have an undefined reference to `magma_dsgesv_gpu_'. I have tried the -nosecond_underscore compile option with this and get the same error.

I tired to use a C$PRAGMA like so:

C$PRAGMA C(magma_dsgesv_gpu)
call magma_dsgesv_gpu(ny,1,am,ny,indx,f,ny,info) ! Double precision version
dy=f

With that, the compiler recognizes magma_dsgesv_gpu but can not find functions with in it. I get this error message:

pgf90 -mp -fastsse -fpic -r8 -o xnet data.o common.o ffn.o full_net.o match.o net.o flux.o eosnom90.o screen.o jacobian_dense.o -L/sw/analysis-x64/magma/0.2/sl5.0_cuda_2.3/lib -lmagma -lmagmablas
/sw/analysis-x64/magma/0.2/sl5.0_cuda_2.3/lib/libmagma.a(dsgesv_gpu.o): In function `magma_dsgesv_gpu':
dsgesv_gpu.cpp:(.text+0xbf): undefined reference to `dlamch_'
dsgesv_gpu.cpp:(.text+0x127): undefined reference to `slamch_'
dsgesv_gpu.cpp:(.text+0x47b): undefined reference to `cublasIdamax'
dsgesv_gpu.cpp:(.text+0x4c0): undefined reference to `cublasGetMatrix'
dsgesv_gpu.cpp:(.text+0x4ea): undefined reference to `cublasIdamax'
dsgesv_gpu.cpp:(.text+0x52a): undefined reference to `cublasGetMatrix'
dsgesv_gpu.cpp:(.text+0x6e2): undefined reference to `cublasIdamax'
dsgesv_gpu.cpp:(.text+0x727): undefined reference to `cublasGetMatrix'
dsgesv_gpu.cpp:(.text+0x751): undefined reference to `cublasIdamax'
dsgesv_gpu.cpp:(.text+0x791): undefined reference to `cublasGetMatrix'
/sw/analysis-x64/magma/0.2/sl5.0_cuda_2.3/lib/libmagma.a(dsgesv_gpu.o):(.eh_frame+0x12): undefined reference to `__gxx_personality_v0'
/sw/analysis-x64/magma/0.2/sl5.0_cuda_2.3/lib/libmagma.a(dgemm_gpu.o): In function `magmablas_dgemm':
.... And it goes on.

What else can I try to get this to compile ?

[Stan Tomov]

I don't have access to a pgf90 but I tried with gfortran to produce an object file and to link with it without problem. You should not let your compiler add underscores after the routine names (as magma_dsgesv_gpu_ is not defined, only magma_dsgesv_gpu is defined in libmagma.a). Can you add option -fno-underscoring during the compilation.
In the second way of compiling you haven't added the cuda libraries. You have to have something like "-lcuda -lmagma -lmagmablas -lcublas get_nb.cpp".
The undefined reference to `__gxx_personality_v0' is something from the C++ compiler. Try to add something like -lstdc++ in the linking stage to see if it would work.
Can you compile the examples in testing using gfortran or gcc?
Stan

[KoonST]

I have been able to compile the examples but they are all in C. Is there a FORTRAN example anywhere?

I am trying to compile my work and the example on lens, an NCCS computer, lens's MAGAMA modules seem to lack some key files such as all the files stored in the lib directory of MAGMA2.0 or perhaps they are stored elesewhere. I have grabbed the MAGMA2.0.tar.gz files from here and put it in my scratch diretory to make the examples compile. I am stuck using the PGI compiler because my task is to compare MAGMA's speed to a number of other solvers run using the pgi compiler.

[KoonST]

In the testing example for dgesv, testing_dgesv_gpu.cpp, why is magma_dgetrf_gpu called instead of magma_dgesv_gpu? A The only instance of "dgesv" in testing_dgesv_gpu.cpp occur on lines:

testing_dgesv_gpu.cpp: line 21: -- Testing dgesv
testing_dgesv_gpu.cpp:line 44: printf(" testing_dgesv -N %d\n\n", 1024);

[Stan Tomov]

I have been able to compile the examples but they are all in C. Is there a FORTRAN example anywhere?

MAGMA can be called from Fortran programs. Attached is an example for testing the Cholesky factorization. To call MAGMA, a Fortran program (in this case example.f) has to be compiled with "-fno-underscoring" in order to be able directly to link to MAGMA (no need for wrapper; as shown in the example). The memory allocation, e.g., on the GPU or pinned memory on the CPU, is somehow trickier.
Related to the second question, we haven't created a separate magma_dgesv_gpu routine yet. The routine itself is composed of two routines, namely factorization and triangular solvers, so to test dgesv we just hard-coded the sequence of these two routines - first the factorization with magma_dgetrf_gpu and second the triangular solvers with magma_dgetrs_gpu.
Stan

[fletchjp]

I want to make use of the fortran example which is contributed here. I am using MAGMA 1.0 RC2.

I have adjusted the compilation for my system where I am using GotoBLAS and gfortran. Here is my makefile.

# Turn compile into a makefile

example: example.f fortran.cpp makefile
gcc -I/usr/local/cuda/src/ -I/usr/local/cuda/include -c fortran.cpp -o fortran.o
gfortran -fno-underscoring -O2 -I/usr/local/cuda/include -c example.f -o example.o
gfortran -fPIC example.o fortran.o -o example -L/home/fletcher/GotoBLAS2 -L/usr/lib64 -L/usr/local/cuda/lib64 -L../../lib -lgoto2 -lpthread -lcuda -lmagma -lmagmablas -lcublas -lcudart -llapack -lm

I get a segmentation fault in the call to magma_spotrf.

I wonder if something has changed, as the example was posted some time ago.

Please advise me.

John

[Stan Tomov]

We changed the main interface to magma - now integer values are passed by value. Thus, a wrapper is needed in order to call the routines from Fortran. In the case of Cholesky one would need to define function

Code: Select all

void magma_spotrf_( char *uplo, int *n, float *A, int *lda, int *info)
{
    magma_spotrf( uplo[0], *n, A, *lda, info);
}

and call this function from the Fortran program. These new interfaces will be added to MAGMA 1.0 RC3. The new interfaces will allow users to use MAGMA from Fortran as in MAGMA 0.2 (except that we also reduced the number of arguments when possible by removing work space required on the GPU memory).

[fletchjp]

Stan

Thank you for that. I have managed to make it work. I guess I will wait for RC 3 to do more tests with FORTRAN.

I have done some work on this and have adapted your example to make use of the return type.

Code: Select all

/* magma_spotrf.c wrapper function */

#include <stdio.h>
#include "magma.h"

int magma_spotrf_( char *uplo, int *n, float *A, int *lda, int *info)
{
  magma_int_t retval;
  retval =  magma_spotrf( uplo[0], *n, A, *lda, info);
  if (retval != 0) printf("magma_spotrf returns error value %d\n",retval);
  return retval;
}

John

[fletchjp]

Stan

I am still having difficulties getting FORTRAN versions to run. Is there some information on how best to get the memory transfers to the GPU organised?

I am attempting to get DGESV working, based on the example code in the NAG documentation, adapted to call MAGMA as well.

My code is like this. It fails with a segmentation fault when I call cublas_set_matrix.

Code: Select all

c test_magma_dgesv.f
c extending
c test_dgesv.f to test magma_degsv_gpu as well
c
c http://www.nag.com/numeric/fl/manual/pdf/F07/F07AAF.pdf
c
* F07AAF Example Program Text
* Mark 21 Release. NAG Copyright 2004.
* .. Parameters ..
      INTEGER NIN, NOUT
      PARAMETER (NIN=5,NOUT=6)
      INTEGER NMAX
      PARAMETER (NMAX=8)
      INTEGER LDA, LDB
      PARAMETER (LDA=NMAX,LDB=NMAX)
      INTEGER NDMAX
      PARAMETER (NDMAX=1024)
* .. Local Scalars ..
      INTEGER I, IFAIL, INFO, J, N, NRHS, retval, II
      integer :: sizeof_double = 8
      integer*8 off
* .. Local Arrays ..
      DOUBLE PRECISION A(LDA,NMAX), B(NMAX)
      INTEGER IPIV(NMAX)
* .. Local GPU Pointers
      real*8, TARGET :: phony(1)
      real*8, pointer :: h_A(:), h_B(:)
      real*8, pointer :: d_A(:), d_B(:)
*      integer :: d_A, d_B
* .. External Functions ..
      INTEGER magma_dgesv_gpu_
* .. External Subroutines ..
      EXTERNAL DGESV
* .. Executable Statements ..
      call cublas_init_( )
      print *,'done call cublas_init_( )'
      WRITE (NOUT,*) 'DGESV Example Program Results'
      WRITE (NOUT,*)
* Skip heading in data file
      READ (NIN,*)
      READ (NIN,*) N
      IF (N.LE.NMAX) THEN
          allocate(h_A(ldmax*n))
          allocate(h_B(n))
          n2 = ldmax*n
c------ Allocate GPU memory
          phony(1) = 0
          print *,'About to do cublas_alloc'
c          call cublas_alloc_(ldmax*N, sizeof_double, d_A)
c          call cublas_alloc_(ldmax*NRHS, sizeof_double, d_B)
          call cublas_Alloc_(NDMAX*NDMAX, sizeof_double, phony(1), off)
          d_A => phony(off+1:off+1+NDMAX*NDMAX)
          call cublas_Alloc_(NDMAX, sizeof_double, phony(1), off)
          d_B => phony(off+1:off+1+NDMAX)
          print *,'done call cublasAlloc( )'
*         print *,'off = ',off
*
* Read A and B from data file
*
          READ (NIN,*) ((A(I,J),J=1,N),I=1,N)
          READ (NIN,*) (B(I),I=1,N)
*
          II = 0
          DO I = 1,N
             h_B(I) = B(I)
             DO J = 1,N
                II = II + 1
                h_A(II) = A(J,I)
             END DO
          END DO
          NRHS = 1
          LDDA = NDMAX
          LDDB = NDMAX
      print *, N, LDA, LDDA
      print *,'About to set matrix values'
* I have not got this to work properly.
      call cublas_Set_Matrix_( N, N,   sizeof_double,h_A,lda,d_A,ldda)
      print *,'Set matrix A'
      call cublas_Set_Matrix_( N, NRHS,sizeof_double,h_B,ldb,d_B,lddb)
      print *,'Set matrix B'
*
*
* Solve the equations Ax = b for x
*
          CALL DGESV_(N,1,A,LDA,IPIV,B,N,INFO)
*
          IF (INFO.EQ.0) THEN
*
* Print solution
*
              WRITE (NOUT,*) 'Solution'
              WRITE (NOUT,99999) (B(I),I=1,N)
*
* Print details of factorization
*
              WRITE (NOUT,*)
              IFAIL = 0
C      CALL X04CAF( General ,   ,N,N,A,LDA,
C     +  Details of factorization ,IFAIL)
              DO I = 1,N
                  WRITE (NOUT,99996) I,(A(I,J),J=1,N)
              END DO
*
* Print pivot indices
*
              WRITE (NOUT,*)
              WRITE (NOUT,*) 'Pivot indices'
              WRITE (NOUT,99998) (IPIV(I),I=1,N)
*
          ELSE
              WRITE (NOUT,99997) 'The (', INFO, ',', INFO, ')',
     +               ' element of the factor U is zero'
          END IF
c
c  Now do the GPU calculation.
c
          retval = magma_dgesv_gpu_(N,1,d_A,LDDA,IPIV,d_B,LDDB,INFO)
          print *,'INFO from GPU = ',INFO
          print *,'retval = ',retval
      ELSE
          WRITE (NOUT,*) 'NMAX too small'
      END IF
c      call cublas_free_(d_A)
c      call cublas_free_(d_B)
      call cublas_shutdown_()
      STOP
*
99999 FORMAT ((3X,7F11.4))
99998 FORMAT ((3X,7I11))
99997 FORMAT (1X,A,I3,A,I3,A,A)
99996 FORMAT ((I3,7F11.4))
      END

and the test data is

Code: Select all

DGESV Example Program Data
4 :Value of N
1.80 2.88 2.05 -0.89
5.25 -2.95 -0.95 -3.80
1.58 -2.69 -2.90 -1.04
-1.11 -0.66 -0.59 0.80 :End of matrix A
9.52 24.35 0.77 -6.22 :End of vector b

Makefile

Code: Select all

test_magma_dgesv: test_magma_dgesv.f makefile magma_dgesv_gpu.c
	gfortran -fno-underscoring -O2 -c test_magma_dgesv.f -o test_magma_dgesv.o
	gcc -I../../include -I/usr/local/cuda/include -c magma_dgesv_gpu.c -o magma_dgesv_gpu.o
	gfortran -fPIC test_magma_dgesv.o magma_dgesv_gpu.o fortran.o ../lin/liblapacktest.a -L../../lib -L/usr/local/cuda/lib64 -lmagma -lmagmablas -lcuda -lcublas -llapack -lblas -otest_magma_dgesv

Interface file (untested as yet)

Code: Select all

/* magma_dgesv_gpu.c wrapper function */

#include <stdio.h>
#include "magma.h"

/*
Original definition in dgesv_gpu.cpp 
extern "C" magma_int_t
magma_dgesv_gpu( magma_int_t n, magma_int_t nrhs, 
                 double *dA, magma_int_t ldda,
		 magma_int_t *ipiv, 
                 double *dB, magma_int_t lddb, 
                 magma_int_t *info)
*/
int magma_dgesv_gpu_(int *n, int *nrhs, double *dA, int *ldda, 
                      int *ipiv, double *dB, int *lddb, int *info)
{
  magma_int_t retval;
  retval =  magma_dgesv_gpu(*n, *nrhs, dA, *ldda, ipiv, dB, *lddb, info);
  if (retval != 0) printf("magma_dgesv_gpu returns error value %d\n",retval);
  return retval;
}

Thanks

John

[Stan Tomov]

John,

We changed the way to call MAGMA from Fortran. We described it in this article

http://www.pgroup.com/lit/articles/insider/v2n4a4.htm

Briefly, you have to use the bindings in file fortran.cpp from NVIDIA's CUDA SDK. Wrappers have to be written for the current MAGMA routines (as now integers are passed by value). Pointers to memory on the GPU have to be declared as devptr_t *devicePtr, e.g., as in

Code: Select all

void MAGMA_DGETRS_GPU( char *trans, magma_int_t *n, magma_int_t *nrhs, devptr_t *A, magma_int_t *lda, 
                       magma_int_t *ipiv, devptr_t *b, magma_int_t *ldb, magma_int_t *info)
{
    double *d_a = (double *)(*A);
    double *d_b = (double *)(*b);
    magma_dgetrs_gpu( trans[0], *n, *nrhs, d_a, *lda, ipiv, d_b, *ldb, info);
}

We will provide all the wrappers needed in MAGMA 1.0 RC3.

Stan