What does this error message mean ?

[arad]

The error message I get from my code is

MPI Application rank 3 killed before MPI_Finalize() with signal 10

this rank value changes each time i run it...

Does this has to do with LWORK value ? I dont know how to set LWORK so i set = -1 ...

My code after running SUCCESFULLY the routine PDGEQRF it calls the routine PDORMQR to multiply the Q returned from the previus routine, with the vector C. In that point the error appears. When i set LWORK = 200, 300 etc then it returns that the INFO parametre of PDORMQR is -5.
Here is the code :

Code: Select all

       PROGRAM TESTINGQR
*
*      PARAMETERS
       INTEGER      M, N, MB, NB, RSRC, CSRC, MXLLDA,
     $              IA, JA, LWORK , MXLOCC , NOUT, DLEN_ , K ,IC, JC,
     $              NRHS, T , MXLLDC , NBRHS
       PARAMETER  ( M = 9, N = 9, MB = 2, NB = 2, RSRC = 0, CSRC = 0,
     $              MXLLDA = 5, IA = 1, JA = 1, LWORK = -1, MXLOCC = 4,
     $              NOUT = 6, DLEN_ = 9, K = 9, IC = 1, JC = 1,
     $              MXLLDC = 5, NBRHS = 1, NRHS = 1)
*      SCALARS   *
       INTEGER      ICTXT, MYROW, MYCOL, NPCOL, NPROW, INFO
*
*      ARRAYS
       INTEGER DESCA(DLEN_) , DESCC(DLEN_)
       DOUBLE PRECISION A(MXLLDA, MXLOCC), TAU(1,5), WORK(LWORK ),
     $                  C(5,1)
*
*      EXTERNAL SUBROUTINES
       EXTERNAL    BLACS_EXIT, BLACS_GRIDEXIT, BLACS_GRIDINFO,
     $             DESCINIT, MATINIT, SL_INIT
*
       DATA NPROW / 2 /, NPCOL / 3 /
*
*      ---------------------------
*      INITIALIZE THE PROCESS GRID
*      ---------------------------
       CALL SL_INIT( ICTXT, NPROW, NPCOL )
       CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL )
*
*     If I'm not in the process grid, go to the end of the program
*
      IF( MYROW.EQ.-1 )
     $ GO TO 10
*
*      -------------------------------------------
*      DISTRIBUTE THE MATRIX OVER THE PROCESS GRID
*      -------------------------------------------
       CALL DESCINIT( DESCA, M, N, MB, NB, RSRC, CSRC, ICTXT, 
     $               MXLLDA, INFO )
       CALL DESCINIT( DESCC, N, NRHS , NB, NBRHS , RSRC, CSRC, ICTXT,
     $               MXLLDC, INFO )
*
*      -------------------------------------------------------
*      GENERATE MATRICES A / C AND DISTRIBUTE ON THE PROCESS GRID
*      -------------------------------------------------------
       CALL MATINIT( A, DESCA , C , DESCC )
*
*
*      ------------------------
*      CALL THE ROUTINE PDGEQRF
*      ------------------------
       CALL PDGEQRF( M, N, A, IA, JA, DESCA, TAU, WORK, LWORK,
     $                    INFO )
*
*      -----------------------
*      Multiplies Q determined by PDGEQRF with C
*      -----------------------
       CALL PDORMQR( 'L', 'T', M, 1, K, A, IA, JA, DESCA, TAU,
     $               C, IC, JC, DESCC, WORK, LWORK, INFO )
*       IF( MYROW.EQ.0 .AND. MYCOL.EQ.0 ) THEN
         WRITE( NOUT, FMT = 9999 )
         WRITE( NOUT, FMT = 9998 )M, N, NB
         WRITE( NOUT, FMT = 9997 )NPROW*NPCOL, NPROW, NPCOL
         WRITE( NOUT, FMT = 9996 )INFO
         END IF
*
*      ------------------------
*      RELEASE THE PROCESS GRID
*      ------------------------
*
      CALL BLACS_GRIDEXIT( ICTXT )
*
 10   CONTINUE
*
*
*
       CALL BLACS_EXIT( 0 )
*
*
 9999 FORMAT( / 'ScaLAPACK tESTING QR' )
 9998 FORMAT( / 'COMPUTING A=QR WHERE A IS A ', I3, ' by ', I3,
     $      ' matrix with a block size of ', I3 )
 9997 FORMAT( 'Running on ', I3, ' processes, where the process grid',
     $      ' is ', I3, ' by ', I3 )
 9996 FORMAT( / 'INFO code returned = ', I3 )
      STOP
       END
*
*
*
      SUBROUTINE MATINIT( AA, DESCA , CC , DESCC)
*
*     MATINIT generates and distributes matrices A and B (depicted in
*     figures 2.5 and 2.6) to a 2 x 3 process grid
*
*     .. Array Arguments ..
      INTEGER            DESCA( * ), DESCC( * )
      DOUBLE PRECISION   AA( * )   , CC (*)
*     .. Parameters ..
      INTEGER            CTXT_, LLD_
      PARAMETER          ( CTXT_ = 2, LLD_ = 9 )
*     ..
*     .. Local Scalars ..
      INTEGER            ICTXT, MXLLDA, MYCOL, MYROW, NPCOL, NPROW
      DOUBLE PRECISION   A, C, K, L, P, S
*     ..
*     .. External Subroutines ..
      EXTERNAL           BLACS_GRIDINFO
*     ..
*     .. Executable Statements ..
*
      ICTXT = DESCA( CTXT_ )
      CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL )
*
      S = 19.0D0
      C = 3.0D0
      A = 1.0D0
      L = 12.0D0
      P = 16.0D0
      K = 11.0D0
*
      MXLLDA = DESCA( LLD_ )
*
      IF( MYROW.EQ.0 .AND. MYCOL.EQ.0 ) THEN
         AA( 1 ) = S
         AA( 2 ) = -S
         AA( 3 ) = -S
         AA( 4 ) = -S
         AA( 5 ) = -S
         AA( 1+MXLLDA ) = C
         AA( 2+MXLLDA ) = C
         AA( 3+MXLLDA ) = -C
         AA( 4+MXLLDA ) = -C
         AA( 5+MXLLDA ) = -C
         AA( 1+2*MXLLDA ) = A
         AA( 2+2*MXLLDA ) = A
         AA( 3+2*MXLLDA ) = A
         AA( 4+2*MXLLDA ) = A
         AA( 5+2*MXLLDA ) = -A
         AA( 1+3*MXLLDA ) = C
         AA( 2+3*MXLLDA ) = C
         AA( 3+3*MXLLDA ) = C
         AA( 4+3*MXLLDA ) = C
         AA( 5+3*MXLLDA ) = -C
         CC( 1 ) = 0.0D0
         CC( 2 ) = 0.0D0
         CC( 3 ) = 0.0D0
         CC( 4 ) = 0.0D0
         CC( 5 ) = 0.0D0
      ELSE IF( MYROW.EQ.0 .AND. MYCOL.EQ.1 ) THEN
         AA( 1 ) = A
         AA( 2 ) = A
         AA( 3 ) = -A
         AA( 4 ) = -A
         AA( 5 ) = -A
         AA( 1+MXLLDA ) = L
         AA( 2+MXLLDA ) = L
         AA( 3+MXLLDA ) = -L
         AA( 4+MXLLDA ) = -L
         AA( 5+MXLLDA ) = -L
         AA( 1+2*MXLLDA ) = K
         AA( 2+2*MXLLDA ) = K
         AA( 3+2*MXLLDA ) = K
         AA( 4+2*MXLLDA ) = K
         AA( 5+2*MXLLDA ) = K
      ELSE IF( MYROW.EQ.0 .AND. MYCOL.EQ.2 ) THEN
         AA( 1 ) = A
         AA( 2 ) = A
         AA( 3 ) = A
         AA( 4 ) = -A
         AA( 5 ) = -A
         AA( 1+MXLLDA ) = P
         AA( 2+MXLLDA ) = P
         AA( 3+MXLLDA ) = P
         AA( 4+MXLLDA ) = P
         AA( 5+MXLLDA ) = -P
      ELSE IF( MYROW.EQ.1 .AND. MYCOL.EQ.0 ) THEN
         AA( 1 ) = -S
         AA( 2 ) = -S
         AA( 3 ) = -S
         AA( 4 ) = -S
         AA( 1+MXLLDA ) = -C
         AA( 2+MXLLDA ) = -C
         AA( 3+MXLLDA ) = -C
         AA( 4+MXLLDA ) = C
         AA( 1+2*MXLLDA ) = A
         AA( 2+2*MXLLDA ) = A
         AA( 3+2*MXLLDA ) = A
         AA( 4+2*MXLLDA ) = -A
         AA( 1+3*MXLLDA ) = C
         AA( 2+3*MXLLDA ) = C
         AA( 3+3*MXLLDA ) = C
         AA( 4+3*MXLLDA ) = C
         CC( 1 ) = 1.0D0
         CC( 2 ) = 0.0D0
         CC( 3 ) = 0.0D0
         CC( 4 ) = 0.0D0
      ELSE IF( MYROW.EQ.1 .AND. MYCOL.EQ.1 ) THEN
         AA( 1 ) = A
         AA( 2 ) = -A
         AA( 3 ) = -A
         AA( 4 ) = -A
         AA( 1+MXLLDA ) = L
         AA( 2+MXLLDA ) = L
         AA( 3+MXLLDA ) = -L
         AA( 4+MXLLDA ) = -L
         AA( 1+2*MXLLDA ) = K
         AA( 2+2*MXLLDA ) = K
         AA( 3+2*MXLLDA ) = K
         AA( 4+2*MXLLDA ) = K
      ELSE IF( MYROW.EQ.1 .AND. MYCOL.EQ.2 ) THEN
         AA( 1 ) = A
         AA( 2 ) = A
         AA( 3 ) = -A
         AA( 4 ) = -A
         AA( 1+MXLLDA ) = P
         AA( 2+MXLLDA ) = P
         AA( 3+MXLLDA ) = -P
         AA( 4+MXLLDA ) = -P
      END IF
      RETURN
      END

thanks a lot !!!!!

[Julien Langou]

At first glance, your code is full of mistake.

1- LWORK is -1, then you allocate an array WORK of size LWORK (so -1).
What does it mean?

2- The first time you call ScaLAPACK PDGEQRF with LWORK=-1,
ScaLAPACK is not going to perform anything but the computation of the
workspace size necessary to perform, so yes your code without PDORMQR
does not crash but it is not doing exactly what you want, PDGEQRF is
going to write the optimal value of LWORK in WORK(1). So since WORK is
an array of size -1 and we are still trying to figure out how your compiler
has interpreted this, PDGEQRF is probably going to write somewhere
in your memory and it is no surprise if your code crashes at the next
ScaLAPACK code.

3- When you set LWORK=200, then you see an error message INFO=-5
coming from PDORMQR, well this means that the value of K is not correct
with respect to previous value of other parameters, so
if SIDE = 'L' you want K to be between 0 and M,
if SIDE = 'R' you want K to be between 0 and N.
In your case you have SIDE = 'R', N=1 and K=9, PDORMQR is right that
does not work.

For your workspace query problem (#2), you need to set LWORK=-1, you
allocate WORK an array of one element and then you call ScaLAPACK and
in output you (only) get in WORK(1) the optimal size for WORK, no
computation is done, WORK(1) depends on SIDE, TRANS, M, N, K, + proc
coordinates + etc. Once you have this number, you put it in LWORK,
deallocate WORK, allocate WORK of size LWORK and then call the
ScaLAPACK routine again.

Julien.

[arad]

Well the "R" option i put it here accidentally (because I pasted the code while i was doing some experiments on it)..

well I had it as L but again there was problem, the same problem...

now i read your message (thank you very much for your time) and i try to understand the purpose of LWORK... this LWORK value is same for both routines ? it confuces me so much...

I will read again your reply !

[Julien Langou]

Here is a suggestion,
you keep almost the same program, just that you write
DOUBLE PRECISION WORK(1)
instead of what you have written.
After each ScaLAPACK call, you write the value of WORK(1)
on each processor. This is the size you want WORK to be if
you want ScaLAPACK to run (optimized).

[arad]

dear it runned after setting LWORK = 200 and some other changes!! no error returned and info =0.. i will try your suggestion !! but tell me ..what will happen if i set LWORK =200 or other value that might not be the exact that the program wants ? could this value return wrong results ? is it just spending more memory or it destroys the results ?

here is my code now

Code: Select all

       PROGRAM TESTINGQR
*
*      PARAMETERS
       INTEGER      M, N, MB, NB, RSRC, CSRC, MXLLDA,
     $              IA, JA, LWORK , MXLOCC , NOUT, DLEN_ , K ,IC, JC,
     $              NRHS, T , MXLLDC , NBRHS
       PARAMETER  ( M = 9, N = 9, MB = 2, NB = 2, RSRC = 0, CSRC = 0,
     $              MXLLDA = 5, IA = 1, JA = 1, LWORK = 200, MXLOCC = 4,
     $              NOUT = 6, DLEN_ = 9, K = 8, IC = 1, JC = 1,
     $              MXLLDC = 5, NBRHS = 1, NRHS = 1)
*      SCALARS   *
       INTEGER      ICTXT, MYROW, MYCOL, NPCOL, NPROW, INFO
*
*      ARRAYS
       INTEGER DESCA(DLEN_) , DESCC(DLEN_)
       DOUBLE PRECISION A(MXLLDA, MXLOCC), TAU(1,5), WORK( LWORK ),
     $                  C(5,1)
*
*      EXTERNAL SUBROUTINES
       EXTERNAL    BLACS_EXIT, BLACS_GRIDEXIT, BLACS_GRIDINFO,
     $             DESCINIT, MATINIT, SL_INIT
*
       DATA NPROW / 2 /, NPCOL / 3 /
*
*      ---------------------------
*      INITIALIZE THE PROCESS GRID
*      ---------------------------
       CALL SL_INIT( ICTXT, NPROW, NPCOL )
       CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL )
*
*     If I'm not in the process grid, go to the end of the program
*
      IF( MYROW.EQ.-1 )
     $ GO TO 10
*
*      -------------------------------------------
*      DISTRIBUTE THE MATRIX OVER THE PROCESS GRID
*      -------------------------------------------
       CALL DESCINIT( DESCA, M, N, MB, NB, RSRC, CSRC, ICTXT, 
     $               MXLLDA, INFO )
       CALL DESCINIT( DESCC, N, NRHS , NB, NBRHS , RSRC, CSRC, ICTXT,
     $               MXLLDC, INFO )
*
*      -------------------------------------------------------
*      GENERATE MATRICES A / C AND DISTRIBUTE ON THE PROCESS GRID
*      -------------------------------------------------------
       CALL MATINIT( A, DESCA , C , DESCC )
*
*
*      ------------------------
*      CALL THE ROUTINE PDGEQRF
*      ------------------------
       CALL PDGEQRF( M, N, A, IA, JA, DESCA, TAU, WORK, LWORK,
     $                    INFO )
*
*      -----------------------
*      Multiplies Q determined by PDGEQRF with C
*      -----------------------
       CALL PDORMQR( 'L', 'T', M, 1, K, A, IA, JA, DESCA, TAU,
     $               C, IC, JC, DESCC, WORK, LWORK, INFO )
*
*
       IF( MYROW.EQ.0 .AND. MYCOL.EQ.0 ) THEN
         WRITE( NOUT, FMT = 9999 )
         WRITE( NOUT, FMT = 9998 )M, N, NB
         WRITE( NOUT, FMT = 9997 )NPROW*NPCOL, NPROW, NPCOL
         WRITE( NOUT, FMT = 9996 )INFO
         END IF
*
*      ------------------------
*      RELEASE THE PROCESS GRID
*      ------------------------
*
      CALL BLACS_GRIDEXIT( ICTXT )
*
 10   CONTINUE
*
*
*
       CALL BLACS_EXIT( 0 )
*
*
 9999 FORMAT( / 'ScaLAPACK tESTING QR' )
 9998 FORMAT( / 'COMPUTING A=QR WHERE A IS A ', I3, ' by ', I3,
     $      ' matrix with a block size of ', I3 )
 9997 FORMAT( 'Running on ', I3, ' processes, where the process grid',
     $      ' is ', I3, ' by ', I3 )
 9996 FORMAT( / 'INFO code returned by PDGEQRF = ', I3 )
      STOP
       END
*
*
*
      SUBROUTINE MATINIT( AA, DESCA , CC , DESCC)
*
*     MATINIT generates and distributes matrices A and B (depicted in
*     figures 2.5 and 2.6) to a 2 x 3 process grid
*
*     .. Array Arguments ..
      INTEGER            DESCA( * ), DESCC( * )
      DOUBLE PRECISION   AA( * )   , CC (*)
*     .. Parameters ..
      INTEGER            CTXT_, LLD_
      PARAMETER          ( CTXT_ = 2, LLD_ = 9 )
*     ..
*     .. Local Scalars ..
      INTEGER            ICTXT, MXLLDA, MYCOL, MYROW, NPCOL, NPROW
      DOUBLE PRECISION   A, C, K, L, P, S
*     ..
*     .. External Subroutines ..
      EXTERNAL           BLACS_GRIDINFO
*     ..
*     .. Executable Statements ..
*
      ICTXT = DESCA( CTXT_ )
      CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL )
*
      S = 19.0D0
      C = 3.0D0
      A = 1.0D0
      L = 12.0D0
      P = 16.0D0
      K = 11.0D0
*
      MXLLDA = DESCA( LLD_ )
*
      IF( MYROW.EQ.0 .AND. MYCOL.EQ.0 ) THEN
         AA( 1 ) = S
         AA( 2 ) = -S
         AA( 3 ) = -S
         AA( 4 ) = -S
         AA( 5 ) = -S
         AA( 1+MXLLDA ) = C
         AA( 2+MXLLDA ) = C
         AA( 3+MXLLDA ) = -C
         AA( 4+MXLLDA ) = -C
         AA( 5+MXLLDA ) = -C
         AA( 1+2*MXLLDA ) = A
         AA( 2+2*MXLLDA ) = A
         AA( 3+2*MXLLDA ) = A
         AA( 4+2*MXLLDA ) = A
         AA( 5+2*MXLLDA ) = -A
         AA( 1+3*MXLLDA ) = C
         AA( 2+3*MXLLDA ) = C
         AA( 3+3*MXLLDA ) = C
         AA( 4+3*MXLLDA ) = C
         AA( 5+3*MXLLDA ) = -C
         CC( 1 ) = 0.0D0
         CC( 2 ) = 0.0D0
         CC( 3 ) = 0.0D0
         CC( 4 ) = 0.0D0
         CC( 5 ) = 0.0D0
      ELSE IF( MYROW.EQ.0 .AND. MYCOL.EQ.1 ) THEN
         AA( 1 ) = A
         AA( 2 ) = A
         AA( 3 ) = -A
         AA( 4 ) = -A
         AA( 5 ) = -A
         AA( 1+MXLLDA ) = L
         AA( 2+MXLLDA ) = L
         AA( 3+MXLLDA ) = -L
         AA( 4+MXLLDA ) = -L
         AA( 5+MXLLDA ) = -L
         AA( 1+2*MXLLDA ) = K
         AA( 2+2*MXLLDA ) = K
         AA( 3+2*MXLLDA ) = K
         AA( 4+2*MXLLDA ) = K
         AA( 5+2*MXLLDA ) = K
      ELSE IF( MYROW.EQ.0 .AND. MYCOL.EQ.2 ) THEN
         AA( 1 ) = A
         AA( 2 ) = A
         AA( 3 ) = A
         AA( 4 ) = -A
         AA( 5 ) = -A
         AA( 1+MXLLDA ) = P
         AA( 2+MXLLDA ) = P
         AA( 3+MXLLDA ) = P
         AA( 4+MXLLDA ) = P
         AA( 5+MXLLDA ) = -P
      ELSE IF( MYROW.EQ.1 .AND. MYCOL.EQ.0 ) THEN
         AA( 1 ) = -S
         AA( 2 ) = -S
         AA( 3 ) = -S
         AA( 4 ) = -S
         AA( 1+MXLLDA ) = -C
         AA( 2+MXLLDA ) = -C
         AA( 3+MXLLDA ) = -C
         AA( 4+MXLLDA ) = C
         AA( 1+2*MXLLDA ) = A
         AA( 2+2*MXLLDA ) = A
         AA( 3+2*MXLLDA ) = A
         AA( 4+2*MXLLDA ) = -A
         AA( 1+3*MXLLDA ) = C
         AA( 2+3*MXLLDA ) = C
         AA( 3+3*MXLLDA ) = C
         AA( 4+3*MXLLDA ) = C
         CC( 1 ) = 1.0D0
         CC( 2 ) = 0.0D0
         CC( 3 ) = 0.0D0
         CC( 4 ) = 0.0D0
      ELSE IF( MYROW.EQ.1 .AND. MYCOL.EQ.1 ) THEN
         AA( 1 ) = A
         AA( 2 ) = -A
         AA( 3 ) = -A
         AA( 4 ) = -A
         AA( 1+MXLLDA ) = L
         AA( 2+MXLLDA ) = L
         AA( 3+MXLLDA ) = -L
         AA( 4+MXLLDA ) = -L
         AA( 1+2*MXLLDA ) = K
         AA( 2+2*MXLLDA ) = K
         AA( 3+2*MXLLDA ) = K
         AA( 4+2*MXLLDA ) = K
      ELSE IF( MYROW.EQ.1 .AND. MYCOL.EQ.2 ) THEN
         AA( 1 ) = A
         AA( 2 ) = A
         AA( 3 ) = -A
         AA( 4 ) = -A
         AA( 1+MXLLDA ) = P
         AA( 2+MXLLDA ) = P
         AA( 3+MXLLDA ) = -P
         AA( 4+MXLLDA ) = -P
      END IF
      RETURN
      END

[arad]

I set now LWORK=200 and i called to return the value of C after the multiplication... i got responce from all 6 processors and two of them (the 0 , 0 and the 1 , 0) returned me values while others just zeros because the vector C is distributed only in 0,0 and 1,0 processors..

then i set LWORK=100 and i got the same exact results from C

when i set LWORK =50 i got only the non zeros results from those 2 processors wheareas the other processors (zeros) didnt appear.. the result of the appeared one though where the same as LWORK=200 etc... so i guess i get the right results...

plus when i set LWORK=-1 i got the problem i told in my first post...
so i m ready to go in the next step with solving triangular system..

thank you very much for answering fast , you are great.

[arad]

ok I Got the triangular system done...

well there is a problem...the solution is not the exact as given in the example 1 (i used the same matrices) .. However 2 values from the solution vector are correct... weird !

[arad]

Here is a suggestion,
you keep almost the same program, just that you write
DOUBLE PRECISION WORK(1)
instead of what you have written.
After each ScaLAPACK call, you write the value of WORK(1)
on each processor. This is the size you want WORK to be if
you want ScaLAPACK to run (optimized).

Ok i did it... after the first call i wrote as u told me and i called the work(1) of each processor

the results i got were these

FOR PROCESSOR 0 - 0 THEN work = 22.0
FOR PROCESSOR 1 - 0 THEN work = 20.0
FOR PROCESSOR 0 - 2 THEN work = 18.0
FOR PROCESSOR 1 - 2 THEN work = 16.0
FOR PROCESSOR 1 - 1 THEN work = 18.0
FOR PROCESSOR 0 - 1 THEN work = 20.0

WHEREAS AFTER THE CALL OF THE SECOND ROUTINE I GOT THESE

FOR PROCESSOR 1 - 1 THEN work = 12.0
FOR PROCESSOR 0 - 1 THEN work = 14.0
FOR PROCESSOR 1 - 2 THEN work = 12.0
FOR PROCESSOR 0 - 2 THEN work = 14.0
FOR PROCESSOR 1 - 0 THEN work = 14.0
FOR PROCESSOR 0 - 0 THEN work = 16.0

ETC

SO WHATS THE CORRECT VALUE I SHOULD SET IN THE MAIN LWORK VALUE OF THE PROGRAM ?

[Julien Langou]

Very good, so as you can see the optimal workspace is not the same on
all the processors and is not the same from one routine to the other.
This is completely expected.

To have it easy on this one, take DOUBLE PRECISION WORK(22) and
LWORK=22 at the beginning of your code and that should be OK.

-julien.