collecting the distributed data

[klangfeld]

Dear All,

I have calculated the eigenvalues using PZHEEV
using 16 nodes. This all works fine. PZHEEV exits with info=0.

How do I most efficiently collect the eigenvalues from the nodes?
I tried PDELGET with little success. I can write the eigenvalues
to a file with PDLAWRITE, but that is not what I need.

One could work with MPI_SEND etc, but I hope that there is a simple
solution to this standard solution. I apologize if this has been covered before....

Thank you in advance for any information.

Best regards,

Kurt

[klangfeld]

Dear All,

My problem seems to be bigger than I thought.
Please have a look at the toy problem below; the code is in f90.

Instead of getting all 1296 eigenvalues, I get only the first 324, but those
4 times.

Any help concerning the data distribution would be very welcome.

Best regards,

Kurt

=============================================================
program main

!*****************************************************************************80
!
!! eigenvalues using ScaLAPACK.
!
! matrix size: A(1296,1296), A(i,k) = i for i=k, 0 else
! eigenvalues: 1 .... 1296
! block size : 324
! processor grid: 4x4
!

implicit none

integer, parameter :: nb=324

complex*16 :: a(nb,nb), z(nb,nb)
integer, parameter :: lwork=106272, lrwork=648
complex*16 :: work(lwork),rwork(lrwork),beta

double precision :: w(nb),alpha
integer :: context, desca(9), descz(9), descw(9)
integer :: i,j,iam,info,m,mycol,myrow,nprocs,nprow,npcol
integer :: err


CALL MPI_INIT (ERR)

nprow =4
npcol =4

call blacs_pinfo ( iam, nprocs )

write ( *, '(a)' ) ' '
write ( *, '(a,i6,a)' ) ' Process ', iam, ' is active.'


call blacs_get ( -1, 0, context )
call blacs_gridinit ( context, 'Row-major', nprow, npcol )
call blacs_gridinfo ( context, nprow, npcol, myrow, mycol )

! sets the absolute size of the matrix
m = nprow * nb

! initialises the matrix descriptors
call descinit ( desca, m, m, nb,nb, 0, 0, context, nb, info )
descz = desca
call descinit ( descw, m, 1, nb, 1, 0, 0, context, nb, info )

do i=1,m
do j=1,m
if (i==j) then
CALL PZELSET( A, I, J, DESCA, dcmplx(i,0d0) )
else
CALL PZELSET( A, I, J, DESCA, dcmplx(0d0,0d0) )
end if
end do
end do

! just checking which local matrix has arrived at the processor grid 2,2:

if ( myrow == 2 .and. mycol == 2 ) then
do i=1,nb
write(*,*) i,a(i,i)
end do
end if


!
! Solve the system.
!

! lwork= -1 !!! only to get the size of WORK
! lrwork=-1

call PZHEEV( 'N', 'U', nb, A, 1, 1, DESCA, W, Z, 1, 1, &
DESCZ, WORK, LWORK, RWORK, LRWORK, INFO )

! if ( myrow == 0 .and. mycol == 0 ) write(*,*) work(1),rwork(1)
! call blacs_exit ( 0 )
! stop



if ( myrow == 0 .and. mycol == 0 ) then
write ( *, '(a)' ) ' '
write ( *, '(a,i6)' ) ' PZHEEV returned INFO = ', info
end if

call PDLAWRITE( 'out', M, 1, W, 1, 1, DESCW, 0,0, WORK )


!
! Shut down the processes.
!
! Free the BLACS context.
!
call blacs_gridexit ( context )
!
! Break the connection of this process to the BLACS.
!
call blacs_exit ( 0 )

if ( myrow == 0 .and. mycol == 0 ) then
write ( *, '(a)' ) ' '
write ( *, '(a)' ) 'test:'
write ( *, '(a)' ) ' Normal end of execution.'
end if

stop
end

[klangfeld]

Hi All,

The problem is solved.

Key points are:
* Since the matrix A is (potentially) too big to fit into the memory of one node,
it is important to allocate it only to blocksize NB<< absolute dimension M: -> complex*16 :: A(nb,nb)
* The 1-dimensional array W which contains the eigenvalues IS of absolute size: -> double precision:: w(M)
* the eigenvalues w(1..M) are the just available at the "headnode" myrow=0 and mycol=0

The amended code below produces the correct result using 16 nodes.

Best regards,

Kurt

program main

!*****************************************************************************80
!

!! eigenvalues using ScaLAPACK.
!
! matrix size: A(1296,1296), A(i,k) = i for i=k, 0 else
! eigenvalues: 1 .... 1296
! block size : 324
! processor grid: 4x4
!

implicit none

integer, parameter :: nb=324, m=1296

complex*16 :: a(nb,nb), z(nb,nb)
integer, parameter :: lwork=109188, lrwork=2592
complex*16 :: work(lwork),rwork(lrwork)

! complex*16 :: work(1),rwork(1) !!! initial setting to find the size of work and rwork
! integer :: lwork,lrwork

double precision :: w(m) !!! w is of size: w(1296)
integer :: context, desca(9), descz(9), descw(9)
integer :: i,j,iam,info,mycol,myrow,nprocs,nprow,npcol
integer :: err


CALL MPI_INIT (ERR)

nprow =4
npcol =4

call blacs_pinfo ( iam, nprocs )

write ( *, '(a)' ) ' '
write ( *, '(a,i6,a)' ) ' Process ', iam, ' is active.'


call blacs_get ( -1, 0, context )
call blacs_gridinit ( context, 'Row-major', nprow, npcol )
call blacs_gridinfo ( context, nprow, npcol, myrow, mycol )


call descinit ( desca, m, m, nb,nb, 0, 0, context, nb, info )
descz = desca
call descinit ( descw, m, 1, m, 1, 0, 0, context, m, info )

do i=1,m
do j=1,m
if (i==j) then
CALL PZELSET( A, I, J, DESCA, dcmplx(i,0d0) )
else
CALL PZELSET( A, I, J, DESCA, dcmplx(0d0,0d0) )
end if
end do
end do

if ( myrow == 2 .and. mycol == 2 ) then !!!! just checking whether the correct matrix arrived at the grid (2,2)
do i=1,nb
write(*,*) myrow,mycol,i,a(i,i)
end do
end if


! lwork= -1 !!! to find the size of work and rwork
! lrwork=-1
call PZHEEV( 'N', 'U', M , A, 1,1, DESCA, W, Z, 1, 1, &
DESCZ, WORK, LWORK, RWORK, LRWORK, INFO )

! if ( myrow == 0 .and. mycol == 0 ) write(*,*) work(1),rwork(1)
! call blacs_exit ( 0 )
! stop



if ( myrow == 0 .and. mycol == 0 ) then
write ( *, '(a)' ) ' '
write ( *, '(a,i6)' ) ' PZHEEV returned INFO = ', info
do i=1,m
write(*,*) i,w(i) !!! here we print ALL eigenvalues
end do
end if


call blacs_gridexit ( context )
!
call blacs_exit ( 0 )

if ( myrow == 0 .and. mycol == 0 ) then
write ( *, '(a)' ) ' '
write ( *, '(a)' ) 'test:'
write ( *, '(a)' ) ' Normal end of execution.'
end if

stop
end