PDLAWRITE can not be recgonized

[bruce]

Hi, there:

I am using PDSYEV routine to get eigenvalues and eigenvectors. I am trying to use PDLAWRITE routine to write eigenvectors to a txt file. However, I found that I did not have this routine PDLAWRITE in my current SCALAPACK libraries directories/SRC and ../TOOLS. So I downloaded the fortran source code of PDLAWRITE and put it in my current directory with my main function source code.

I called PDLAPRNT without any problem from my C code but when I called PDLAWRITE routine. It gave me error saying that function not referenced.

[xiaofeng@athena ver5]$ make
mpicc -c pdsyev.c -o pdsyev.o
mpif77 pdsyev.o /home/xiaofeng/SCALAPACK/libscalapack.a /home/xiaofeng/BLACS/LIB/blacsCinit_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacsF77init_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacs_MPI-LINUX-1.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libf77blas.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libatlas.a /opt/MPICH2/lib/libmpich.a -o pdsyev
pdsyev.o(.text+0x8d0): In function `main':
: undefined reference to `pdlawrite_'
collect2: ld returned 1 exit status
make: *** [pdsyev] Error 1
========================

I declared the function in the scalapack.h header file and even in my main function C code. It does not help. I am thinking if I passed the wrong parameter type so the compiler can not recgonize the routine.

I tried to copy PDLAWRITE.f to /$SCALAPACKHOME/TOOLS and ../SRC and then rebuild the library. It still doesn't work.

Any help is highly appreciated!

Bruce

[Julien Langou]

Hello,

pdlaprnt is part of ScaLAPACK, pdlawrite is not.

If you want to use pdlawrite, you simply need to compile it yourself 'by hand'.
and then add it yourself at link time.

It's not because you copy the file in the ScaLAPACK directory SRC or TOOLS that the
file will be compiled and add to the ScaLAPACK library. To do this, you need to change one
of the Makefile. This is not hard but I'd recommend you do this by hand. If you change
you ScaLAPACK library to add pdlawrite, your code will be relying on a nonstandard
scalapack library and you will end up with a non portable library.

Actually, we might want to add pdlawrite in ScaLAPACK in a next release. This is a useful
piece of code. Thanks for reminding this to us.

Best wishes,
Julien.

[bruce]

Hi, Julien:

Thank you for your answer! I appreciated!. Do you mean even in version 1.7.5 there is no PDLAWRITE. I am using 1.7.4 or 1.7.0. What do you mean by next version. Is it version 2.0?

I will try that and see if I could do that. Thank you.

Bruce

[Julien Langou]

Yep I confirm no PDLAWRITE up to 1.7.5 which is the latest release.
You have to download it, compile it and link with it on your own.
That's the way it works for this routine.

We could insert PDLAWRITE fairly easily: just needs to write and test the S, C
and Z variants. We'll put it on the TODO list. I think this can be useful for
others.

Best wishes,
Julien.

[bruce]

I appreciated your help! Yes, scalapack is going to be a standard and some of the tools are very helpful!

Best regards,

Bruce

[bruce]

Hi, Julien:

I compiled and linked the PDLAWRITE to my main program. I also modified the Makefile of compiling the main program. I have no problem with compiling and linking. However, when I was trying to run the executable, I got error of MPI finalizing..., I have no idea of this. Can you help me to take a look at it?

[xiaofeng@athena ver5]$ make
mpicc -c pdsyev.c -o pdsyev.o
mpif77 pdsyev.o /home/xiaofeng/SCALAPACK/libscalapack.a /home/xiaofeng/BLACS/LIB/blacsCinit_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacsF77init_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacs_MPI-LINUX-1.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libf77blas.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libatlas.a /opt/MPICH2/lib/libmpich.a pdlawrite.o /home/xiaofeng/SCALAPACK/PBLAS/SRC/pdgeadd_.o /home/xiaofeng/SCALAPACK/PBLAS/SRC/PTOOLS/PB_Cpgeadd.o /home/xiaofeng/SCALAPACK/PBLAS/SRC/PTOOLS/PB_Cptran.o -o pdsyev
[xiaofeng@athena ver5]$ mpiexec -n 4 pdsyev
myrank_mpi is: 0
nprocs_mpi is: 4
myrank_mpi is: 1
nprocs_mpi is: 4
myrank_mpi is: 3
nprocs_mpi is: 4
myrank_mpi is: 2
nprocs_mpi is: 4


1.000000 0.800000 1.600000 2.400000 3.200000 4.000000
0.800000 1.000000 2.400000 3.200000 4.000000 4.800000
1.600000 2.400000 1.000000 4.000000 4.800000 5.600000
2.400000 3.200000 4.000000 1.000000 5.600000 6.400000
3.200000 4.000000 4.800000 5.600000 1.000000 7.200000
4.000000 4.800000 5.600000 6.400000 7.200000 1.000000
this is to test LDA and LDB
LDA is 3
LDB is 3

1.000000 0.800000 1.600000 2.400000 3.200000 4.000000
0.800000 1.000000 2.400000 3.200000 4.000000 4.800000
1.600000 2.400000 1.000000 4.000000 4.800000 5.600000
2.400000 3.200000 4.000000 1.000000 5.600000 6.400000
3.200000 4.000000 4.800000 5.600000 1.000000 7.200000
4.000000 4.800000 5.600000 6.400000 7.200000 1.000000
this is to test LDA and LDB
LDA is 3
LDB is 3

1.000000 0.800000 1.600000 2.400000 3.200000 4.000000
0.800000 1.000000 2.400000 3.200000 4.000000 4.800000
1.600000 2.400000 1.000000 4.000000 4.800000 5.600000
2.400000 3.200000 4.000000 1.000000 5.600000 6.400000
3.200000 4.000000 4.800000 5.600000 1.000000 7.200000
4.000000 4.800000 5.600000 6.400000 7.200000 1.000000
this is to test LDA and LDB
LDA is 3
LDB is 3


1.000000 0.800000 1.600000 2.400000 3.200000 4.000000
0.800000 1.000000 2.400000 3.200000 4.000000 4.800000
1.600000 2.400000 1.000000 4.000000 4.800000 5.600000
2.400000 3.200000 4.000000 1.000000 5.600000 6.400000
3.200000 4.000000 4.800000 5.600000 1.000000 7.200000
4.000000 4.800000 5.600000 6.400000 7.200000 1.000000
this is to test LDA and LDB
LDA is 3
LDB is 3
info is: 0
e1=-6.713121
e2=-4.988263
e3=-3.260246
e4=-1.514115
e5=0.279999
e6=22.195745
( 1, 1)= -0.168588351613459725E+00
( 2, 1)= -0.178183914425807666E+00
( 3, 1)= -0.194583151036895469E+00
( 4, 1)= -0.228996593962212530E+00
( 5, 1)= -0.347273628460883554E+00
( 6, 1)= 0.853772478241143662E+00
( 1, 2)= 0.191039758784591007E+00
( 2, 2)= 0.214318630151213080E+00
( 3, 2)= 0.264313919369151940E+00
( 4, 2)= 0.448946894859801104E+00
( 5, 2)= -0.801377226026875777E+00
( 6, 2)= -0.628547099473254212E-01
( 1, 3)= 0.228077511608766825E+00
( 2, 3)= 0.293372637798953007E+00
( 3, 3)= 0.555739443525711141E+00
( 4, 3)= -0.737367791271290485E+00
( 5, 3)= -0.966346871691139420E-01
( 6, 3)= -0.415844086107383659E-02
( 1, 4)= -0.317141701645299257E+00
( 2, 4)= -0.679935257369152057E+00
( 3, 4)= 0.649887422909681156E+00
( 4, 4)= 0.118097049002848509E+00
( 5, 4)= 0.242324963350731837E-01
( 6, 4)= -0.148794336542613648E-01
( 1, 5)= 0.845666045891930285E+00
( 2, 5)= -0.517844412730845582E+00
( 3, 5)= -0.121986064085094909E+00
( 4, 5)= -0.366049024294220482E-01
( 5, 5)= 0.674190668543794835E-03
( 6, 5)= 0.215668677154208233E-01
( 1, 6)= 0.259485080236158794E+00
( 2, 6)= 0.325231798554012963E+00
( 3, 6)= 0.382354492562651860E+00
( 4, 6)= 0.432445550422359493E+00
( 5, 6)= 0.476728053931297580E+00
( 6, 6)= 0.516156973096453142E+00
rank 3 in job 73 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 3: killed by signal 11
0 starting to write vector to a text file...
1 starting to write vector to a text file...
3 starting to write vector to a text file...
[cli_1]: aborting job:
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(233).........................: MPI_Finalize failed
MPI_Finalize(153).........................:
MPID_Finalize(91).........................:
MPIDI_CH3U_VC_WaitForClose(219)...........: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3_Progress_wait(217)..............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(415):
MPIDU_Socki_handle_read(670)..............: connection failure (set=0,sock=3,errno=104:Connection reset by peer)
[cli_0]: aborting job:
Fatal error in MPI_Testall: See the MPI_ERROR field in MPI_Status for the error code
rank 1 in job 73 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 1: return code 1
rank 0 in job 73 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 0: return code 1
[xiaofeng@athena ver5]$
================================



The makefile of compiling my whole code is like this:
********************************************
include Makefile.opts


COMPILE = $(CC) -c
LIBPATH = /home/xiaofeng/SCALAPACK/PBLAS/SRC/PTOOLS
%.o:%.c
$(COMPILE) $*.c -o $@

SOURCES=psdyev.c
OBJECTS=$(SOURCES:%.c=%.o)

PDSYEV=pdsyev


all: $(PDSYEV)



LINK=$(F77) $(LDFLAGS)

LIBS = $(LIBSCALAPACK) $(LIBBLACS) $(LIBBLAS) $(LIBMPI) pdlawrite.o /home/xiaofeng/SCALAPACK/PBLAS/SRC/pdgeadd_.o $(LIBPATH)/PB_Cpgeadd.o $(LIBPATH)/PB_Cptran.o

pdsyev: pdsyev.o
$(LINK) pdsyev.o $(LIBS) -o $@

clean:
rm -f $(PDSYEV)
******************************
Makefile.opts is like this:
CC = mpicc
F77 =mpif77
INCLUDEMPI = -I/opt/MPICH2/include


LIBSCALAPACK = /home/xiaofeng/SCALAPACK/libscalapack.a

LIBBLACS = /home/xiaofeng/BLACS/LIB/blacsCinit_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacsF77init_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacs_MPI-LINUX-1.a

LIBBLAS = /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libf77blas.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libatlas.a

LIBMPI = /opt/MPICH2/lib/libmpich.a
************************************

The eigenvectors you see is what it did by the PDLAPRNT routine.

The makefile for compiling and getting the PDLAWRITE object file is like this:
***********************
include Makefile.opts
F77=mpif77

COMPILE = $(F77) -c
all: pdlawrite.o
$(COMPILE) pdlawrite.f pdlawrite.o -l $(LIBS)


LIBS = $(LIBSCALAPACK) $(LIBBLACS) $(LIBBLAS) $(LIBMPI)
******************************

I guess my problem lies in the compilation of PDLAWRITE. I am still trying to find it.

Thank you for your help!

Bruce

[bruce]

Yes, I think my problem is because I got a bug pdlawrite.o file. I rerun the makefile and here is the error message, however, it still gave me an object file. Maybe I forgot to read it and linked it with my main code.

================================
[xiaofeng@athena PDLAWRITE]$ make
mpif77 -c pdlawrite.f pdlawrite.o -l /home/xiaofeng/SCALAPACK/libscalapack.a /home/xiaofeng/BLACS/LIB/blacsCinit_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacsF77init_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacs_MPI-LINUX-1.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libf77blas.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libatlas.a /opt/MPICH2/lib/libmpich.a
g77: pdlawrite.o: linker input file unused because linking not done
g77: -l/home/xiaofeng/SCALAPACK/libscalapack.a: linker input file unused because linking not done
g77: /home/xiaofeng/BLACS/LIB/blacsCinit_MPI-LINUX-1.a: linker input file unused because linking not done
g77: /home/xiaofeng/BLACS/LIB/blacsF77init_MPI-LINUX-1.a: linker input file unused because linking not done
g77: /home/xiaofeng/BLACS/LIB/blacs_MPI-LINUX-1.a: linker input file unused because linking not done
g77: /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libf77blas.a: linker input file unused because linking not done
g77: /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libatlas.a: linker input file unused because linking not done
g77: /opt/MPICH2/lib/libmpich.a: linker input file unused because linking not done


Can you give me some instructions for this compilation?

Thank you!

Bruce

[Julien Langou]

Code: Select all

mpif77 -c pdlawrite.f -o pdlawrite.o


creates the object file

Then you link the object file with your libraries to create your executable something like:

Code: Select all

mpif77 *o -l /home/xiaofeng/SCALAPACK/libscalapack.a /home/xiaofeng/BLACS/LIB/blacsCinit_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacsF77init_MPI-LINUX-1.a /home/xiaofeng/BLACS/LIB/blacs_MPI-LINUX-1.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libf77blas.a /home/xiaofeng/ATLAS/lib/linux_x8664sse2/libatlas.a /opt/MPICH2/lib/libmpich.a


[bruce]

Hi, Julien:

I am pretty sure that this time I did everything correctly like in your instructions. However, in running the code with PDLAWRITE , I still got the following errors.

====================================
rank 3 in job 102 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 3: killed by signal 11
[cli_0]: aborting job:
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(233).........................: MPI_Finalize failed
MPI_Finalize(153).........................:
MPID_Finalize(91).........................:
MPIDI_CH3U_VC_WaitForClose(219)...........: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3_Progress_wait(217)..............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(415):
MPIDU_Socki_handle_read(670)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)
[cli_1]: aborting job:
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(233).........................: MPI_Finalize failed
MPI_Finalize(153).........................:
MPID_Finalize(91).........................:
MPIDI_CH3U_VC_WaitForClose(219)...........: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3_Progress_wait(217)..............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(415):
MPIDU_Socki_handle_read(670)..............: connection failure (set=0,sock=3,errno=104:Connection reset by peer)
[cli_2]: aborting job:
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(233).........................: MPI_Finalize failed
MPI_Finalize(153).........................:
MPID_Finalize(91).........................:
MPIDI_CH3U_VC_WaitForClose(219)...........: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3_Progress_wait(217)..............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(415):
MPIDU_Socki_handle_read(670)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)
rank 2 in job 102 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 2: return code 1
rank 1 in job 102 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 1: return code 1
rank 0 in job 102 athena.cs.siu.edu_40934 caused collective abort of all ranks
exit status of rank 0: return code 1
==============================

For my output data, I can just use the pipeline to redirect the result on the screen to a txt file using
mpiexec -n 4 pdsyev > out.txt

Hope you can help me figure that out! I appreciated!

Bruce

[Julien Langou]

Hello, so from the error, I guess that the code did everything has expected and crashed in
the MPI_Finalize (or the BLACS_EXIT or etc.) So if this is the case, those bugs are hard to
correct.

First remove the MPI_Finalize and the BLACS_EXIT and Cie and the end of your code.
Do you still get the error?
If no, in this case, remove everything execpt:
- initilaization of the grid
- exit
Do you crash?
If yes well, your code should be five lines and easy to debug.

If no add progressively the line of codes to see which one MPI/BLACS does not like.
Sometimes you have too many process in the MPI environment and if you do not
block them from going in the ScaLAPACK code well, that leads to problem like this. For
example. Buts first try the frist two steps.

Julien.