LAPACK/ScaLAPACK Development

Hi,

I am trying to distribute a matrix on root process to other processes. In order to do that, I choose scalapack subrouinte pdgemr2d,

It works for some cases, but it sometimes gives error, which is as follows
"xxmr2d:out of memory"

I checked all allocation very carefully and it does correctly, but it is still problematic for some cases,

has anybody encontoured this kind of problem when using pdgemr2d subroutine,

I have copied important part of my applications which is written in C++,

sl_init( &ictxt, nprow, npcol );
bool isRootNode = false;

if ( myrow == 0 && mycol == 0 ) isRootNode = true;

//loaad matrix on root node and create desc for it
int descA0[9], lm_a0, info, rsrc = 0, csrc = 0;
if ( isRootNode ) {
lm_a0 = numroc( M, M, myrow, rsrc, nprow);
assert( input->getVecSpaceTimeMtrx().at(dim).rows() == lm_a0 );
descinit( descA0, M, N, M, N, rsrc, csrc, rootNodeContext, std::max(1,lm_a0), &info );
assert( !info );
} else {
for ( int i = 0; i < 9; i++ ) descA0[i] = 0;
descA0[1] = -1;
}

//create desc for distributed matrix
int lm_a, ln_a;
rsrc = 0; csrc = 0;

lm_a = numroc( M, block_size, myrow, rsrc, nprow);
ln_a = numroc( N, block_size, mycol, csrc, npcol);
assert( input->getLocalVecSpaceTimeMtrx().at(dim).rows() == lm_a );
assert( input->getLocalVecSpaceTimeMtrx().at(dim).cols() == ln_a );
descinit( descA.at(dim).data(), M, N, block_size, block_size, rsrc, csrc, ictxt, std::max(1,lm_a), &info);
assert( !info );

}


//distribute data
int IA = 1, JA = 1, IB = 1, JB = 1;

pdgemr2d( M, N, input->getVecSpaceTimeMtrxP(dim), IA, JA, descA0, input->getLocalVecSpaceTimeMtrxP(dim),
IB, JB, descA.at(dim).data(), descA.at(dim)(1));


I hope that you can help me out,

thanks anyway,

I have the same problem. The error occurs as soon my matrix is larger than 2GB. I'm investigating the issue now, but if anyone knows a solution, I'd be glad to hear it.

Thomas

have a look here guys:
http://icl.cs.utk.edu/lapack-forum/viewtopic.php?t=491

Interesting. Unfortunaly this is not the problem here. What we are looking at is p_gemr2d calling mr2d_malloc(int n), which is already negative at the sizes we are looking at. I'll try to work around it, by making it unsigned and long, but its also in the long term not a good idea that p_gemr2d needs a temporary buffer the size of the matrix its supposed to distribute.

This is a bit off-topic, but has anyone looked at BLACS or tried to optimize the whole BLACS communication shebang in ScaLapack in recent years ?

Thomas

Edit: changing the function mr2d_malloc to have an argument unsigned long int n instead of int n, does the trick.

Can you tell us ,where did you do the changes,and what changes exactly.?
Does scalapack 1.8 has a fix or sth?

Ty very much

Hello,

it is very simple:
You need to modify the file REDIST/SRC/pgemraux.c and change
void *
mr2d_malloc(n)
unsigned int n;
{

to
void *
mr2d_malloc(n)
unsigned long int n;
{

That should do it. Needless to say this will only work on a 64 bit operating system.

Thomas

Thank you very much for the info,It was really helpfull.I just made the changes and everything seems to work ok.

Thanks a lot