Huge matrix

[axelyamel]

Hi. I'm having problem calculating huge matrixes (by huge I means at least 100000 x 100 matrix). I'm having a segmentation fault. Also i set mi LWORK = 3 * m * m + 4 *n *n + 4 *n

Thank you very much

[axelyamel]

I'm doing dgesdd

#include "stdio.h"
#include "stdlib.h"
#include "time.h"
#include "math.h"

extern void dgesdd_ ( char *JOBZ, int *M, int *N, double *A, int *LDA,double *S, double *U, int *LDU, double
*VT, int *LDVT, double *WORK,int *LWORK, int *IWORK, int *INFO );

int main()
{

time_t start, finish;
start = time(NULL);
double ttime;
char JOBU;
int k,p,h,i,j,l,v,b;
int M;
int N;
FILE *fp;

fp = fopen("vector1.txt","r");

fscanf(fp,"%d",&M);
fscanf(fp,"%d",&N);

int LWORK;

LWORK = 3 * pow(M,4) + 4 * pow(N,4);

printf("%lld",LWORK);

int INFO;
double a[M*N];
double s[N];
double work[LWORK];
int iwork[M*N];
double uu[M*N];
double vt[N*N];


for ( k = 0; k < M*N; k++){

fscanf(fp,"%lf",&a[k]);

}


/* for(h = 0; h < M; h++){
for(b = 0; b<N; b++){
printf("a[%d][%d] = %lf\t",h,b,a[h + (b*M)]);
}
printf("\n");
}*/

printf("\n");
fclose(fp);


JOBU = 'A';
dgesdd_( &JOBU, &M, &N, a, &M, s, uu, &M, vt, &N, work, &LWORK, iwork, &INFO);



/* for(i = 0; i<M; i++){
for(j = 0;j<N; j++){
printf("uu[%d][%d] = %lf\t",i,j,uu[i + (j*M)]);
}
printf("\n");
}*/

printf("\n");

/* for(l = 0; l<N; l++){
printf("s[%d] = %lf\n",l,s[l]);
}*/

printf("\n");

/* for(v = 0; v<N; v++){
for(p = 0; p<N; p++){
printf("vt[%d][%d] = %lf\t",v,p,vt[v + (p*N)]);
}
printf("\n");
}*/



printf("\n INFO=%d\n", INFO );

finish = time(NULL);

ttime = difftime(finish, start);

printf("\nTotal time running: %lf sec.\n\n",ttime);

return 0;
}


this is the code, it worked with 10,000 x 100 matrix

[axelyamel]

You can't declare large arrays like this.

double* a = new double[M*N];
double* s = new double[N];
double* work = new double[LWORK];
int* iwork = new int[M*N];
double* uu = new double[M*N];
double* vt = new double[N*N];

The memory allocation is handled, when a correct allocation is commanded.

So what you recomend for make svd to huge matrixes?

[axelyamel]

The rest of the code besides the size of WORK remains the same. But tell us how much ram you have and if you are running a 32bit or 64bit OS.

18000x100 consume my RAM limit of 4GB on 64bit linux with some more programs running. If I take floats in stead of doubles and sgesdd instead of dgesdd then obviously you can run up to 40000x100

I'm using 2gb with 32bit linux

[axelyamel]

The rest of the code besides the size of WORK remains the same. But tell us how much ram you have and if you are running a 32bit or 64bit OS.

18000x100 consume my RAM limit of 4GB on 64bit linux with some more programs running. If I take floats in stead of doubles and sgesdd instead of dgesdd then obviously you can run up to 40000x100

I'm using 2gb with 32bit linux

Will never be enough.

If you choose JOBU = 'A' M=100000 N=100
A=O(MxN) U=O(MxM) V=O(NxN)
Let's do some math for doubles.

size(A) = 76 MB ok.
size(V) = 70 KB very ok.

but

size(U) = 75GB NOT ok!!!!

This matrix would even bust our research group's cluster with scalapack.

woww!! thanks

[gortiz]

Dear Forum,

when running scalapack application
I have a lot more RAM memory demands
on node where I lunched such apps
that I could expect from the size on each
node required for distributed matrix.

Any idea the cause that it could be?

Thank a lot

g.

[gortiz]

Hello,

I find it from myself

Problem was with no deallocation
of memory assigned to some array.

Regards,

g.