"No enought room on startup disk for Application Memory" whe

[FLorenzo]

Dear Sirs,
I am running what I know is a large problem for a scientific application (tochnog) a finite element solver that I help develop and maintain. The application runs out of the OSX Terminal application. Recently, I implemented the use of the LAPACK library built into the Accelerate framework in OSX as the main solver. The other option being a biconjugate solver.

When I use the application tries to solve 1,320,000 simultaneous linear equations, the problem starts when I use the Accelerate Framework (Which Contains LAPACK) as the Virtual Memory size allocated by OSX jumps from 142 G to about 576 G after the library (LAPACK) is called to solve the system.The same behavior does not happens if I use the biconjugate solver that does not calls LAPACK.

The machine is a mac pro desktop with 8 GB of ram, the 2.66 GHz Quad-core Intel and the standard 640 GB hard drive. The system tells me that I have 487 GB available on hard drive.

After a short while, I get the "No enought room on startup disk for Application Memory" error.

This is a screen capture of the application attempting to solve the problem using the LAPACK library inside the Accelerate framework: Here are the numbers as reported by the activity Monitor.

Tochnog Real Memory 6.68 GB
System Memory Free: 33.8 MB, Wired 378.8 MB, Active 5.06 GB, Inactive 2.53 GB, Used 7.96 GB.
VM size 567.52 GB, Page ins 270.8 MB, Page outs 108.2 MB, Swap used 505 MB


This is a screen copy of the same application solving the same problem without using the Accelerate framework.

Tochnog Real Memory 1.96 GB,
System Memory Free: 4.52 MB, Wired 382.1 MB, Active 2.69 GB, Inactive 416.2 GB, Used 3.47 GB.
VM size 148.60 GB, Page ins 288.8 MB, Page outs 108.2 MB, Swap used 2.5 MB


I can not understand the disparity in the behavior for the same case. As I said before, the only difference is the use of Accelerate in the first case. Also, as you can see, I thought that 8 GB of ram memory was a lot.

What will be the amount of memory required by LAPACK to solve such a problem, for a 64-bit system like OSX?
Do you think that this may be an issue with LAPACK or OSX?

Your help will be greatly appreciated

Best regards,

F Lorenzo

[Julien Langou]

Hi Fernando,

I am running what I know is a large problem for a scientific application (tochnog) a finite element solver that I help develop and maintain.

If you are doing finite element, then your matrix is sparse, correct? How many nonzero elements per row do you have from your discretization? What is the bandwidth of the matrix? (i.e. is your matrix banded? And if yes, what is the bandwidth?) How are the matrices stored using the biconjugate solver? Are they stored as sparse matrices? For LAPACK, you store them dense, correct?

When I use the application tries to solve 1,320,000 simultaneous linear equations,

Just out of curiosity, how big are these systems? Is it the same coefficient matrix with 1,320,000 different right-hand sides, or is it 1,320,000 different coefficient matrices?

I have implemented the use of the DGSV solver for tochnog

You mean DGESV. (Or another one? Which LAPACK subroutine do you use?)

What will be the amount of memory required by LAPACK to solve such a problem, for a 64-bit system like OSX?

LAPACK does not perform any memory allocation. (Same for CLAPACK.) It is not clear what the Accelerate framework (vecLib) does. vecLib can pretty much does whatever it wants as far as memory allocation goes. So it might do memory allocation behind your back. I doubt it though. To answer your question, the amount of memory necessary for LAPACK for the function DGESV is nothing else than what you have already provided from the interface. We need the coefficient matrix A, and we write in it the factors and L and U, etc. So no extra storage.

Can you please answer my question, and then I can follow up with you some more?

Cheers, Julien.

[FLorenzo]

Thanks Julien,
Here is my reply:

If you are doing finite element, then your matrix is sparse, correct? How many nonzero elements per row do you have from your discretization? What is the bandwidth of the matrix? (i.e. is your matrix banded? And if yes, what is the bandwidth?) How are the matrices stored using the biconjugate solver? Are they stored as sparse matrices? For LAPACK, you store them dense, correct?

Tochnog converts the stiffness matrix into a banded matrix before solving. Tochnog uses DSBEV and DSBGV for eigenvalue analysis and I implemented the use of DGBSV to use the converted banded matrix. I did not participate in the implementation of the bi-conjugate solver but the reference in the file indicate that the matrix is stored in profile format for the bi-conjugate.

There is an extra option to use the Compressed Column Storage to use Superlu, but the use of Superlu only lasted until the operating System moved to 64-bit. Since then, I get segmentation faults and other issues with Superlu.

As I mentioned before DGBSV gives much better performance than the bi-conjugate solver and that is the reason for my interest in making sure that it works also for large problems. As I said before small and medium size problems do not have issues.

Just out of curiosity, how big are these systems? Is it the same coefficient matrix with 1,320,000 different right-hand sides, or is it 1,320,000 different coefficient matrices?

Is the estimate of the number of unknowns. Each node has 3 degrees of freedom. Only one right hand side.

You mean DGESV. (Or another one? Which LAPACK subroutine do you use?)

No, the system uses DSBEV, DSBGV and DGBSV. I tried implementing DGESV but found that for the same problems, it was slower than the bi-conjugate solver.

LAPACK does not perform any memory allocation. (Same for CLAPACK.) It is not clear what the Accelerate framework (vecLib) does. vecLib can pretty much does whatever it wants as far as memory allocation goes. So it might do memory allocation behind your back. I doubt it though. To answer your question, the amount of memory necessary for LAPACK for the function DGESV is nothing else than what you have already provided from the interface. We need the coefficient matrix A, and we write in it the factors and L and U, etc. So no extra storage

It is very weird. I posted the question to the Apple team, and did not get a straight answer. Then just recently they asks me if the problem persisted after OSX moved to Mavericks, I ran the same problem and still got the same answer, i.e, the message advising that the application ran out of memory.

Interestingly enough, if I build LAPACK and link tochnog with this library I still get the same result: not enough memory. It is odd also that when tochnog is not using the Lapack library, the system reports that tochnog is using 100% of the resources. However when we use LAPACK, the resources used by Tochnog drop to almost zero.

Thank you for all your help.

Fernando Lorenzo