MAGMA use for steady CFD simualtions

[dreamer]

I am testing MAGMA sparse to solve linear systems arising from NS equations in a CFD steady simulation. Existing code uses Paralution ( cusparse ILU) and works fine. I am interested on having a faster ILU than what Paralution provides. I adapted the example_sparse.c to read an mtx file for the matrix and a txt file for the rhs. It works fine and I got performance results for ILU with cusparse and ILU with ISAI.

I have though observed that the repeated use of the ILU and the solver increases the memory consumption (attached file: memoryCheck.c).
I an other test I solve the same system twice but this time every run is between a magma_init() and magma_finalize() (attached file memoryCheck2.c).
My Matrix has 18400 nnz and the memory consumption status with nvidia-smi after each step is like that:

• ILU1
  | 0 27593 C ./memoryCheck2 138MiB |
  FGMRES1
  | 0 27593 C ./memoryCheck2 138MiB |
  After magma_finalize()
  | 0 27593 C ./memoryCheck2 132MiB |
  
  ILU2
  | 0 27593 C ./memoryCheck2 148MiB |
  FGMRES2
  | 0 27593 C ./memoryCheck2 148MiB |
  After magma_finalize()
  | 0 27593 C ./memoryCheck2 142MiB |

Am I forgetting to free some of the used structure between the first and the second use?

[hartwig anzt]

Dear "dreamer",

thank you for pointing out this issue. I appreciate attaching code that I can follow what is happening.

The increased memory usage comes from the fact that the preconditioner is not deallocated after being applied. This means, the ILU factorization stays in the GPU memory, and when the preconditioner setup is called the second time, it does not update the existing preconditioner, but allocates a new one.

to avoid this, just call

magma_dprecondfree( &opts.precond_par,queue);

to free the preconditioner, or

magma_dsolverinfo_free( &opts.solver_par, &opts.precond_par,queue);

to free all information in the preconditioner and the solver. I admit that this should happen automatically in case the same preconditioner gets generated, and will fix it in the next release.

I don't have all details about your NS solver, but I could imagine this being a very nice setting for not completely regenerating a preconditioner, but updating an existing factorization via
magma_dpariluupdate( A, &precond, sweeps, queue );
see : http://www.netlib.org/utk/people/JackDo ... LU_GPU.pdf

If you are interested to look into this, please just send me an email.

Also, I see you are using GMRES solver, maybe try IDR with restart 4: My experience is that IDR is often much faster...

Hartwig

[dreamer]

Dear Dr Anzt,
Thanks for your prompt answer. Yes freeing the solver solved the problem. I assumed that magma_finalize() would have done this.
For heavy matrix of few millions of nnz I see that the cost of preparing the preconditioner is prohibitive so for sure I will need the dpariluupdate.

Med