linear solver sgmres very slow

[Noran]

Hi @ all,
I've tested the solver sgmres with two sparse matrices A and B. A is 24 x 24 and B is 24 x 3. My implementation see below. The whole algorithm needs more than 30 seconds for this small matrices. For the matrix with one column it needs about 12 seconds. Can somebody tell me, please, why it's so slow? Can i change something in my implementation to speed it up?

Thank you in advance
Noran

Code: Select all

int
myFunction (	magma_int_t rowsMatrixA,
			magma_int_t colsMatrixA,
			magma_index_t *rowPointerA,
			magma_index_t *colIndA,
			float *valA,
			magma_int_t colsMatrixB,
			float *valB,
			float *valAB)
{
	int error = MAGMA_SUCCESS;
	int nonZeroElementsA = rowPointerA[rowsMatrixA];
	int elementsB = colsMatrixA * colsMatrixB;
	int elementsAB = rowsMatrixA * colsMatrixB;
	magma_s_sparse_matrix d_A;
	magma_index_t *d_rowPointerA;
	magma_index_t *d_colIndA;
	magma_s_vector d_b;
	magma_s_vector d_ab;
	magma_s_solver_par solverPar;

	DEBUG_LOG("Allocating GPU Memory...");
	float *d_valA;
	magma_index_malloc(&d_rowPointerA,rowsMatrixA+1);
	magma_index_malloc(&d_colIndA, nonZeroElementsA);
	magma_smalloc(&d_valA,nonZeroElementsA);
	float *d_valB;
	magma_smalloc(&d_valB, elementsB);
	float *d_valAB;
	magma_smalloc(&d_valAB, elementsAB);

	DEBUG_LOG("Transferring matrices...");
	magma_index_setvector(rowsMatrixA+1, rowPointerA, 1, d_rowPointerA, 1);
	magma_index_setvector(nonZeroElementsA, colIndA, 1, d_colIndA, 1);
	magma_ssetvector(nonZeroElementsA, valA, 1, d_valA, 1);
	magma_ssetmatrix(colsMatrixA, colsMatrixB, valB, colsMatrixA, d_valB, colsMatrixA);
	magma_ssetmatrix(rowsMatrixA, colsMatrixB, valAB, rowsMatrixA, d_valAB, rowsMatrixA);

	DEBUG_LOG("Initializing matrix and vectors");
	d_A.col = d_colIndA;
	d_A.row = d_rowPointerA;
	d_A.val = d_valA;
	d_A.memory_location = Magma_DEV;
	d_A.storage_type = Magma_CSR;
	d_A.nnz = nonZeroElementsA;
	d_A.num_rows = rowsMatrixA;
	d_A.num_cols = colsMatrixA;

	d_b.memory_location = Magma_DEV;
	d_b.nnz = colsMatrixA;
	d_b.num_rows = colsMatrixA;
	d_b.val = d_valB;

	d_ab.memory_location = Magma_DEV;
	d_ab.nnz = rowsMatrixA;
	d_ab.num_rows = rowsMatrixA;
	d_ab.val = d_valAB;

	DEBUG_LOG("Initializing solver_par");
	solverPar.solver = Magma_GMRES;
	solverPar.epsilon = 10e-16;
	solverPar.maxiter = 1000;
	solverPar.restart = 30;
	solverPar.version = 0;
	solverPar.verbose = 0;
	solverPar.num_eigenvalues = 0;
	solverPar.res_vec = (double*) (malloc(sizeof(double)));

	DEBUG_LOG("Running solver...");
	for (int col = 0; col < colsMatrixB; col++)
	{
		printf("col = %i\n", col);
		error = magma_sgmres(d_A, d_b, &d_ab, &solverPar);
		if (error != MAGMA_SUCCESS)
		{
			printf("Errorcode %i\n",error);
			magma_free(d_valA);
			magma_free(d_valB);
			magma_free(d_valAB);
			magma_free(d_colIndA);
			magma_free(d_rowPointerA);
			return error;
		}
		d_b.val = d_b.val + colsMatrixA;
		d_ab.val = d_ab.val + rowsMatrixA;
	}

	magma_sgetmatrix(rowsMatrixA, colsMatrixB, d_valAB, rowsMatrixA, valAB, rowsMatrixA);

	magma_free(d_valA);
	magma_free(d_valB);
	magma_free(d_valAB);
	magma_free(d_colIndA);
	magma_free(d_rowPointerA);

	return error;
}

[Noran]

Ok I've found my failure:

Code: Select all

solverPar.epsilon = 10e-16;

It's a little bit to small. I've used the following:

Code: Select all

solverPar.epsilon = 10e-8;

Greetz Noran

[hartwig anzt]

Noran,

good catch - GMRES may have problems to converge to this accuracy. Also, the sparse-iter package is originally designed for significantly larger systems, typically containing 500K-2M unknowns. You may have a special reason, but systems that small are usually solved with a direct solver - e.g. LU. Finally, using the GPU always causes some overhead (initialization, allocation, memory transfers). This is why one usually sticks to the CPU if solving a small system.

Anyhow, if it works for you now - great!

Cheers, Hartwig

[Noran]

Hi Hartwig,
this System was only for Debugging and functional Test. We have a System: A = 400x400 and B = 400x10 Elements. So I hope that the MAGMA-Library will speed it up a little bit. But thank you for your Answer, hence we will test all with direct LU too. We will see :)

Greetz and thanks Noran