example_sparse.c failing to compile

[dariofigueira]

I'm in Windows, VS2017. Project solution generated by CMake with parameters:

• GPU_TARGET=Pascal
  LAPACK_LIBRARIES=mkl_intel_lp64.lib;mkl_intel_thread.lib;mkl_core.lib;libiomp5md.lib (copied from IntelSWTools/parallel_studio_xe_2019.0.045/ compilers_and_libraries_2019/windows/mkl/lib/intel64_win)
  MKLROOT=D:/Program Files (x86)/IntelSWTools/parallel_studio_xe_2019.0.045/compilers_and_libraries_2019/windows/mkl
  USE_FORTRAN=Yes

Compiled and then Statically linked (/MT linker flag)

I compiled and ran testing_dgels in Debug and in Release mode successfully.


Copied example_sparse.c to a new folder, renamed it example_sparse.cpp

example_sparse.cpp

(3.68 KiB) Downloaded 172 times

Created a simple CMakeLists.txt

CMakeLists.txt

(677 Bytes) Downloaded 148 times

Code: Select all

add_executable(magma-test example_sparse.cpp)

find_package( CUDA ) # just to set CUDA_INCLUDE_DIRS

target_include_directories(magma-test PUBLIC D:/Work/Magma/magma-2.4.0/include D:/Work/Magma/magma-2.4.0/sparse/include ${CUDA_INCLUDE_DIRS})
target_link_libraries(magma-test debug D:/Work/Magma/magma-2.4.0/build/lib/Debug/magma.lib debug D:/Work/Magma/magma-2.4.0/build/lib/Debug/magma_sparse.lib)

set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /MT")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")

which upon compilation yields
2>magma_sparse.lib(magma_sparse_generated_djacobisetup.cu.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in magma.lib(interface.obj)

(what?..)

Just grasping at straws I also compiled MAGMA in Release mode, and then tried to compile the example code in Release mode, which yields
1>magma_sparse.lib(magma_dmio.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in magma_sparse.lib(magma_sparse_generated_djacobisetup.cu.obj)


What may be happening here?
What more information do we need to divine this issue?
Shall I recompile MAGMA from scratch?

[mgates3]

This appears to be an issue with compiling with Microsoft Visual Studio. For instance, see this post.
https://stackoverflow.com/questions/766 ... t/12950627
Sorry I can't help any further, as I don't use VS.

BTW, I noticed you listed mkl_intel_ilp64 again, which might give problems, instead of mkl_intel_lp64.

-mark

[dariofigueira]

(copy paste error on the lp64, fixed in the post now)

Let me investigate further then, thanks

EDIT:
- I've "clean" MAGMA project on "Release", which deleted all the *.obj and the .lib from the Release folder
- I've ran again testing_dgels in "Debug" successfully (which does not call magma_sparse.lib)
- But the example_sparse project
1>magma_sparse.lib(magma_sparse_generated_djacobisetup.cu.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in magma.lib(interface.obj)
1>magma_sparse.lib(magma_sparse_generated_djacobisetup.cu.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MTd_StaticDebug' in magma.lib(interface.obj)

So, now we're sure he's not getting the magma_sparse.lib from the Release folder.
This suggests there is something wrong with the magma_sparse.lib in the Debug folder.

EDIT2:
- I've searched the hard-drive and the magma.lib and magma_sparse.lib in the Debug folder are the only exemplars in the whole hard-drive
- I've renamed magma.lib and magma_sparse.lib in the Debug folder to make sure they are the ones being used and , indeed it says "1>LINK : fatal error LNK1104: cannot open file 'D:\Work\Magma\magma-2.4.0\build\lib\Debug\magma_sparse.lib'

[dariofigueira]

the magma compilation command line parameters are:
/MP /GS /TP /W4 /Zc:wchar_t /I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\include" /I"D:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.0.045\compilers_and_libraries_2019\windows\mkl\include" /I"D:\Work\Magma\magma-2.4.0\include" /I"D:\Work\Magma\magma-2.4.0\control" /I"D:\Work\Magma\magma-2.4.0\magmablas" /I"D:\Work\Magma\magma-2.4.0\sparse\include" /I"D:\Work\Magma\magma-2.4.0\sparse\control" /I"D:\Work\Magma\magma-2.4.0\testing" /Zi /Gm- /Od /Ob0 /Fd"magma.dir\Debug\magma.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "MAGMA_NOAFFINITY" /D "HAVE_CUBLAS" /D "MIN_CUDA_ARCH=600" /D "MAGMA_WITH_MKL" /D "CMAKE_INTDIR=\"Debug\"" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /GR /Gd /MTd /openmp /Fa"Debug/" /EHsc /nologo /Fo"magma.dir\Debug\" /Fp"magma.dir\Debug\magma.pch" /diagnostics:classic

the magma_sparse compilation command line parameters are:
/MP /GS /TP /W4 /Zc:wchar_t /I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\include" /I"D:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.0.045\compilers_and_libraries_2019\windows\mkl\include" /I"D:\Work\Magma\magma-2.4.0\include" /I"D:\Work\Magma\magma-2.4.0\control" /I"D:\Work\Magma\magma-2.4.0\magmablas" /I"D:\Work\Magma\magma-2.4.0\sparse\include" /I"D:\Work\Magma\magma-2.4.0\sparse\control" /I"D:\Work\Magma\magma-2.4.0\testing" /Zi /Gm- /Od /Ob0 /Fd"magma_sparse.dir\Debug\magma_sparse.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "MAGMA_NOAFFINITY" /D "HAVE_CUBLAS" /D "MIN_CUDA_ARCH=600" /D "MAGMA_WITH_MKL" /D "CMAKE_INTDIR=\"Debug\"" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /GR /Gd /MTd /openmp /Fa"Debug/" /EHsc /nologo /Fo"magma_sparse.dir\Debug\" /Fp"magma_sparse.dir\Debug\magma_sparse.pch" /diagnostics:classic



the magma librarian command line parameters are:
/OUT:"D:\Work\Magma\magma-2.4.0\build\lib\Debug\magma.lib" /MACHINE:X64 /NOLOGO
the magma_sparse librarian command line parameters are:
/OUT:"D:\Work\Magma\magma-2.4.0\build\lib\Debug\magma_sparse.lib" /MACHINE:X64 /NOLOGO

Do you spot anything off?

[dariofigueira]

Ok, I just determined this is reproducible in my machine:
- I just deleted MAGMA folder completely
- and did a complete rebuild, *just* in Debug mode, as described in https://stackoverflow.com/questions/531 ... 2017-magma
- and the same behavior was observed

I'm now at a point of attempting to reproduce this in a separate computer.

Would any kind soul attempt to reproduce this in your computer? To further this investigation

[dariofigueira]

Just attempted to compile in Release and

Code: Select all

1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol MKL_Get_Version referenced in function magma_print_environment
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol MKL_Get_Max_Threads referenced in function magma_print_environment
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaGetLastError referenced in function magma_is_devptr
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaGetDeviceCount referenced in function magma_getdevices
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaGetDeviceProperties referenced in function magma_init
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaSetDevice referenced in function magma_mem_size
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaGetDevice referenced in function magma_getdevice
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaStreamCreate referenced in function magma_queue_create_internal
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaStreamDestroy referenced in function magma_finalize
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaStreamWaitEvent referenced in function magma_queue_wait_event
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaStreamSynchronize referenced in function magma_queue_sync_internal
1>magma.lib(copy_v2.obj) : error LNK2001: unresolved external symbol cudaStreamSynchronize
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaEventCreate referenced in function magma_event_create
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaEventRecord referenced in function magma_event_record
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaEventSynchronize referenced in function magma_event_sync
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaEventDestroy referenced in function magma_event_destroy
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaMemGetInfo referenced in function magma_mem_size
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaPointerGetAttributes referenced in function magma_is_devptr
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaDriverGetVersion referenced in function magma_print_environment
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cudaRuntimeGetVersion referenced in function magma_print_environment
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cublasCreate_v2 referenced in function magma_queue_create_from_cuda_internal
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cublasDestroy_v2 referenced in function magma_finalize
1>magma.lib(interface.obj) : error LNK2019: unresolved external symbol cublasSetStream_v2 referenced in function magma_queue_create_from_cuda_internal
1>magma_sparse.lib(magma_dcuspmm.obj) : error LNK2001: unresolved external symbol cusparseCreate
1>magma_sparse.lib(dcustomilu.obj) : error LNK2001: unresolved external symbol cusparseCreate
1>magma_sparse.lib(dcustomic.obj) : error LNK2001: unresolved external symbol cusparseCreate
1>magma_sparse.lib(magma_d_blaswrapper.obj) : error LNK2001: unresolved external symbol cusparseCreate
1>magma_sparse.lib(magma_sparse_generated_dmergecg.cu.obj) : error LNK2001: unresolved external symbol cusparseCreate
...

Mark, does the above even compile in linux?
example_sparse.cpp
CMakeLists.txt

[mgates3]

Yes, it compiles on Unix. (I'm actually using macOS.)

Code: Select all

g++ -Wall -fopenmp \
-I ${MAGMADIR}/include \
-I ${MAGMADIR}/sparse/include \
-o example_sparse example_sparse.cpp \
-L ${MAGMADIR}/lib -lmagma -lmagma_sparse \
-L ${CUDADIR} -lcublas -lcudart -lcusparse \
-L /opt/OpenBLAS/lib -lopenblas

From the missing symbols, it looks like you didn't add the CUDA runtime (cudart), cuBLAS, cuSparse, and MKL libraries.
-mark

[dariofigueira]

The example code compiler arguments line generated is

Code: Select all

/GS 
/TP 
/W3 
/Zc:wchar_t 
/I"D:\Work\Magma\magma-2.4.0\include" 
/I"D:\Work\Magma\magma-2.4.0\sparse\include" 
/I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\include" 
/Zi 
/Gm- 
/Od 
/Ob0 
/Fd"magma-test.dir\Debug\vc141.pdb" 
/Zc:inline 
/fp:precise 
/D "WIN32" 
/D "_WINDOWS" 
/D "CMAKE_INTDIR=\"Debug\"" 
/D "_MBCS" /errorReport:prompt 
/WX- 
/Zc:forScope 
/RTC1 
/GR 
/Gd 
/MTd 
/Fa"Debug/" 
/EHsc 
/nologo 
/Fo"magma-test.dir\Debug\" 
/Fp"magma-test.dir\Debug\magma-test.pch" 
/diagnostics:classic 

The example code linker arguments line generated is

Code: Select all

/OUT:"D:\Work\bob-domino-master\magma-test-clean\build\Debug\magma-test.exe" 
/MANIFEST 
/NXCOMPAT 
/PDB:"D:/Work/bob-domino-master/magma-test-clean/build/Debug/magma-test.pdb" 
/DYNAMICBASE "D:\Work\Magma\magma-2.4.0\build\lib\Debug\magma.lib" "D:\Work\Magma\magma-2.4.0\build\lib\Debug\magma_sparse.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib" 
/IMPLIB:"D:/Work/bob-domino-master/magma-test-clean/build/Debug/magma-test.lib" 
/DEBUG 
/MACHINE:X64 
/INCREMENTAL 
/PGD:"D:\Work\bob-domino-master\magma-test-clean\build\Debug\magma-test.pgd" 
/SUBSYSTEM:CONSOLE 
/MANIFESTUAC:"level='asInvoker' uiAccess='false'" 
/ManifestFile:"magma-test.dir\Debug\magma-test.exe.intermediate.manifest" 
/ERRORREPORT:PROMPT 
/NOLOGO 
/TLBID:1 

Would I need to link CUDA and MKL when I linked magma with the Static flag?

Mark, you never used CMake? I think newcomers would benefit from a working example CMakeLists.txt, are you aware of anyone that uses CMake that we could ask?

[dariofigueira]

Finally compiling and running.

Here is the CMakeLists.txt that did it

Code: Select all

add_executable(magma-test example_sparse.cpp)

find_package( CUDA ) 
set( MKLROOT "D:/Program Files (x86)/IntelSWTools/parallel_studio_xe_2019.0.045/compilers_and_libraries_2019/windows/mkl" )
set( LAPACK_LIBRARIES 
	"D:/Program Files (x86)/IntelSWTools/parallel_studio_xe_2019.0.045/compilers_and_libraries_2019/windows/mkl/lib/intel64_win/mkl_intel_lp64.lib"
	"D:/Program Files (x86)/IntelSWTools/parallel_studio_xe_2019.0.045/compilers_and_libraries_2019/windows/mkl/lib/intel64_win/mkl_intel_thread.lib"
	"D:/Program Files (x86)/IntelSWTools/parallel_studio_xe_2019.0.045/compilers_and_libraries_2019/windows/mkl/lib/intel64_win/mkl_core.lib"
	"D:/Program Files (x86)/IntelSWTools/compilers_and_libraries_2019.0.117/windows/compiler/lib/intel64_win/libiomp5md.lib")

target_include_directories(magma-test PUBLIC 
	"D:/Work/Magma/magma-2.4.0/include" 
	"D:/Work/Magma/magma-2.4.0/sparse/include" 
	${CUDA_INCLUDE_DIRS}
	${MKLROOT}/include)
target_link_libraries(magma-test 
	${CUDA_CUDART_LIBRARY}
	${CUDA_CUBLAS_LIBRARIES}
	${CUDA_cusparse_LIBRARY}
	${LAPACK_LIBRARIES}
	debug D:/Work/Magma/magma-2.4.0/build/lib/Debug/magma.lib 
	debug D:/Work/Magma/magma-2.4.0/build/lib/Debug/magma_sparse.lib
	optimized D:/Work/Magma/magma-2.4.0/build/lib/Release/magma.lib 
	optimized D:/Work/Magma/magma-2.4.0/build/lib/Release/magma_sparse.lib)

set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /MT")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")

EDIT: Wait, no. It's compiling and running in Release, but not in Debug..

[dariofigueira]

The above compiles and runs only in Release mode..

Anyone can think of a reason for that? (which then I'll try to verify)

EDIT: Dear Mike, or Hartwig,
are you able to compile

example_sparse.cpp

(3.68 KiB) Downloaded 170 times

with debug symbols? (-g flag in linux correct?)
and statically? (-static flag in linux I believe)

(https://stackoverflow.com/questions/369 ... ary-exists instructs on how to choose only a selection of libraries to be statically linked while others dinamic)