#2 Compiler/MakeFile Error with CUDA

[cement_head]

Getting this error when trying to compile GRASShopPER:

nws_algorithm_option.o: In function `align::NWSAlgorithmOption::deviceQuery()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:389: undefined reference to `cudaGetDeviceCount'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:431: undefined reference to `cudaSetDevice'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:434: undefined reference to `cudaMalloc'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:435: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:438: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:439: undefined reference to `cudaFree'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_algorithm_option.cpp:441: undefined reference to `cudaGetDeviceProperties'
nws_single_gpu_runnable_init.o: In function `align::NWSSingleGPURunnableInit::run()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:12: undefined reference to `cudaSetDevice'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:19: undefined reference to `cudaMalloc'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:20: undefined reference to `cudaMalloc'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:21: undefined reference to `cudaMalloc'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:22: undefined reference to `cudaMalloc'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:23: undefined reference to `cudaMalloc'
nws_single_gpu_runnable_init.o:/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:24: more undefined references to `cudaMalloc' follow
nws_single_gpu_runnable_init.o: In function `checkError(char const*)':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:71: undefined reference to `cudaGetLastError'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:77: undefined reference to `cudaDeviceSynchronize'
nws_single_gpu_runnable_init.o: In function `align::NWSSingleGPURunnableInit::run()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:31: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:32: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:33: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_init.cpp:34: undefined reference to `cudaMemcpy'
nws_single_gpu_runnable_init.o: In function `checkError(char const*)':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:71: undefined reference to `cudaGetLastError'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:77: undefined reference to `cudaDeviceSynchronize'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:80: undefined reference to `cudaGetErrorString'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:74: undefined reference to `cudaGetErrorString'
nws_single_gpu_runnable.o: In function `align::NWSSingleGPURunnable::run()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable.cpp:12: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable.cpp:13: undefined reference to `cudaMemcpy'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable.cpp:14: undefined reference to `cudaMemset'
nws_single_gpu_runnable.o: In function `checkError(char const*)':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:71: undefined reference to `cudaGetLastError'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:77: undefined reference to `cudaDeviceSynchronize'
nws_single_gpu_runnable.o: In function `align::NWSSingleGPURunnable::run()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable.cpp:20: undefined reference to `cudaMemcpy'
nws_single_gpu_runnable.o: In function `checkError(char const*)':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/cuda_declarations.h:80: undefined reference to `cudaGetErrorString'
nws_single_gpu_runnable.o: In function `align::NWSSingleGPURunnable::run()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable.cpp:21: undefined reference to `cudaDeviceSynchronize'
nws_single_gpu_runnable_finalization.o: In function `align::NWSSingleGPURunnableFinalization::run()':
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_finalization.cpp:13: undefined reference to `cudaFree'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_finalization.cpp:14: undefined reference to `cudaFree'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_finalization.cpp:15: undefined reference to `cudaFree'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_finalization.cpp:16: undefined reference to `cudaFree'
/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_finalization.cpp:17: undefined reference to `cudaFree'
nws_single_gpu_runnable_finalization.o:/home/cbfgws3/Programs/GRASShopPER/grasshopper-build/nws_single_gpu_runnable_finalization.cpp:18: more undefined references to `cudaFree' follow
main_cu.o: In function `__cudaUnregisterBinaryUtil()':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x417): undefined reference to `__cudaUnregisterFatBinary'
main_cu.o: In function `__nv_init_managed_rt_with_module(void**)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x432): undefined reference to `__cudaInitModule'
main_cu.o: In function `__device_stub__Z36NeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi16EEvN5align11DevPointersENS0_9DevParamsE(align::DevPointers&, align::DevParams&)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x4e4): undefined reference to `__cudaPopCallConfiguration'
main_cu.o: In function `__device_stub__Z36NeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi1EEvN5align11DevPointersENS0_9DevParamsE(align::DevPointers&, align::DevParams&)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x661): undefined reference to `__cudaPopCallConfiguration'
main_cu.o: In function `__device_stub__Z36NeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi2EEvN5align11DevPointersENS0_9DevParamsE(align::DevPointers&, align::DevParams&)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x7de): undefined reference to `__cudaPopCallConfiguration'
main_cu.o: In function `__device_stub__Z36NeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi3EEvN5align11DevPointersENS0_9DevParamsE(align::DevPointers&, align::DevParams&)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x95b): undefined reference to `__cudaPopCallConfiguration'
main_cu.o: In function `__device_stub__Z36NeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi4EEvN5align11DevPointersENS0_9DevParamsE(align::DevPointers&, align::DevParams&)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0xad8): undefined reference to `__cudaPopCallConfiguration'
main_cu.o:tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0xc55): more undefined references to `__cudaPopCallConfiguration' follow
main_cu.o: In function `__nv_cudaEntityRegisterCallback(void**)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x2e38): undefined reference to `__cudaRegisterFunction'
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x2e71): undefined reference to `__cudaRegisterFunction'
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x2eaa): undefined reference to `__cudaRegisterFunction'
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x2ee3): undefined reference to `__cudaRegisterFunction'
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x2f1c): undefined reference to `__cudaRegisterFunction'
main_cu.o:tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x2f55): more undefined references to `__cudaRegisterFunction' follow
main_cu.o: In function `__sti____cudaRegisterAll()':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x3456): undefined reference to `__cudaRegisterFatBinary'
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x3487): undefined reference to `__cudaRegisterFatBinaryEnd'
main_cu.o: In function `cudaError cudaLaunchKernel<char>(char const*, dim3, dim3, void**, unsigned long, CUstream_st*)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text+0x3500): undefined reference to `cudaLaunchKernel'
main_cu.o: In function `void RunNeedlemanWunschSemiglobalScoreKernel<16, 3, 2, 4, 16>(int, int, align::DevPointers, align::DevParams, align::NWSAlgorithmOption*)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text._Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi16EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE[_Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi16EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE]+0x8c): undefined reference to `__cudaPushCallConfiguration'
main_cu.o: In function `void RunNeedlemanWunschSemiglobalScoreKernel<16, 3, 2, 4, 1>(int, int, align::DevPointers, align::DevParams, align::NWSAlgorithmOption*)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text._Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi1EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE[_Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi1EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE]+0x8c): undefined reference to `__cudaPushCallConfiguration'
main_cu.o: In function `void RunNeedlemanWunschSemiglobalScoreKernel<16, 3, 2, 4, 2>(int, int, align::DevPointers, align::DevParams, align::NWSAlgorithmOption*)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text._Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi2EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE[_Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi2EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE]+0x8c): undefined reference to `__cudaPushCallConfiguration'
main_cu.o: In function `void RunNeedlemanWunschSemiglobalScoreKernel<16, 3, 2, 4, 3>(int, int, align::DevPointers, align::DevParams, align::NWSAlgorithmOption*)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text._Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi3EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE[_Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi3EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE]+0x8c): undefined reference to `__cudaPushCallConfiguration'
main_cu.o: In function `void RunNeedlemanWunschSemiglobalScoreKernel<16, 3, 2, 4, 4>(int, int, align::DevPointers, align::DevParams, align::NWSAlgorithmOption*)':
tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text._Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi4EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE[_Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi4EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE]+0x8c): undefined reference to `__cudaPushCallConfiguration'
main_cu.o:tmpxft_00001fa3_00000000-5_main_cu.cudafe1.cpp:(.text._Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi5EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE[_Z39RunNeedlemanWunschSemiglobalScoreKernelILi16ELi3ELi2ELi4ELi5EEviiN5align11DevPointersENS0_9DevParamsEPNS0_18NWSAlgorithmOptionE]+0x8c): more undefined references to `__cudaPushCallConfiguration' follow
collect2: error: ld returned 1 exit status
Makefile:53: recipe for target 'grasshopper-build' failed
make: *** [grasshopper-build] Error 1

[cement_head]

The error is occuring in the MAKEFILE, the order of the linking statments is incorrect. For reference, I'm using Ubuntu 18.04 LTS; gcc 7.5.0; CUDA-10.2 as my system.

This is the INCORRECT order as per the MAKEFILE that the compile/build proccess generates at the end of the object file list:
g++ -lstdc++ -I/usr/local/cuda-10.2/include -L/usr/local/cuda-10.2/lib64 -lcudart -fopenmp -g -O2 -std=c++11

The CORRECT order is below (g++ used in VERBOSE mode) and results in proper linking:

$ g++ -v -o grasshopper-build results_option.o pps_kmer_value_option.o file_option.o pairs_option.o pps_results_file_option.o pps_partial_size_option.o test_substitution_matrix_option.o paired_end_option.o thread_manager_runnable.o nws_single_gpu_runnable.o nws_algorithm_option.o gpus_value_option.o exception.o fasta_option.o mutex.o pps_partial_size_value_option.o perm_value_option.o thread.o pairs.o pps_results_option.o substitution_matrix_file_option.o results.o pps_win_size_option.o fasta_file_option.o gpus_option.o pps_errors_option.o promising_pairs_selection.o pps_win_size_value_option.o pps_errors_value_option.o pps_shortest_lexicographical_index_value_option.o results_file_option.o pps_partial_count_value_option.o pps_score_cutoff_value_option.o root_option.o pps_partial_count_option.o all_with_all_option.o substitution_matrix.o substitution_matrix_option.o pps_score_cutoff_option.o gap_penalty_option.o sequences.o hi_res_timer.o help_option.o thread_manager.o promising_pairs.o frequency_characteristics.o complete_option.o pps_kmer_option.o pairs_per_gpu_value_option.o nws_single_gpu_runnable_init.o algorithm_option.o paired_end_value_option.o pps_shortest_lexicographical_index_option.o gold_option.o nws_single_gpu_runnable_finalization.o gap_penalty_value_option.o option.o option_manager.o pairs_per_gpu_option.o main.o pairs_file_option.o perm_option.o verify_results_option.o sem.o main_cu.o fasta_y.o pairs_y.o sm_y.o fasta_l.o sm_l.o pairs_l.o -lstdc++ -fopenmp -g -O2 -std=c++11 -I/usr/local/cuda-10.2/include -L/usr/local/cuda-10.2/lib64 -lcudart

$ make
make: Nothing to be done for 'all'.

[Wojciech Frohmberg]

Commit 487f910d3d2e70bdf0151878bd0d12435c13a262 brings the hotfix for the issue for cuda toolkit 10.2. Appearently new toolkit introduced the problem with linking cuda kernels with g++. The issue was fixed by relying on nvcc in linkage phase.

[cement_head]

Cool, that's makes sense.