CUDA

CUDA is a parallel computing architecture from NVIDIA. It is used to program instructions for GPUs.



External Links:
CUDA toolkit and driver compatibility table
CUDA toolkit website
NVIDIA Easy Introduction to CUDA

How to run on GPUs in the gpu queues

All GPU nodes are now running Red Hat Enterprise Linux 9. Following is an example job script to request use of an A30 GPU:

#!/bin/bash
#SBATCH --job-name=cuda_test
#SBATCH --partition=gpu_partners
#SBATCH --qos=short_gpu
#SBATCH --time=00:30:00
#SBATCH --ntasks=1
#SBATCH --gres=gpu:a30:1
#SBATCH --output=out.%j
#SBATCH --error=err.%j

nvidia-smi

Note: Some GPU types are not yet available to jobs submitted through Slurm during the ongoing LSF-to-Slurm migration. This example targets an A30 GPU, which is currently available.

Quick test of GPU availability

sinfo -p gpu -o "%N %G %t"

and

sinfo -p gpu_partners -o "%N %G %t"

Loading CUDA

There are various versions of CUDA on Hazel. CUDA 12.6 is the current default and recommended version. To see the various versions available, type module avail cuda and ls /usr/local/apps/cuda/*. To set the environment, either source the appropriate script or load the default module

module load cuda

Loading the module cuda will put the CUDA compiler nvcc in the path, as well as setting the path to the CUDA libraries.
Troubleshooting: Running CUDA binaries on Slurm compute nodes can fail with CUDA driver is a stub library if the cuda module's LD_LIBRARY_PATH includes the stubs directory. This is a known, recurring issue with the cuda/12.6 module, not specific to any one example. Fix: strip stubs from LD_LIBRARY_PATH at runtime before executing the binary:

export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' | grep -v stubs | paste -sd:)

Exclusive use of the GPUs

Unlike LSF, Slurm on Hazel does not currently support GPU sharing between jobs. Every --gres=gpu:<type>:N request gives your job exclusive use of that GPU by default — there's no equivalent to LSF's mode=shared vs mode=exclusive_process flags to choose between, since exclusive access is simply how Slurm allocates GPUs here. Other users' jobs can still use the node's other free GPUs, same as LSF's per-GPU exclusive behavior, just without needing any special flag to get it.

How to compile with the correct CUDA version on Hazel

What follows is two approaches to compiling and running code on the GPUs. They are:
[1] (a) to first install/compile the application according to the application's documentation, then (b) reserve suitable resources to run them.
[2] to compile/install your code to target certain GPU hardware on Hazel.

Method [1]
Most users will use this method. The application's documentation will specify which version of cuda, and which compute capability (cc) the code should be compiled with.

module avail cuda

shows all of the cuda toolkit packages available. These should cover any application. So for example, if the application requires cuda toolkit 10.1, then

module load cuda/10.1

will prepare the environment variables so that when you compile your code, the appropriate nvcc, cuda libraries and cuda include files can be found.
Next, running your code: Having compiled with a certain toolkit, and a certain cc, then find the range of the drivers and hardware that will support that toolkit and cc, by looking at these two tables:
CUDA toolkit - driver compatibility table and cc - driver compatibility table
For example, CUDA 10.1 requires a driver >=418.39 (as seen from the 1st linked table above). In the table below, you will see that the rtx2080 GPU node is able to support this application (because the driver installed is 418.74). Next, check the cc: Suppose you compiled your code with cc 7.5. The table below shows that the rtx2080 node can support this cc. Therefore, to run this code, you have to target this node with a batch script like:

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=00:30:00
#SBATCH --partition=gpu
#SBATCH --gres=gpu:rtx2080:1
#SBATCH --output=out.%j
#SBATCH --error=err.%j
module load PrgEnv-nvidia/26.1-slurm
module load cuda/10.1
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' | grep -v stubs | paste -sd:)
./nnetworks.exe


Method [2]
Some users may want to target certain GPUs. For example, suppose a user wants to take advantage of the older GPUs. First, look at the GPU node reference below to see the cc and drivers for these nodes - they are cc = 6.0, and the driver is 525.60.13. Then look here, CUDA toolkit - driver compatibility table, to see what cuda toolkit should be used. This shows that CUDA 12.x will work. So when preparing the environment variable for compiling, use:

module load cuda/12.6

since that is available on our system. Also make sure that the code is compiled with cc = 6.0.
After compilation, run the code to target the intended resources with a batch script that might look like:

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=00:30:00
#SBATCH --partition=gpu
#SBATCH --gres=gpu:gtx1080:1
#SBATCH --output=out.%j
#SBATCH --error=err.%j
module load PrgEnv-nvidia/26.1-slurm
module load cuda/12.6
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' | grep -v stubs | paste -sd:)
./nnetworks.exe




GPU node types, partitions, and drivers

GPU node types, partitions/QOS, and driver information are cluster-wide Slurm concepts rather than CUDA-specific ones, so they're documented on the general GPU resources page instead of duplicated here: see GPU Jobs with Slurm for the full list of available GPU types, compute capabilities, and drivers.

Example codes

Use of CUDA on the GPUs is demonstrated with the following example code that adds two vectors.

CUDA C/C++ Example:
ReadMe
C/C++ Makefile
vectorAdd.cu

CUDA for Fortran Example:
ReadMe
Fortran Makefile
Fortran file
Cuda File

Last modified: August 14 2026 16:19:20.