CUDA

CUDA is a parallel computing architecture from NVIDIA. It is used to program instructions for GPUs.



External Links:
CUDA toolkit and driver compatibility table
CUDA toolkit website
NVIDIA Easy Introduction to CUDA

How to run on GPUs in the gpu queues

All GPU nodes are now running Red Hat Enterprise Linux 9. Following is an example job script to request use of four A100 GPUs:

#!/bin/bash
#BSUB -n 1
#BSUB -W 30
#BSUB -q gpu
#BSUB -R "select[a100]"
#BSUB -gpu "num=4:mode=shared:mps=yes"
#BSUB -o out.%J
#BSUB -e err.%J
nvidia-smi

Quick test of GPU availability

lsload -gpuload

Loading CUDA

There are various versions of CUDA on Hazel. CUDA 12.6 is the current default and recommended version. To see the various versions available, type module avail cuda and ls /usr/local/apps/cuda/*. To set the environment, either source the appropriate script or load the default module

module load cuda

Loading the module cuda will put the CUDA compiler nvcc in the path, as well as setting the path to the CUDA libraries.

Exclusive use of the GPUs

• Exclusive per-GPU

  -gpu "mps=yes:mode=exclusive_process"

  This locks GPUs one at a time, not the whole node: other jobs (yours or others') can still use the node's other free GPUs. num=4 locks all four individually, which is why that setting alone blocks other GPU jobs on the node.
• Shared

  -gpu "mps=yes:mode=shared"

  mps only matters here: mode=shared is the only case where multiple processes actually land on the same GPU at once, and mps determines whether that sharing is scheduled efficiently.

  Aside: LSF treats mps=yes:mode=exclusive_process and mps=no:mode=exclusive_process identically — both are fully exclusive, since our LSF doesn't let a user's jobs share an MPS server even though the underlying NVIDIA MPS architecture would otherwise permit it. In short, mps is a no-op outside mode=shared.

• Whole node

  Note that users would rarely need to do this, and should not use this capability without serious consideration. Use

  #BSUB -x

  if the queue allows this. If not, for a 4 GPU node, use

  -gpu "num=4:mps=yes:mode=exclusive_process"

  or

  -gpu "num=4:mps=no:mode=exclusive_process"

How to compile with the correct CUDA version on Hazel

What follows is two approaches to compiling and running code on the GPUs. They are:
[1] (a) to first install/compile the application according to the application's documentation, then (b) reserve suitable resources to run them.
[2] to compile/install your code to target certain GPU hardware on Hazel.

Method [1]
Most users will use this method. The application's documentation will specify which version of cuda, and which compute capability (cc) the code should be compiled with.

module avail cuda

shows all of the cuda toolkit packages available. These should cover any application. So for example, if the application requires cuda toolkit 10.1, then

module load cuda/10.1

will prepare the environment variables so that when you compile your code, the appropriate nvcc, cuda libraries and cuda include files can be found.
Next, running your code: Having compiled with a certain toolkit, and a certain cc, then find the range of the drivers and hardware that will support that toolkit and cc, by looking at these two tables:
CUDA toolkit - driver compatibility table and cc - driver compatibility table
For example, CUDA 10.1 requires a driver >=418.39 (as seen from the 1st linked table above). In the table below, you will see that the rtx2080 GPU node is able to support this application (because the driver installed is 418.74). Next, check the cc: Suppose you compiled your code with cc 7.5. The table below shows that the rtx2080 node can support this cc. Therefore, to run this code, you have to target this node with a batch script like:

#!/bin/bash
#BSUB -n 1
#BSUB -W 30
#BSUB -q gpu
#BSUB -R "select[rtx2080]"
#BSUB -gpu "num=1:mode=shared:mps=yes"
#BSUB -o out.%J
#BSUB -e err.%J
module load PrgEnv-pgi
module load cuda/10.1
./nnetworks.exe

Note: CUDA 12.6 is the current default toolkit and is recommended for new work. CUDA 10.1 is used above only as an illustrative example of the compatibility-lookup process, not as a version recommendation.


Method [2]
Some users may want to target certain GPUs. For example, suppose a user wants to take advantage of the older GPUs. First, look at the table in (3) below to see the cc and drivers for these nodes - they are cc = 6.0 (deprecated but still supported as of CUDA 12.6; support for cc 6.0/Pascal is removed starting CUDA 13.0), and the driver is 525.60.13. Then look here, CUDA toolkit - driver compatibility table, to see what cuda toolkit should be used. This shows that CUDA 12.x will work. So when preparing the environment variable for compiling, use:

module load cuda/12.6

since that is available on our system. Also make sure that the code is compiled with cc = 6.0.
After compilation, run the code to target the intended resources with a batch script that might look like:

#!/bin/bash
#BSUB -n 1
#BSUB -W 30
#BSUB -q gpu
#BSUB -R "select[gtx1080]"
#BSUB -gpu "num=1:mode=shared:mps=yes"
#BSUB -o out.%J
#BSUB -e err.%J
module load PrgEnv-pgi
module load cuda/12.6
./nnetworks.exe




List of GPU nodes, their compute capability(cc) and GPU drivers

This information can be obtained with

lshosts -gpu

Resource type   Description       cc   Driver
          (NVIDIA)
a100   Node with A 100 GPUs   8.0 535.86.10
a30   Node with A 30 GPUs   8.0 535.86.10
a10   Node with A 10 GPUs   8.0 535.86.10
rtx2080   Node with RTX 2080 GPUs   7.5 525.60.13
gtx1080   Node with GTX 1080 GPUs   6.1 525.60.13
p100    Node with P100 GPUs    6.0 525.60.13


Example codes

Use of CUDA on the GPUs is demonstrated with the following example code that adds two vectors.

CUDA C/C++ Example:
ReadMe
C/C++ Makefile
vectorAdd.cu

CUDA for Fortran Example:
ReadMe
Fortran Makefile
Fortran file
Cuda File

Last modified: August 14 2026 16:19:20.