Running Jobs with Slurm

Submit batch jobs to run applications on compute nodes

The cluster uses the Slurm workload manager for scheduling jobs on compute nodes. Running applications directly on login nodes is not permitted. Users must submit a batch script or request an interactive session via Slurm.

A batch script is a text file containing resource requirements (cores, time, memory) and the commands to run your application. Slurm uses directives beginning with #SBATCH to specify these requirements.

Quick Links

• Jump to basic Slurm instructions
• Slurm FAQs:
• Choosing partitions, cores, and resources
• Monitoring jobs and cluster status
• Job priority and fairshare
• What the cluster does automatically
• Running partner jobs
• Estimating memory requirements

• Examples and templates
• Generic batch script template
• Parallel and GPU batch scripts
• Software-specific batch scripts

• Available resources
• Partitions and QOS
• Advanced
• Array jobs
• Automating job submissions
• Staging data for a compute job
• Topology and job placement
• Migrating from LSF?

Step 1: Create batch script

Serial job

The following batch script run_mycode.sh runs a serial application mycode.exe:

#!/bin/bash
#SBATCH --job-name=mycode
#SBATCH --output=stdout.%j
#SBATCH --error=stderr.%j
#SBATCH --ntasks=1
#SBATCH --mem=4G
#SBATCH --time=02:00:00

./mycode.exe

The #SBATCH directives specify job parameters:

• --ntasks=1 requests one CPU core
• --mem=4G requests 4 GB of memory per node
• --time=02:00:00 sets a 2-hour time limit
• --job-name sets the job name displayed by squeue
• --output and --error specify where stdout and stderr are written (%j is replaced by the job ID)

Parallel job

→ Read this note before submitting parallel jobs.

For a parallel application using 4 cores on a single node:

#!/bin/bash
#SBATCH --job-name=mycode
#SBATCH --output=stdout.%j
#SBATCH --error=stderr.%j
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=8G
#SBATCH --time=02:00:00

./my_parallel_code.exe

Using --nodes=1 ensures all 4 tasks run on the same node. For MPI applications, use srun to launch:

srun ./my_mpi_code.exe

Advanced options

• Generic batch script template
• Array jobs for parameter sweeps

Step 2: Submit job

Batch job

Submit your batch script using sbatch:

sbatch run_mycode.sh

Slurm returns a job ID that you can use to monitor the job.

Interactive job

For GUI applications requiring display, use an HPC-VCL node.

• Using the HPC-VCL

For testing and debugging, start a short interactive session:

Production jobs should always use batch scripts. For testing, request an interactive session using salloc:

salloc --ntasks=4 --nodes=1 --time=00:30:00

This allocates 4 cores on one node for 30 minutes. Once allocated, you'll have a shell on a compute node.

For a serial job with exclusive node access:

salloc --ntasks=1 --exclusive --time=00:30:00

Interactive sessions should be kept brief. Idle sessions may be terminated per the Acceptable Use Policy.

Step 3: Monitor job

Job status

Use squeue to check your jobs:

$ squeue -u $USER
JOBID  PARTITION  NAME    USER    ST  TIME  NODES  NODELIST
12345  standard   mycode  unityID  R  0:45      1  c001n01

Job states (ST): PD=pending, R=running, CG=completing.

The sq local command gives a focused view of each state, with the columns that actually matter for it — priority and the reason a job is waiting when pending, elapsed time against the wall limit when running:

$ sq --pend
JobID    User     Partition  QOS     Priority  Reason
1842001  unityID  compute    normal  18422     Resources

$ sq --run
JobID    User     Partition  QOS     Elapsed   TimeLimit
1841190  unityID  compute    normal  18:22:47  3-00:00:00

For one job's full resource request and timestamps, use sj:

sj JOBID                # request + state for a job
sj JOBID_7              # an array task element

squeue -j JOBID -l      # native equivalent

→ Monitoring jobs and cluster status

Is a running job making progress?

squeue only shows that a job is running, not whether it is doing any work. Use sjs (local wrapper for the sstat command) to see live CPU, memory, and disk usage:

$ sjs 464645
464645  unityID, RUNNING, elapsed 4:44:21
  alloc: cpu=12,mem=48000M,node=1,billing=23

Step          NTasks  AveCPU    %CPU  MaxRSS  %Mem  MaxDiskRead  MaxDiskWrite
464645.batch  1       04:55:28  9%    9.24G   20%   181.72G      235.09G

Here %CPU of 9% means the job is using about 1 of its 12 requested cores. Add -r to re-sample and watch what changes, which is how you tell a busy job from a stuck one.

→ Is my running job making progress?

Cancel a job

Use scancel with the job ID:

scancel JOBID

To cancel all your jobs:

scancel -u $USER

Partition and node status

View partition information with si (local wrapper for sinfo command):

$ si
Partition         Architecture  Avail  Alloc  Total
compute           Haswell         232    388    620
compute           Broadwell       654    234    936
compute           Cascadelake     468    364    896
gpu               Genoa            72     56    128

$ si --gpus            # report GPUs instead of cores
Partition         GPU       Avail  Alloc  Total
gpu               RTX_2080      3      1      4
gpu               A30           0      8      8
gpu               H100         16     24     40

Idle hardware is not the whole story: a QOS group limit can stop your job even when Avail is non-zero. Add --qos to cap Avail by what a job under that QOS could actually claim right now, and to show the group limit and how much of it is already in use:

$ si --gpus --qos gpu
Partition         GPU       GrpTRES  Used  Avail  Alloc  Total
gpu               H100           40    24     16     24     40

→ Choosing partitions, cores, and resources · The cluster status page provides detailed node availability.

MPI job

#!/bin/bash
#SBATCH --job-name=hydro
#SBATCH --output=hydro.out.%j
#SBATCH --error=hydro.err.%j
#SBATCH --ntasks=32
#SBATCH --mem=16G
#SBATCH --time=02:00:00
#SBATCH --partition=compute

module load PrgEnv-intel
srun ./hydro.exe

Runs an MPI code using 32 tasks for up to 2 hours. The srun command launches the MPI tasks across allocated nodes.

Hybrid MPI+OpenMP job

#!/bin/bash
#SBATCH --job-name=chemtest
#SBATCH --output=chemtest.out.%j
#SBATCH --error=chemtest.err.%j
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=8
#SBATCH --time=02:00:00
#SBATCH --exclusive

module load openmpi-gcc
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./chemtest.exe

Runs a hybrid code with 8 MPI tasks (4 per node) and 8 OpenMP threads per task. Uses --exclusive for dedicated node access. See hybrid jobs guide.

GPU job

#!/bin/bash
#SBATCH --job-name=nnetworks
#SBATCH --output=out.%j
#SBATCH --error=err.%j
#SBATCH --ntasks=1
#SBATCH --mem=32G
#SBATCH --time=00:30:00
#SBATCH --partition=gpu
#SBATCH --gres=gpu:a100:1

module load cuda
./nnetworks.exe

Runs a CUDA application using one A100 GPU. GPU type is required in the --gres directive (e.g., --gres=gpu:a100:1). Available types: a10, a30, a100, gtx1080, h100, h200, l40, l40s, p100, rtx_2080. See the GPU jobs guide and CUDA software page.

Common commands

Common directives

Local commands