Preemptable Jobs

What This Is For

Partner projects buy dedicated nodes, and some nodes are not busy all the time. Anyone can already use the idle capacity through the short (CPU) and short_gpu (GPU) QOS, which are capped at 2 hours to meet the partner service level agreement to have access to their resources within 2 hours.

The scavenger QOS remove the 2 hour cap with the tradeoff that:

• jobs never outrank anyone: the QOS carries no priority bonus, so on partner hardware partner work always wins.
• when a partner needs their nodes, jobs are stopped and put back in the queue to start again from the beginning.

Which One To Use

There is one for each kind of partner hardware. Everything on this page applies to both — only the QOS name and the partition change:

Choosing Between the Short and Scavenger QOS

Both run on partner hardware — short / scavenger in the compute_partners partition, short_gpu / scavenger_gpu in gpu_partners. They have the following differences:

If your job fits in two hours, use short or short_gpu. You get a guarantee that the scavenger QOS do not offer.

Submitting a Job to a Scavenger QOS

Neither is anyone's default QOS, so you have to request it specifically. On CPU partner nodes:

#!/bin/bash
#SBATCH --job-name=long_run
#SBATCH --partition=compute_partners
#SBATCH --qos=scavenger
#SBATCH --time=3-00:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=16
#SBATCH --mem=32G
#SBATCH --output=long_run_%j.out

./my_program --checkpoint-dir=/share/$GROUP/$USER/ckpt/$SLURM_JOB_ID

And on GPU partner nodes:

#!/bin/bash
#SBATCH --job-name=long_train
#SBATCH --partition=gpu_partners
#SBATCH --qos=scavenger_gpu
#SBATCH --time=3-00:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --gres=gpu:l40s:1
#SBATCH --mem=64G
#SBATCH --output=long_train_%j.out

./my_program --checkpoint-dir=/share/$GROUP/$USER/ckpt/$SLURM_JOB_ID

Submitting to compute_partners or gpu_partners without --qos still gets you short / short_gpu (or your partner QOS, if you are in a partner project), exactly as before. Nothing about your existing jobs changes.

To confirm you have access and see the current limits:

sqos

What Happens When You Are Preempted

1. A partner job needs nodes your job is holding.
2. Slurm signals your job and gives it a grace period of 5 minutes.
3. At the end of the grace period the job is stopped and requeued — it goes back into the pending queue with the same job ID.
4. It starts again from the beginning when resources are next free. Your batch script is re-run from its first line.

Two rules decide which job gets preempted, and both are meant to protect long-running work:

• A job is not preemptable during its first 30 minutes. Every attempt gets at least that much uninterrupted time, so a job cannot be requeued over and over without ever making progress.
• After that, the job that started most recently is preempted first. If several jobs could free the needed nodes, the one that has been running the shortest time loses the least work, so it is the one chosen. A job that has been running for days is the last to be touched.

Writing a Job That Survives Preemption

Checkpoint, and restart from the checkpoint

This is the whole game. Your script is re-run from the top, so it needs to notice that a previous attempt got partway and resume from there. Many scientific codes have this built in (GROMACS -cpi, LAMMPS read_restart, NAMD reinitatomsics, PyTorch checkpoint files); if yours does, use it.

CKPT=/share/$GROUP/$USER/ckpt/$SLURM_JOB_ID
mkdir -p $CKPT

if [ -f $CKPT/state.rst ]; then
    ./my_program --restart-from $CKPT/state.rst
else
    ./my_program --checkpoint-to $CKPT/state.rst
fi

The job ID stays the same across a requeue, so naming the checkpoint directory after $SLURM_JOB_ID gives each job a stable place to resume from.

Write output so a rerun is safe

A restarted job re-runs everything, including any output your first attempt already wrote. Append-mode logs will contain the partial first attempt; files written with > will be overwritten, which is usually what you want. Be careful with anything that has a side effect outside the job — submitting follow-on work, sending mail, updating a database — because it will happen again.

Catch the grace-period signal (optional)

If your program can save state on demand, trap the signal Slurm sends at the start of the grace period and use the 5 minutes:

cleanup() {
    echo "Preempted at $(date) -- saving state"
    kill -USR1 $PID       # whatever your program uses to checkpoint on demand
    wait $PID
}
trap cleanup SIGTERM

./my_program &
PID=$!
wait $PID

Do not use --no-requeue

Batch jobs only

The scavenger QOS are for sbatch. An interactive session (salloc, or srun run directly from a login node) cannot meaningfully be requeued — your shell would simply vanish. For interactive work on partner hardware use --qos=short or --qos=short_gpu, which are not preemptable. See Interactive jobs.

Checking Whether Your Job Was Preempted

A requeued job keeps its job ID, so its history has more than one entry. Ask sacct for all of them:

sacct -j jobid --duplicates --format=JobID,State,Start,End,Elapsed

An attempt that was preempted shows a state of PREEMPTED or REQUEUED, followed by a later attempt that ran again. While the job is waiting to restart, squeue shows it pending with a reason of JobHeldUser or Resources.

The job monitoring guide covers reading these in more detail.

Common Questions

Related Pages

• Running Partner Jobs — how partner projects reach their own hardware.
• Partitions and Resources — the full partition list.
• GPU Jobs — requesting GPUs, models and CUDA.
• Priority and Fair Share — where the scavenger QOS sit against the others.
• Monitoring Jobs — reading sacct and squeue output.