1. OIT HPC
  2. Running Jobs
  3. Running Partner Jobs

Running Partner Jobs

How members of HPC Partner projects access their dedicated cluster resources, and how partner contributions are reflected in scheduling priority.

Am I a Partner Project Member?

Run this command from a login node to see your associations: 

sacctmgr show association user=$USER format=account,partition,qos,defaultqos

You are a partner project member if you see a row with a partner partition and a partner QOS, e.g.: 

   Account     Partition       QOS                          DefQOS
smith_lab_cpu                  normal,short,gpu,short_gpu   normal
smith_lab_cpu  compute_partners p_smith_lab,short            p_smith_lab
smith_lab_gpu                  normal,short,gpu,short_gpu   normal
smith_lab_gpu  gpu_partners    p_smith_lab_gpu,short_gpu    p_smith_lab_gpu

The p_<group> / p_<group>_gpu entries identify your partner project (smith_lab in this example). If no partner QOS appears, your project does not have partner access.

Your Project Has Two Halves

Every Slurm account on Hazel exists as both a CPU half (e.g. smith_lab_cpu) and a GPU half (smith_lab_gpu). You don't normally need to think about this:

  • Submit with --account=group (or omit --account entirely — the cluster uses your default account, which is the CPU half) and the scheduler routes the job to the right half based on the partition you chose:
    • -p compute or -p compute_partners → group_cpu
    • -p gpu or -p gpu_partners → group_gpu
  • The suffixed name (_cpu or _gpu) shows up in sacctmgr queries and in sacct output so you can see which half a job ran under.
  • You can specify the suffixed name explicitly (--account=smith_lab_cpu), but if it doesn't match the partition (e.g. --account=smith_lab_gpu -p compute) the submit is rejected with an error.

Partner Partitions

Partner projects have access to two extended partitions in addition to the standard partitions:

PartitionIncludesWho can submit
compute_partners All standard CPU nodes plus partner-contributed CPU nodes Partner project members
gpu_partners All standard GPU nodes plus partner-contributed GPU nodes Partner project members

See Partitions and Resources for the full list of partitions and the Venn diagrams showing how partner partitions overlap with standard ones.

Partner QOS

When a partner project is set up, two QOS may be created based on the project's contribution:

QOSAllowed in partitionPurpose
p_<group> compute_partners Higher priority on partner-contributed CPU nodes
p_<group>_gpu gpu_partners Higher priority on partner-contributed GPU nodes

Each partner partition also allows a short-job QOS for quick test work without a priority allocation:

  • compute_partners allows p_<group> and short.
  • gpu_partners allows p_<group>_gpu and short_gpu.

Default QOS in Partner Partitions

You do not need to specify --qos when submitting to a partner partition. Each partner project member has a partition-specific default:

Submitting to…Default QOS
-p compute_partnersp_<group>
-p gpu_partnersp_<group>_gpu
-p compute (general)normal — non-partner work is unchanged
-p gpu (general)gpu — non-partner work is unchanged

This means submitting to your partner partition automatically uses your partner allocation. To opt into a short test job on partner hardware, override the default with --qos=short (CPU) or --qos=short_gpu (GPU).

Submitting Partner Jobs

CPU partner job (uses p_<group> by default)

#!/bin/bash
#SBATCH --job-name=partner_cpu
#SBATCH --output=cpu.out.%j
#SBATCH --error=cpu.err.%j
#SBATCH --partition=compute_partners
#SBATCH --ntasks=16
#SBATCH --time=24:00:00

./my_program

GPU partner job (uses p_<group>_gpu by default)

#!/bin/bash
#SBATCH --job-name=partner_gpu
#SBATCH --output=gpu.out.%j
#SBATCH --error=gpu.err.%j
#SBATCH --partition=gpu_partners
#SBATCH --gres=gpu:h100:1
#SBATCH --ntasks=1
#SBATCH --time=08:00:00

module load cuda
./train.py

Quick test on partner hardware (overrides the default)

#SBATCH --partition=compute_partners
#SBATCH --qos=short
#SBATCH --time=01:00:00

Without an explicit --qos, the partner QOS is used; with --qos=short (or --qos=short_gpu), the job runs under the short-job allocation instead.

Fair Share for Partners

Partner projects accumulate scheduling priority faster than non-partner projects of equal age. The mechanism, in brief:

  • Each partner project is granted an elevated fair share weighted by the resources it has contributed (CPU cores and GPUs, by model).
  • That share rolls up the account hierarchy — project → department → college → institution — so partner activity also raises the priority of the units containing it.
  • Non-partner accounts retain a baseline share, so partner contributions do not depress non-partner priority below its normal level.

The practical effect: a partner job and a non-partner job submitted at the same time, with similar age, will see the partner job scheduled first — especially on the partner partitions where the partner QOS also adds priority. See Priority and Fair Share for how fair share factors into Slurm's overall multifactor priority.

Common Issues

ProblemCauseSolution
Job rejected: "Invalid qos specification" Specified a QOS not allowed in the chosen partition On partner partitions, only p_<group> / p_<group>_gpu and short / short_gpu are allowed. Drop --qos to use the default.
Job rejected: "account is the CPU half but this is a GPU job" (or vice versa) Explicit --account=<group>_cpu with a GPU partition, or --account=<group>_gpu with a CPU partition Use --account=<group> (unsuffixed) and let the scheduler pick the right half, or match the suffix to the partition.
Job pending with reason "QOSMaxJobsPerUserLimit" or similar Hit a limit on the partner QOS Wait for running partner jobs to finish, or submit non-priority work to compute / gpu.
Want to run on partner hardware without consuming partner allocation Need a short-job QOS Use --qos=short on compute_partners or --qos=short_gpu on gpu_partners.

Related Pages