Running Partner Jobs

On This Page

• Am I a Partner Project Member? — reading your sa output, and why the QOS may be named for your department
• Your Project Has Two Halves — the _cpu / _gpu accounts
• Partner Partitions — compute_partners and gpu_partners
• Partner QOS — who the QOS belongs to: project, department, college or institution
• Default QOS in Partner Partitions — why --qos is usually unnecessary
• Automatic Hardware Placement — how jobs are steered onto the hardware you purchased
• Submitting Partner Jobs — example batch scripts
• Watching Your Allocation in Use — how much is free (si --qos) and whose jobs hold it (sq --qos)
• Fair Share for Partners — how contributions raise scheduling priority
• Common Issues — rejections and pending reasons, with fixes
• Related Pages

Am I a Partner Project Member?

Run this command from a login node to see your associations:

sa

You have partner access if a partner partition row carries a QOS whose name begins p_:

   Account     Partition       QOS                          DefQOS
smithlab_cpu                   normal,short,gpu,short_gpu   normal
smithlab_cpu  compute_partners p_smithlab,short             p_smithlab
smithlab_gpu                   normal,short,gpu,short_gpu   normal
smithlab_gpu  gpu_partners     p_smithlab_gpu,short_gpu     p_smithlab_gpu

If no p_ QOS appears on a compute_partners or gpu_partners row, your project does not have partner access.

   Account     Partition       QOS                          DefQOS
smithlab_cpu                   normal,short,gpu,short_gpu   normal
smithlab_cpu  compute_partners p_civeng,short               p_civeng

Both kinds can appear at once. A project that bought hardware of its own and sits under a department that pooled a purchase holds both QOS, with its own as the default:

   Account     Partition       QOS                          DefQOS
smithlab_cpu  compute_partners p_smithlab,p_civeng,short    p_smithlab

Submit with --qos=p_civeng to use the department allocation instead of your own.

Your Project Has Two Halves

Every Slurm account on Hazel exists as both a CPU half (e.g. smithlab_cpu) and a GPU half (smithlab_gpu). You don't normally need to think about this:

• Submit with --account=group (or omit --account entirely — the cluster uses your default account, which is the CPU half) and the scheduler routes the job to the right half based on the partition you chose:
  • -p compute or -p compute_partners → group_cpu
  • -p gpu or -p gpu_partners → group_gpu
• The suffixed name (_cpu or _gpu) shows up in sacctmgr queries and in sacct output so you can see which half a job ran under.
• You can specify the suffixed name explicitly (--account=smithlab_cpu), but if it doesn't match the partition (e.g. --account=smithlab_gpu -p compute) the submit is rejected with an error.

Partner Partitions

Partner projects have access to two extended partitions in addition to the standard partitions:

See Partitions and Resources for the full list of partitions and the Venn diagrams showing how partner partitions overlap with standard ones.

Partner QOS

A partner contribution creates up to two QOS — one per partner partition — based on what was purchased:

Who the QOS belongs to

<name> identifies the partner, which is not always a single project. A contribution is recorded at one of four levels, and the level decides both the QOS name and who may use it:

The three organization levels exist so a department (or college) can pool several groups' purchases into one allocation that all of its projects share, rather than each group holding a slice too small to be useful. From your side nothing changes — the QOS is your default on the partner partition and the automatic hardware placement below works identically. Only the name and the size of the pool differ.

Purchases made at different levels stack. If your department is a partner and your own project also bought hardware, you hold both QOS; your own is the default and the department's is available with an explicit --qos.

Each partner partition also allows a short-job QOS for quick test work without a priority allocation:

• compute_partners allows your partner QOS and short.
• gpu_partners allows your partner GPU QOS and short_gpu.

Each also allows a preemptable QOS for long non-partner work — scavenger on compute_partners, scavenger_gpu on gpu_partners. Both are a limited pilot, granted on request, and both give way to partner jobs.

Default QOS in Partner Partitions

You do not need to specify --qos when submitting to a partner partition. Each partner project member has a partition-specific default:

This means submitting to your partner partition automatically uses your partner allocation. To opt into a short test job on partner hardware, override the default with --qos=short (CPU) or --qos=short_gpu (GPU).

Automatic Hardware Placement

Partner projects contribute specific hardware — particular CPU generations (architectures) and/or GPU models. When you run under your partner QOS, the scheduler can automatically steer your job onto hardware matching what your project purchased, so your priority allocation is spent on nodes matching architecture contributed rather than scattered across the partner partition.

CPU architecture

On compute_partners under your partner QOS (the default there), when you give no --constraint of your own the scheduler adds a --constraint for the most capable architecture your partner allocation has enough cores of to run this job (and accounts for it in billing). If more than one architecture was purchased, a note is printed at submit time naming the choice and the alternatives:

Note: QOS 'p_smithlab' defaulted --constraint to 'genoa' (most capable purchased
that fits this request). Override with --constraint=<arch>; allowed: genoa, skylake.

Job size is part of that choice. Suppose the allocation covers 128 genoa cores and 352 skylake cores. A 64-core job is placed on genoa; a 200-core job does not fit there, so it is placed on skylake instead. You get the best hardware that can actually run the job, rather than the best hardware outright.

If the job is larger than any single purchased architecture, it is rejected at submit time rather than left pending:

Error: QOS 'p_smithlab' cannot run a 400-core job on hardware this partner owns.
Cores available to one job, per architecture: genoa:128, skylake:352. Request fewer
cores, split the work, or submit outside the partner allocation (e.g. --qos=short).

A job cannot span two architectures — a single --constraint names one — so 480 cores split across two generations cannot be gathered into one 400-core job. The refusal is immediate and names the limits, so you can resize rather than wait.

To run on a different architecture your project purchased, set --constraint yourself — the scheduler leaves an explicit constraint untouched:

#SBATCH --partition=compute_partners
#SBATCH --constraint=skylake     # use the project's skylake nodes instead of the genoa default

If only one CPU architecture was purchased, it is pinned automatically and no note is printed — but a job too large for it is still refused as above.

GPU model

On gpu_partners under your partner QOS (the default there), GPU model handling mirrors the CPU case but is enforced (GPU models are tracked only as generic resources, not as node features, so they cannot be expressed with --constraint):

• Request an untyped GPU (--gres=gpu:N) and the scheduler fills in the most capable purchased model with at least N GPUs — e.g. rewriting --gres=gpu:1 to --gres=gpu:h200:1 — and prints a note listing the alternatives. As with CPU architectures, the count matters: an allocation holding one h200 and thirty-two l40s places a 1-GPU job on the h200 and a 4-GPU job on the l40s.
• Request a specific model (--gres=gpu:model:N) and it must be one that was purchased; any other model is rejected at submit time.
• Ask for more GPUs than any single purchased model provides and the request is rejected, listing the per-model limits. One --gres names one model, so such a job cannot run under the partner QOS however it is placed.

To run a non-purchased GPU model on partner GPU nodes, opt out of the partner QOS with --qos=short_gpu.

#SBATCH --partition=gpu_partners
#SBATCH --gres=gpu:h100:2         # h100 must be in your project's purchase, else rejected

Which hardware did my project purchase?

The allowed architectures and GPU models appear in the submit-time notes above, and you can also read them directly from your partner QOS's group resource limits:

sqos                             # every QOS you can use, with its limits
sqos -m p_name -v             # limits + allowed partitions, and who holds the QOS

sacctmgr show qos p_name      format=name,grptres   # native equivalent
sacctmgr show qos p_name_gpu  format=name,grptres

Take p_name from your own sa output — for an organization-level partner it is the department, college or institution name, not your project's. Plain sqos needs no name at all and lists whatever you actually hold, which makes it the safest starting point.

The gres/cpu:<arch>=N and gres/gpu:<model>=N entries in GrpTRES name the architectures and GPU models (with counts) your project contributed. Those names are exactly the values accepted by --constraint (CPU) and --gres=gpu:<model> (GPU). si --qos p_name presents the same limits alongside how much of each is currently in use — see Watching Your Allocation in Use below. See Partitions and Resources for the cluster-wide list of CPU architectures and GPU models.

Submitting Partner Jobs

CPU partner job (uses your partner QOS by default)

#!/bin/bash
#SBATCH --job-name=partner_cpu
#SBATCH --output=cpu.out.%j
#SBATCH --error=cpu.err.%j
#SBATCH --partition=compute_partners
#SBATCH --ntasks=16
#SBATCH --mem=32G
#SBATCH --time=24:00:00

./my_program

GPU partner job (uses your partner GPU QOS by default)

#!/bin/bash
#SBATCH --job-name=partner_gpu
#SBATCH --output=gpu.out.%j
#SBATCH --error=gpu.err.%j
#SBATCH --partition=gpu_partners
#SBATCH --gres=gpu:h100:1
#SBATCH --ntasks=1
#SBATCH --mem=80G
#SBATCH --time=08:00:00

module load cuda
./train.py

Quick test on partner hardware (overrides the default)

#SBATCH --partition=compute_partners
#SBATCH --qos=short
#SBATCH --time=01:00:00

Without an explicit --qos, the partner QOS is used; with --qos=short (or --qos=short_gpu), the job runs under the short-job allocation instead.

Watching Your Allocation in Use

Two commands answer the questions partner members commonly ask: how much of what we bought is free right now? and whose jobs are using it? Both work for every member of the partner project (take p_name from your own sa output).

What resources are free: si --qos

si --qos p_name               # CPU cores, per architecture
si --gpus --qos p_name_gpu    # GPUs, per model

Partition         Architecture    GrpTRES  Used  Avail  Alloc  Total
compute_partners  Genoa               128    64     64    480    832
compute_partners  Skylake             352     0    352   1473   3680
(QOS usage as of 2026-09-29 08:15:05, 3 min ago)

Reading the columns:

• GrpTRES — what your partner allocation may hold on that architecture or GPU model, in total across the whole group.
• Used — how much of that the group's running jobs hold right now, everyone's jobs counted, not just yours.
• Avail — what a job submitted right now could actually get: the remaining allocation, further limited by how much of that hardware is physically free on nodes able to accept work. An architecture can show Avail 0 with allocation left when its nodes are full or offline.
• Alloc / Total — the partition-wide hardware picture (all users, all QOS), for context.

Architectures or models your partner did not purchase are not listed — under your partner QOS a job cannot be placed on them (see Automatic Hardware Placement above).

Whose jobs are using it: sq --qos

Normally each user's squeue shows only their own jobs. Partner QOS are the exception: naming a partner QOS you hold shows every member's jobs under it — the group bought the hardware together, so its members can see how it is being spent:

sq --qos p_name                 # every member's jobs, running and pending
sq --run --qos p_name --tres    # running only, with per-job resources + a TOTAL
sq --pend --qos p_name          # jobs queued behind the allocation

JobID    User      Partition         QOS         ST  Time        Reason
973648   rverma    compute_partners  p_smithlab  R   21:17:04    -
978911   lbiber    compute_partners  p_smithlab  R   6:12:21     -
978979   lbiber    compute_partners  p_smithlab  PD  0:00        QOSGrpCpuLimit
(group view: all jobs under QOS p_smithlab, as of 2026-09-29 08:15:05, 3 min ago)

Running jobs are listed longest-running first; each job's TimeLimit (shown under --run) tells you the latest its resources will free up. The --tres total is the same figure as si --qos's Used column, broken down per job.

The data behind both commands is refreshed about every 10 minutes; the timestamp line under each table says how fresh it is. If you name a partner QOS you do not hold, sq falls back to showing only your own jobs.

Fair Share for Partners

Partner projects accumulate scheduling priority faster than non-partner projects of equal age. The mechanism, in brief:

• Each partner is granted an elevated fair share weighted by the resources it has contributed (CPU cores and GPUs, by model).
• That share rolls up the account hierarchy — project → department → college → institution — so partner activity also raises the priority of the units containing it.
• A contribution made at an organization level starts higher up that chain: a department partner's share is applied to the department, so every project beneath it inherits the benefit rather than one group holding it alone. Project-level and organization-level shares add together where they meet.
• Non-partner accounts retain a baseline share, so partner contributions do not depress non-partner priority below its normal level.

The practical effect: a partner job and a non-partner job submitted at the same time, with similar age, will see the partner job scheduled first — especially on the partner partitions where the partner QOS also adds priority. See Priority and Fair Share for how fair share factors into Slurm's overall multifactor priority.

Common Issues

Related Pages

• Partitions and Resources — full partition list with hardware overlap diagrams.
• Priority and Fair Share — multifactor priority and the Partner Priority section.
• GPU Jobs — GPU type selection and request syntax.
• HPC Partner Program — partnership program overview and how to join.
• Preemptable Jobs — how non-partners run long jobs on idle partner nodes with the scavenger and scavenger_gpu QOS, and why partner work always wins.