Job Priority and Fairshare

How Job Priority Works

When multiple jobs are waiting for resources, Slurm uses a priority score to decide which job runs first. Higher priority jobs are scheduled before lower priority jobs.

The priority score is calculated from several factors. Each factor is normalised to a value between 0 and 1 and multiplied by a configured weight; the weights on Hazel, largest first:

The QOS factor carries the largest weight, but nearly every general-access QOS (normal, long, short, short_gpu, scavenger, scavenger_gpu) has a priority value of 0, so between non-partner competing jobs that factor is a tie. What that means in practice:

• Among jobs of the same QOS, fairshare is the deciding factor, an order of magnitude above waiting time.
• When a partner QOS job is in the mix, its QOS factor outweighs any possible fairshare or age difference: on partner hardware, partner work runs first regardless (see QOS Priority below).

Fairshare

Fairshare ensures that all users get their fair portion of cluster resources over time.

How fairshare works

• Each account has a share allocation representing their portion of the cluster
• Slurm tracks your recent usage (CPU-hours consumed)
• If you've used less than your share, your priority increases
• If you've used more than your share, your priority decreases
• Usage decays over time, so heavy past usage won't penalize you forever

The fairshare algorithm: Fair Tree

Hazel runs Slurm's Fair Tree algorithm (PriorityFlags=FAIR_TREE). Rather than computing an isolated score for each user from a formula, Fair Tree ranks everyone on the cluster:

• At each level of the account tree, sibling accounts are compared by Level FS = NormShares / EffectvUsage — share divided by usage. An account that has used less than its share scores above 1; one that has used more scores below 1.
• Slurm walks the tree from the root, visiting higher-scoring siblings first, and ranks every user in the order visited.
• Your fairshare factor is derived from your rank position, so it always lands between 0 and 1.

A factor near 1.0 means your account branch has used little of its share recently (highest priority); a factor near 0 means your branch has used far more than its share. Because accounts are ranked before users, heavy usage by a sibling project in your department lowers your whole branch's standing — and a lightly-used account's users all rank ahead of a heavily-used account's users.

Check your fairshare

$ sshare -u $USER
             Account       User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare
-------------------- ---------- ---------- ----------- ----------- ------------- ----------
       myproject_cpu    unityID        100    0.005000      150000      0.003500   0.750000
       myproject_gpu    unityID        100    0.001000       50000      0.001100   0.870000

Each project shows up as two rows: myproject_cpu for your CPU jobs (jobs submitted to the compute or compute_partners partitions) and myproject_gpu for your GPU jobs (gpu / gpu_partners). Slurm tracks usage and priority on each side independently — heavy CPU usage doesn't reduce your priority for GPU work, and vice versa.

Key columns:

• NormShares: Your share as a fraction of total cluster shares
• EffectvUsage: Your effective usage as a fraction of total usage
• FairShare: Your fairshare factor (higher is better)

View detailed fairshare

# Show full account hierarchy
sshare -a

# Show one half of your project
sshare -A myproject_cpu
sshare -A myproject_gpu

# Show all users in one half
sshare -A myproject_cpu -a

QOS Priority

Each QOS has a priority value that adds to your job's base priority:

The QOS factor is normalised: Slurm divides a QOS's priority value by the largest value in use anywhere on the cluster, which is the 100 carried by the partner QOS. So a partner job gets the whole QOS weight, a gpu job gets a tenth of it, and every QOS at 0 contributes nothing from this factor. Because the QOS weight is the largest of the five factors, that gap is decisive: on partner hardware a partner job outranks a non-partner one regardless of fairshare or waiting time.

What puts a scavenger job last is preemption, not its priority number. At 0 it is tied with normal and short, not below them. The difference is that a scavenger job is the only kind that can be stopped and requeued once it is already running, so partner work reclaims its nodes even after it has started.

Job Age

Jobs gain priority the longer they wait in the queue. This prevents starvation where low-priority jobs never run.

The age factor increases linearly up to a maximum. After reaching maximum age priority, the job won't gain additional priority from waiting.

Checking Job Priority

View priority of pending jobs

$ sprio -u $USER
  JOBID     USER   PRIORITY        AGE  FAIRSHARE    QOS
1842001  unityID      77100       2100      75000      0
1842002  unityID      76400       1400      75000      0

The PRIORITY column shows the total priority score. Higher values run first. Both jobs belong to the same user, so they share a fairshare contribution and the older job ranks higher on age alone. With the weights above, the QOS column can only be 0 (all the priority-0 QOS), 100,000 (the gpu QOS), or 1,000,000 (a partner QOS). Totals here omit the small partition and job-size contributions.

View priority breakdown

sprio -j JOBID -l

View all priority factors

sprio -w

This shows the weights assigned to each priority factor.

Why Is My Job Waiting?

Use sq --pend to see why your jobs are pending — it sorts by priority, highest first, so the job nearest to running is at the top:

$ sq --pend
JobID    User     Partition  QOS     Priority  Reason
1842001  unityID  compute    normal  18422     Priority
1842010  unityID  compute    normal  9100      Resources

Equivalent native command: squeue -u $USER -o "%.10i %.9P %.20j %.8u %.2t %.10M %.6D %R"

Common pending reasons

Estimate start time

squeue -j JOBID --start

Note: Start time estimates are approximate and change as other jobs complete or are submitted.

Improving Your Priority

Use appropriate QOS

For jobs under 2 hours, use short or short_gpu. These do not raise your priority, but they let the job run on idle partner nodes as well as the general ones, so there is more hardware it can be scheduled on and it often starts sooner.

For work that needs longer than 2 hours on partner hardware, the preemptable scavenger and scavenger_gpu QOS lift that cap, at the cost of being requeued when a partner needs the nodes back. They are a limited pilot, granted on request.

Request only what you need

Smaller jobs are easier to schedule and consume less of your fairshare allocation:

• Request only the cores your application can use
• Set accurate time limits (jobs ending early return resources)
• Request appropriate memory, not maximum

Use job arrays efficiently

For many similar jobs, array jobs are more efficient than individual submissions and count as a single job against limits.

Spread usage over time

Running many large jobs at once rapidly consumes your fairshare. Spreading jobs over time keeps your priority higher.

Backfill Scheduling

Slurm uses backfill scheduling to improve cluster utilization. Even if your job has lower priority, it may start earlier if:

• It fits in a gap before higher-priority jobs can start
• It won't delay higher-priority jobs
• It has an accurate time limit

Tip: Accurate time limits help backfill scheduling. If your job requests 4 days but only runs 2 hours, you lose backfill opportunities.

Partner Priority

Research groups that have purchased hardware for the cluster receive:

• Dedicated fairshare allocation for their purchased resources, computed independently for CPU and GPU contributions — CPU cores raise priority on the CPU side of the tree; GPUs raise priority on the GPU side.
• Partner QOS with higher priority on their hardware (p_<group> on compute_partners, p_<group>_gpu on gpu_partners).
• Access to general cluster resources at standard priority.

See Running Partner Jobs for the user-facing details and HPC Partner Program for program-level information.

Account Hierarchy

Fairshare is calculated hierarchically. The Slurm account tree is split at the root into a CPU side and a GPU side; every project, department, college, and institution exists as both a _cpu and a _gpu twin:

Root
├── cpu                                       ├── gpu
│   └── Institution_cpu                       │   └── Institution_gpu
│       └── College_cpu                       │       └── College_gpu
│           └── Department_cpu                │           └── Department_gpu
│               └── Project_cpu (account)     │               └── Project_gpu (account)
│                   └── User                  │                   └── User

Each user has associations on both halves. The job-submit plugin picks the correct half based on the partition you choose, so you don't normally need to think about the suffix — it just shows up in sshare, sacct, and sacctmgr output. See Running Partner Jobs for the routing details.

Usage rolls up through each half independently. Heavy CPU usage by one project in your department reduces priority for that department's _cpu branch but leaves the _gpu side unaffected.

Useful Commands Summary

Further Resources

• Partitions and QOS
• Job Monitoring FAQ
• Job Submission FAQ