Job Priority and Fairshare

How Job Priority Works

When multiple jobs are waiting for resources, Slurm uses a priority score to decide which job runs first. Higher priority jobs are scheduled before lower priority jobs.

The priority score is calculated from several factors:

• Fairshare: Your recent usage compared to your allocation
• QOS priority: Bonus from Quality of Service settings
• Job age: How long the job has been waiting
• Job size: Number of resources requested
• Partition priority: Priority assigned to the partition

Fairshare

Fairshare is the primary factor in job priority. It ensures that all users get their fair portion of cluster resources over time.

How fairshare works

• Each account has a share allocation representing their portion of the cluster
• Slurm tracks your recent usage (CPU-hours consumed)
• If you've used less than your share, your priority increases
• If you've used more than your share, your priority decreases
• Usage decays over time, so heavy past usage won't penalize you forever

Fairshare formula

Your fairshare factor (between 0 and 1) is calculated as:

Fairshare Factor = 2^(-EffectiveUsage / ShareAllocation)

Where:

• EffectiveUsage is your recent resource consumption (with decay)
• ShareAllocation is your account's assigned share

A fairshare factor of 1.0 means you haven't used any resources recently (highest priority). A factor near 0 means you've used significantly more than your share (lowest priority).

Check your fairshare

$ sshare -u $USER
             Account       User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare
-------------------- ---------- ---------- ----------- ----------- ------------- ----------
       myproject_cpu    unityID        100    0.005000      150000      0.003500   0.750000
       myproject_gpu    unityID        100    0.001000       50000      0.001100   0.870000

Each project shows up as two rows: myproject_cpu for your CPU jobs (jobs submitted to the compute or compute_partners partitions) and myproject_gpu for your GPU jobs (gpu / gpu_partners). Slurm tracks usage and priority on each side independently — heavy CPU usage doesn't reduce your priority for GPU work, and vice versa.

Key columns:

• NormShares: Your share as a fraction of total cluster shares
• EffectvUsage: Your effective usage as a fraction of total usage
• FairShare: Your fairshare factor (higher is better)

View detailed fairshare

# Show full account hierarchy
sshare -a

# Show one half of your project
sshare -A myproject_cpu
sshare -A myproject_gpu

# Show all users in one half
sshare -A myproject_cpu -a

QOS Priority

Each QOS has a priority value that adds to your job's base priority:

The QOS factor is normalised: Slurm divides a QOS's priority value by the largest value in use anywhere on the cluster, which is the 100 carried by the partner QOS. So a partner job gets the whole QOS weight, a gpu job gets a tenth of it, and every QOS at 0 contributes nothing from this factor. Because the QOS weight is the largest of the five factors, that gap is decisive: on partner hardware a partner job outranks a non-partner one regardless of fairshare or waiting time.

What puts a scavenger job last is preemption, not its priority number. At 0 it is tied with normal and short, not below them. The difference is that a scavenger job is the only kind that can be stopped and requeued once it is already running, so partner work reclaims its nodes even after it has started.

Job Age

Jobs gain priority the longer they wait in the queue. This prevents starvation where low-priority jobs never run.

The age factor increases linearly up to a maximum. After reaching maximum age priority, the job won't gain additional priority from waiting.

Checking Job Priority

View priority of pending jobs

$ sprio -u $USER
  JOBID     USER   PRIORITY        AGE  FAIRSHARE    QOS
 123456  unityID      10500       1000       9000    500
 123457  unityID      10200        700       9000    500

The PRIORITY column shows the total priority score. Higher values run first.

View priority breakdown

sprio -j JOBID -l

View all priority factors

sprio -w

This shows the weights assigned to each priority factor.

Why Is My Job Waiting?

Use sq --pend to see why your jobs are pending — it sorts by priority, highest first, so the job nearest to running is at the top:

$ sq --pend
JobID    User     Partition  QOS     Priority  Reason
1842001  unityID  compute    normal  18422     Priority
1842010  unityID  compute    normal  9100      Resources

Equivalent native command: squeue -u $USER -o "%.10i %.9P %.20j %.8u %.2t %.10M %.6D %R"

Common pending reasons

Estimate start time

squeue -j JOBID --start

Note: Start time estimates are approximate and change as other jobs complete or are submitted.

Improving Your Priority

Use appropriate QOS

For jobs under 2 hours, use short or short_gpu. These do not raise your priority, but they let the job run on idle partner nodes as well as the general ones, so there is more hardware it can be scheduled on and it often starts sooner.

For work that needs longer than 2 hours on partner hardware, the preemptable scavenger and scavenger_gpu QOS lift that cap, at the cost of being requeued when a partner needs the nodes back. They are a limited pilot, granted on request.

Request only what you need

Smaller jobs are easier to schedule and consume less of your fairshare allocation:

• Request only the cores your application can use
• Set accurate time limits (jobs ending early return resources)
• Request appropriate memory, not maximum

Use job arrays efficiently

For many similar jobs, array jobs are more efficient than individual submissions and count as a single job against limits.

Spread usage over time

Running many large jobs at once rapidly consumes your fairshare. Spreading jobs over time keeps your priority higher.

Backfill Scheduling

Slurm uses backfill scheduling to improve cluster utilization. Even if your job has lower priority, it may start earlier if:

• It fits in a gap before higher-priority jobs can start
• It won't delay higher-priority jobs
• It has an accurate time limit

Tip: Accurate time limits help backfill scheduling. If your job requests 4 days but only runs 2 hours, you lose backfill opportunities.

Partner Priority

Research groups that have purchased hardware for the cluster receive:

• Dedicated fairshare allocation for their purchased resources, computed independently for CPU and GPU contributions — CPU cores raise priority on the CPU side of the tree; GPUs raise priority on the GPU side.
• Partner QOS with higher priority on their hardware (p_<group> on compute_partners, p_<group>_gpu on gpu_partners).
• Access to general cluster resources at standard priority.

See Running Partner Jobs for the user-facing details and HPC Partner Program for program-level information.

Account Hierarchy

Fairshare is calculated hierarchically. The Slurm account tree is split at the root into a CPU side and a GPU side; every project, department, college, and institution exists as both a _cpu and a _gpu twin:

Root
├── cpu                                       ├── gpu
│   └── Institution_cpu                       │   └── Institution_gpu
│       └── College_cpu                       │       └── College_gpu
│           └── Department_cpu                │           └── Department_gpu
│               └── Project_cpu (account)     │               └── Project_gpu (account)
│                   └── User                  │                   └── User

Each user has associations on both halves. The job-submit plugin picks the correct half based on the partition you choose, so you don't normally need to think about the suffix — it just shows up in sshare, sacct, and sacctmgr output. See Running Partner Jobs for the routing details.

Usage rolls up through each half independently. Heavy CPU usage by one project in your department reduces priority for that department's _cpu branch but leaves the _gpu side unaffected.

Useful Commands Summary

Further Resources

• Partitions and QOS
• Job Monitoring FAQ
• Job Submission FAQ