Slurm Job Submission FAQ
Frequently asked questions about submitting jobs, choosing resources, and job parameters.
Quick Links
- Which partition should I use?
- What QOS should I use?
- How many cores should I request?
- When should I use --nodes vs --ntasks?
- How do I request memory?
- How do I estimate time limits?
- When should I use exclusive node access?
- How do I request GPUs?
- How do I request specific node types?
- How do I run many similar jobs?
- How do I make jobs run in sequence?
Which partition should I use?
Choose a partition based on your resource needs and access level:
| Partition | Access | Use for |
|---|---|---|
| compute | All users | CPU-only jobs (default) |
| gpu | All users | Jobs requiring GPUs |
| compute_partners | Partner projects | CPU jobs with access to partner nodes |
| gpu_partners | Partner projects | GPU jobs with access to partner GPUs |
If you don't specify a partition, compute is used. Partner projects should use the partner partitions to access their dedicated resources.
# Standard GPU job (GPU type is required) #SBATCH --partition=gpu #SBATCH --gres=gpu:a100:1 # Partner GPU job #SBATCH --partition=gpu_partners #SBATCH --gres=gpu:h100:1
What QOS should I use?
QOS (Quality of Service) controls priority and resource limits. Each partition allows specific QOS values:
| QOS | Partition | Max Time | Priority | Best for |
|---|---|---|---|---|
| normal | compute | 4 days | Standard | Most CPU jobs |
| gpu | gpu | 4 days | Standard | Most GPU jobs |
| short | compute_partners | 2 hours | Higher | Quick tests; access to idle partner nodes |
| short_gpu | gpu_partners | 2 hours | Higher | Quick GPU tests; access to idle partner GPUs |
| partner | compute_partners | 4 days | Higher | Partner CPU jobs |
| partner_gpu | gpu_partners | 4 days | Higher | Partner GPU jobs |
For short jobs (under 2 hours): All users can access idle partner resources using the short or short_gpu QOS on the partner partitions:
#SBATCH --partition=compute_partners #SBATCH --qos=short #SBATCH --time=01:30:00
Partner projects: If your project has contributed hardware to the cluster, use the partner partitions with your default partner QOS:
#SBATCH --partition=compute_partners # partner QOS is automatically applied
See Partitions and Resources for the full partition/QOS availability grid.
See Job Priority and Fairshare for details on how priority affects scheduling.
How many cores should I request?
Request only the cores your application can actually use:
- Serial code: Use --ntasks=1
- Threaded code (OpenMP): Use --ntasks=1 --cpus-per-task=N where N is the number of threads
- MPI code: Use --ntasks=N where N is the number of MPI ranks
- Hybrid MPI+OpenMP: Use --ntasks=M --cpus-per-task=T
Requesting more cores than your code uses wastes resources and may delay scheduling.
Example: OpenMP application with 8 threads
#SBATCH --ntasks=1 #SBATCH --cpus-per-task=8 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK ./my_openmp_program
Example: MPI application with 32 ranks
#SBATCH --ntasks=32 srun ./my_mpi_program
When should I use --nodes vs --ntasks?
--ntasks specifies the number of tasks (processes). Slurm distributes these across available nodes automatically.
Use --nodes when you need to control node placement:
- --nodes=1 ensures all tasks run on the same node (important for shared-memory codes)
- --nodes=2 --ntasks-per-node=16 requests exactly 16 tasks on each of 2 nodes
Serial or shared-memory parallel job (single node)
#SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=8
Distributed MPI job (multiple nodes OK)
#SBATCH --ntasks=64 # Slurm decides how to distribute across nodes
MPI job with specific layout
#SBATCH --nodes=4 #SBATCH --ntasks-per-node=16 # Exactly 16 tasks on each of 4 nodes = 64 total
How do I request memory?
By default, jobs receive a proportional share of node memory based on cores requested. To request specific amounts:
- --mem=32G - Request 32 GB for the entire job (per node)
- --mem-per-cpu=4G - Request 4 GB per CPU allocated
Use --mem for single-node jobs. Use --mem-per-cpu when tasks may span multiple nodes.
#SBATCH --ntasks=1 #SBATCH --mem=64G
See estimating memory requirements for guidance on determining memory needs.
How do I estimate time limits?
Set --time to slightly more than your job needs. Format: HH:MM:SS or D-HH:MM:SS
- --time=02:00:00 - 2 hours
- --time=1-00:00:00 - 1 day
- --time=4-00:00:00 - 4 days (maximum for normal QOS)
Tips:
- Shorter time limits may schedule faster (backfill scheduling)
- Use short QOS for jobs under 2 hours
- Check completed job times with sacct -j JOBID --format=Elapsed
- Jobs exceeding time limits are terminated automatically
When should I use exclusive node access?
Use --exclusive when:
- Your job needs all memory on a node
- Running hybrid MPI+OpenMP where threads use all cores
- Benchmarking (to avoid interference from other jobs)
- Your application is sensitive to other processes on the node
#SBATCH --nodes=2 #SBATCH --exclusive
Without --exclusive, Slurm may place other jobs on the same node using unused cores.
How do I request GPUs?
Use the gpu partition and --gres to request GPUs. You must specify the GPU type — untyped requests like --gres=gpu:1 will be rejected.
#SBATCH --partition=gpu #SBATCH --gres=gpu:a100:2 # 2 A100 GPUs
Available types: a10, a30, a100, gtx1080, h100, h200, l40, l40s, p100, rtx_2080
For short GPU jobs (under 2 hours), use the short_gpu QOS to access idle partner GPUs:
#SBATCH --partition=gpu #SBATCH --qos=short_gpu #SBATCH --gres=gpu:a30:1 #SBATCH --time=01:00:00
How do I request specific node types?
Use --constraint to request nodes with specific features. Available constraint features:
CPU Architecture
| Constraint | Description | Cores/Node |
|---|---|---|
| haswell | Intel Xeon E5 v3 | 20 |
| broadwell | Intel Xeon E5 v4 | 24 |
| skylake | Intel Xeon Scalable (Gold 6130) | 32 |
| cascadelake | Intel Xeon 2nd Gen (Gold 6226) | 32 |
| icelake | Intel Xeon 3rd Gen (Gold 6326, Plat 8358) | 32-64 |
| sapphirerapids | Intel Xeon 4th Gen (Plat 8462) | 64 |
| genoa | AMD EPYC 4th Gen (9654) | 192 |
Vendor
| Constraint | Description |
|---|---|
| intel | Intel CPU nodes |
| amd | AMD CPU nodes |
| nvidia | Nodes with NVIDIA GPUs |
Instruction Sets
| Constraint | Description |
|---|---|
| avx | AVX instruction support |
| avx2 | AVX2 instruction support |
| avx512 | AVX-512 instruction support (Skylake and newer) |
| sse4_1 | SSE 4.1 instruction support |
| sse4_2 | SSE 4.2 instruction support |
Network
| Constraint | Description |
|---|---|
| ib | InfiniBand interconnect |
Examples
# Request specific architecture #SBATCH --constraint=sapphirerapids # Request AVX-512 support #SBATCH --constraint=avx512 # Combine constraints with AND (&) #SBATCH --constraint="intel&avx512" # Combine constraints with OR (|) #SBATCH --constraint="icelake|sapphirerapids" # InfiniBand for multi-node MPI #SBATCH --constraint=ib
Constraints are useful when:
- Your code is compiled with CPU-specific optimizations
- You need specific instruction set support (AVX-512, etc.)
- You need a minimum number of cores per node
- Multi-node MPI jobs benefit from InfiniBand
- Benchmarking on consistent hardware
How do I run many similar jobs?
Use array jobs to submit many similar jobs with one script:
#SBATCH --array=1-100 # Creates 100 jobs with indices 1-100
# Use $SLURM_ARRAY_TASK_ID in your script
./myprogram input_${SLURM_ARRAY_TASK_ID}.dat
Limit concurrent tasks with %:
#SBATCH --array=1-1000%50 # Max 50 running at once
See array jobs guide for more details.
How do I make jobs run in sequence?
Use --dependency to control job order:
# Submit first job $ sbatch job1.sh Submitted batch job 12345 # Submit second job to run after first completes $ sbatch --dependency=afterok:12345 job2.sh
Dependency types:
- afterok:JOBID - Run after JOBID completes successfully
- afterany:JOBID - Run after JOBID completes (success or failure)
- afternotok:JOBID - Run only if JOBID fails
- after:JOBID - Run after JOBID starts