How to start a spark engine only, then submit subsequent interactive applications against that engine

First start the Spark engine. Use this job submission script, test.slurm.

Then submit it with:

sbatch test.slurm

Use squeue to find your job ID number, so you can use it in the slurm-spark start ... command below.

Now create an application called pi.py. Then start a 1-core interactive allocation on a compute node and connect interactively to the previously started Spark engine:

salloc -p compute_partners --qos short -N 1 --ntasks-per-node 1 -t 30:00 --constraint=ib --switches=1@00:30:00 --exclude=c042n01
module load spark4/4.1.1-stack
slurm-spark start 138941

Last modified: July 01 2026 10:44:32.