How to start a spark engine only, then submit subsequent batch applications against that engine

First start the Spark engine. Use this job submission script, test.slurm.

Then submit it with:

sbatch test.slurm

Use squeue to find your job ID number, so you can use it in the srun --jobid ... command below.

Now create an application called pi.py. Then run the pi.py application against the previously started Spark engine:

srun --jobid=<engine_jobid> --overlap -N 1 -n 1 --pty bash
module load spark4/4.2.0-stack
slurm-spark submit pi.py

Last modified: November 04 2026 12:29:47.