How to start a spark engine only, then submit subsequent interactive applications against that engine

First start the Spark engine. Use this job submission script, test.slurm.

Then submit it with:

sbatch test.slurm

Use squeue to find your job ID number, so you can use it in the srun --jobid ... command below.

Then run a 1-core interactive job against that previously started Spark engine:

srun --jobid=<engine_jobid> --overlap -N 1 -n 1 --pty bash

Important: Wait for the prompt above to change to a compute node before typing the next commands.

module load spark4/4.2.0-stack
slurm-spark shell --attach

Last modified: November 04 2026 12:29:47.