Staging Data for a Compute Job

When to use this

Compute nodes have no route to the outside network: a batch job running on the compute partition cannot rsync from a collaborator's server, wget from a public archive, or pull from cloud storage. The download has to happen first, in a separate xfer job that runs on the data transfer node, which is on the external network and mounts the cluster filesystems.

That leaves you with two jobs that must run in order: the transfer, and then the analysis that reads what it downloaded. You do not want to submit the compute job by hand and hope the transfer has finished — and you certainly do not want it starting on a half-downloaded dataset. Slurm's job dependencies solve this: you tell the compute job to wait until the transfer has completed successfully before it becomes eligible to run.

Step 1: The transfer job

Write a batch script that pulls your data onto /share and submits to the xfer partition. The only directive that matters is --partition=xfer; you do not need --qos or --account. Call this one stage_data.sh:

#!/bin/bash
#SBATCH --partition=xfer            # Run on the data transfer node
#SBATCH --job-name=stage_data
#SBATCH --time=4:00:00              # Default is 1 hour; 1 day is the maximum
#SBATCH --ntasks=1                  # One transfer process, one core (max 4)
#SBATCH --mem=8G                    # Max 24G
#SBATCH --output=stage_data.%j.out

rsync -av --partial collab@remote.example.edu:/data/project/ \
      /share/group_name/user_name/project/

Note the destination path — /share/group_name/user_name/project/. Your compute job in the next step must read from exactly this location. See Transfer Files for what the transfer node can reach, the per-user and per-project limits, and other download tools such as wget, curl, and rclone.

Step 2: The compute job

Write the analysis job as you normally would, reading its input from the path the transfer job wrote to. Nothing about this script mentions the transfer — it just assumes the data is already there. Call it run_analysis.sh:

#!/bin/bash
#SBATCH --job-name=analysis
#SBATCH --time=8:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --output=analysis.%j.out

cd /share/group_name/user_name/project/
./analyze.py --input ./data --output ./results

Step 3: Chain them together

Submit the transfer job first and capture its job ID with --parsable, then submit the compute job with --dependency=afterok: pointing at that ID. A short driver script, submit_workflow.sh, does both:

#!/bin/bash
# submit_workflow.sh

# Submit the transfer job, capture its job ID
XFER=$(sbatch --parsable stage_data.sh)
echo "Submitted transfer: $XFER"

# Submit the compute job; it waits for the transfer to succeed
ANALYSIS=$(sbatch --parsable --dependency=afterok:$XFER run_analysis.sh)
echo "Submitted analysis:  $ANALYSIS (waiting on $XFER)"

Run it from a login node with bash submit_workflow.sh. Both jobs are now in the queue: the transfer starts as soon as the xfer partition has room, and the analysis sits and waits.

If the transfer job is already queued or running and you know its ID, you can attach the compute job to it directly:

sbatch --dependency=afterok:123456 run_analysis.sh

Why afterok and not afterany

afterok releases the compute job only if the transfer finishes with a success exit status. This is what you want: if the download fails or is cut short, running the analysis on missing or partial data wastes a compute allocation and produces results you cannot trust.

When a transfer job fails, its dependent compute job is not run — Slurm leaves it in the queue with the reason DependencyNeverSatisfied and eventually cancels it, so it never consumes compute resources. You will see the failure in the transfer job's --output file and can fix and resubmit.

The related types are occasionally useful: afterany runs the next job regardless of outcome, and afternotok runs only if the first job failed — handy for a cleanup or notification step. The full list is in Job Dependencies.

Watching the workflow

Check both jobs with squeue --me. While the transfer runs, the compute job appears with the reason (Dependency) in the NODELIST(REASON) column, meaning it is waiting on another job. Once the transfer succeeds the reason clears and the compute job becomes eligible to start. See Monitoring Jobs for the full list of job states and reasons.

Things to watch for

• The path the transfer job writes must match the path the compute job reads. A mismatch here is the most common reason a chained workflow runs to completion but the analysis finds no data.
• An xfer job is capped at one day of wall time. If a single download would take longer, split it into several transfers, and make the compute job depend on all of them: --dependency=afterok:$XFER1:$XFER2.
• The xfer partition is for moving data, not computing on it. Do the download in the transfer job and all analysis in the compute job — do not try to combine them.
• A dependency only enforces order, not correctness. afterok confirms the transfer command exited successfully; it does not verify that every file arrived. For critical runs, have the transfer script check its own results (for example, verify a checksum) and exit non-zero on any problem so afterok holds the compute job back.