Job Arrays¶

The embarrassingly parallel problem¶

Many research workflows share the same shape: "Run this analysis on 1,000 input files." Or: "Train this model with 50 different hyperparameter combinations." Or: "Bootstrap this statistic 500 times with different random seeds."

Each run is completely independent: There is no communication between them, no shared state, and no coordination required. This class of problem is called embarrassingly parallel: the work fans out trivially into independent units. If you have the resources to run them simultaneously, you can compress days of serial computation into hours.

Job arrays are Slurm's purpose-built tool for this pattern. Instead of manually submitting hundreds of individual jobs (or writing a shell loop that hammers the scheduler), you submit one job array and Slurm fans it out into N tasks, each with its own task ID.

Your first job array¶

Here's a complete job array script that processes ten input files in parallel:

#!/bin/bash
#SBATCH --job-name=my-array
#SBATCH --array=1-10          # Run tasks numbered 1 through 10
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=00:30:00
#SBATCH --output=logs/task_%a.out   # %a is replaced with the array task ID
#SBATCH --error=logs/task_%a.err

echo "Task ID: $SLURM_ARRAY_TASK_ID"
python analyze.py --input data/file_${SLURM_ARRAY_TASK_ID}.csv

Submit it the same way as any batch job:

mkdir -p logs
sbatch array_job.sh

The key concept: $SLURM_ARRAY_TASK_ID is an environment variable that Slurm sets differently for each task. Task 1 gets SLURM_ARRAY_TASK_ID=1, task 2 gets SLURM_ARRAY_TASK_ID=2, and so on up to 10. You use that variable inside your script to select different inputs, parameters, or configurations. The script logic is the same; only the input changes.

The %a placeholder in --output and --error expands to the task ID, so each task writes to its own log file. Without this, all tasks would try to write to the same file simultaneously, producing garbled output.

Using task IDs to select inputs¶

The task ID is just a number. You have full flexibility in how you map it to inputs.

Map to a list of filenames using a bash array:

#!/bin/bash
#SBATCH --array=0-4   # Zero-indexed to match bash array indexing

FILES=(
    data/sample_A.csv
    data/sample_B.csv
    data/sample_C.csv
    data/sample_D.csv
    data/sample_E.csv
)

INPUT=${FILES[$SLURM_ARRAY_TASK_ID]}
echo "Processing: $INPUT"
python analyze.py --input "$INPUT"

Read from a config file where each line is one task's parameters:

# params.txt — one line per task, task IDs are 1-indexed
# Line 1: --lr 0.001 --dropout 0.1
# Line 2: --lr 0.001 --dropout 0.2
# ...

PARAMS=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)
python train.py $PARAMS

Use the task ID directly as a script argument for parameter sweeps:

# In your Python script, receive the task ID and use it to index a parameter grid
import sys
import itertools

lrs = [1e-4, 1e-3, 1e-2]
dropouts = [0.1, 0.3, 0.5]
grid = list(itertools.product(lrs, dropouts))

task_id = int(sys.argv[1])  # Pass $SLURM_ARRAY_TASK_ID as argument
lr, dropout = grid[task_id]

# In the job script:
python train.py $SLURM_ARRAY_TASK_ID

Controlling concurrency with %¶

By default, Slurm will try to start all array tasks as quickly as resources allow. For a small array (10–50 tasks), this is usually fine. For a large one (hundreds or thousands), you may want to limit how many run simultaneously to avoid overloading shared storage or saturating the scheduler.

The % suffix on --array sets the maximum concurrency:

#SBATCH --array=1-1000%50   # Run up to 50 tasks simultaneously

Tasks beyond the limit queue as PD (pending) with reason ArrayTaskThrottle, and start as earlier tasks finish.

Tracking progress¶

Job arrays appear as a single entry in squeue, with the task range shown in the job ID:

# Show all your jobs (array appears as one row with task range)
squeue --me

# Show only running tasks
squeue --me -t R

# Show detailed state per task
squeue --me -j JOBID

After the array finishes (or partially fails), use sacct to see per-task outcomes:

sacct -j JOBID --format=JobID,State,ExitCode,Elapsed,MaxRSS

Each task appears as JOBID_TASKID (e.g., 12345_7). Tasks that failed with a non-zero exit code will show FAILED or CANCELLED.

Re-running only failed tasks is straightforward. Just pass the specific task IDs:

# Re-run tasks 5, 12, and 47 from a previous array
sbatch --array=5,12,47 array_job.sh

You can also specify ranges with gaps: --array=5-10,15,20-25.

Common pitfalls¶