Job Scheduling¶

If you've submitted your first job, you've already used the scheduler. You wrote a script, asked for some resources, and the cluster ran it somewhere. But what actually happened between sbatch and your output file appearing? Why did your job wait in the queue? And how do you know whether you asked for the right amount of resources?

This page explains how Slurm, the scheduler on Unity, manages the cluster's resources, decides what runs when, and why the choices you make in your #SBATCH directives directly affect how long you wait and how well your job runs.

Why Schedulers Exist¶

Imagine a shared kitchen with 50 cooks and one industrial oven. If everyone just walked up and used the oven whenever they wanted, chaos would follow: someone's soufflé gets interrupted mid-bake, two people try to set different temperatures, and the cook with the biggest dish monopolizes the oven all day while everyone else waits.

An HPC cluster faces the same problem at a much larger scale. Unity has thousands of CPU cores, terabytes of memory, and dozens of GPUs, but also hundreds of researchers who all need them. Without coordination, one user could accidentally (or intentionally) consume the entire machine, leaving everyone else staring at a login node.

The scheduler is the solution. It's a program that sits between users and compute resources, accepting job requests, deciding priority, and allocating resources fairly. No user talks to the hardware directly; everyone goes through the scheduler.

flowchart LR
    U1["User A\nsbatch job_a.sh"] --> S["Scheduler\n(Slurm)"]
    U2["User B\nsbatch job_b.sh"] --> S
    U3["User C\nsbatch job_c.sh"] --> S
    S --> N1["Node 1"]
    S --> N2["Node 2"]
    S --> N3["Node 3"]
    S --> N4["Node ..."]

On your laptop, you run a program and it starts immediately using whatever resources it can grab. On a cluster, you request specific resources, and the scheduler decides when and where to grant them. This is the fundamental shift in thinking: you don't run programs on Unity; you ask Slurm to run them for you.

The Resource Model¶

Every job you submit requests four things, whether you realize it or not: CPUs, memory, time, and a partition. Understanding what each means, and especially the difference between two ways of requesting CPUs, is the key to writing effective job scripts.

CPUs: tasks vs. cores per task¶

This is the single most confusing aspect of Slurm for new users, and it matters more than anything else for getting your jobs to run correctly.

Slurm gives you two knobs for requesting CPU resources:

• --ntasks — how many separate processes to launch
• --cpus-per-task — how many CPU cores each process can use

These aren't interchangeable. They map to fundamentally different parallelism strategies.

--ntasks=4 --cpus-per-task=1 launches 4 separate processes, each with 1 core. This is the model for MPI programs, where multiple independent processes communicate by passing messages. Each process has its own memory space and its own copy of your program.

--ntasks=1 --cpus-per-task=4 launches 1 process that can use 4 cores internally. This is the model for multithreaded programs: a single process that spawns threads to parallelize work. Python's multiprocessing, NumPy's internal threading, and OpenMP all work this way.

graph TB
    subgraph A["--ntasks=4  --cpus-per-task=1"]
        direction LR
        P1["Process 1\n(1 core)"]
        P2["Process 2\n(1 core)"]
        P3["Process 3\n(1 core)"]
        P4["Process 4\n(1 core)"]
    end
    subgraph B["--ntasks=1  --cpus-per-task=4"]
        direction LR
        P0["Process 1"]
        P0 --- C1["core"]
        P0 --- C2["core"]
        P0 --- C3["core"]
        P0 --- C4["core"]
    end

The total CPU count is the same in both cases (4 cores), but the structure is completely different, and using the wrong one means your job either can't parallelize at all, or creates processes that fight over a single core.

Here's a simple decision framework:

Memory¶

Memory (--mem) sets the maximum RAM your job can use. If your job exceeds this, Slurm kills it immediately. No warning, no graceful shutdown. The job exits with state OUT_OF_MEMORY.

#SBATCH --mem=16G         # 16 GB total for the entire job

For MPI jobs where memory needs scale with the number of processes, --mem-per-cpu is often more natural:

#SBATCH --mem-per-cpu=4G  # 4 GB per core (8 tasks × 4G = 32G total)

The Intermediate Slurm Patterns recipe covers the details of choosing between these two options.

Time¶

The --time directive sets a wall-clock time limit, the maximum real-world time your job is allowed to run. When the clock runs out, Slurm kills the job, regardless of whether it's 99% done.

#SBATCH --time=04:00:00   # 4 hours (HH:MM:SS)

Requesting the right amount of time is a balancing act. Too little and your job gets killed before it finishes. Too much and you wait longer in the queue (we'll explain why in the next section). A good starting strategy is to estimate your runtime, then add 25–50% as a buffer.

Partitions¶

A partition is a logical grouping of nodes, typically organized by hardware type or intended use. Unity has multiple partitions. The four general-access partitions available to all users are:

The preempt partitions give you access to nodes purchased by specific research groups. If the owning group needs their hardware, Slurm can kill your job after a 2-hour grace period. This makes preempt partitions ideal for short or checkpoint-able work where the larger hardware pool outweighs the preemption risk.

You can see all partitions and their limits with:

sinfo -o "%20P %10l %5a %10D %20G"

This shows each partition's name, time limit, availability, node count, and available GPUs (if any).

If you don't specify a partition, your job goes to the default partition (cpu on Unity). For GPU work, you must explicitly target a GPU partition. See GPU Computing for details.

QOS: extending or boosting your jobs¶

Unity also provides Quality of Service (QOS) levels that modify scheduling behavior:

• --qos=short (-q short): Priority boost for jobs under 4 hours. Limited to one running job at a time. Excellent for interactive sessions and quick tests.
• --qos=long (-q long): Required for jobs exceeding 48 hours. Use sparingly; very long jobs are harder for the scheduler to place.

QOS is not a partition. You combine it with a partition: --partition=gpu --qos=short.

How Slurm Decides What Runs¶

When you submit a job with sbatch, it doesn't run immediately. It enters a queue of pending jobs, and Slurm decides the order based on several factors. Understanding these factors explains why some jobs start quickly and others wait, and gives you concrete levers to reduce your wait times.

Priority factors¶

Slurm assigns each pending job a priority score based on a combination of factors:

Fairshare — The most important factor. Slurm tracks how many resources each user (and their research group) has consumed recently. If you've been running large jobs all week, your next job gets lower priority to give other users a turn. If you haven't submitted anything in days, your priority is higher. This prevents any single user from monopolizing the cluster.

Job age — Jobs that have been waiting longer get a gradual priority boost. This ensures that no job waits forever, even from a user with low fairshare.

Job size — Smaller resource requests are inherently easier to schedule because they fit into more available gaps. This isn't a priority factor per se, but it's a scheduling reality: a 1-core, 1-hour job has far more opportunities to start than a 64-core, 7-day job.

Backfill scheduling¶

Slurm doesn't just run jobs in strict priority order. It uses a strategy called backfill: while waiting for enough resources to free up for a high-priority large job, it looks for smaller, shorter jobs that can fit in the gap without delaying the large job.

In this example, the small job starts before the large high-priority job because it fits in the gap and finishes before the large job needs those resources. This is why requesting less time directly translates to shorter wait times: Slurm can backfill your job into gaps that a longer request wouldn't fit.

The Job Lifecycle¶

Every job follows the same path through Slurm:

flowchart LR
    S["Submit\n(sbatch)"] --> PD["Pending\n(PD)"]
    PD --> R["Running\n(R)"]
    R --> CD["Completed\n(CD)"]
    R --> F["Failed\n(F)"]
    R --> TO["Timeout\n(TO)"]
    R --> OOM["Out of\nMemory"]

Common job states¶

Why is my job pending?¶

When squeue --me shows your job as PD, the REASON column tells you why:

Essential Commands¶

You don't need to memorize every Slurm command. These six cover 95% of daily use:

For batch job submission details, see Submit Your First Job. For interactive work on compute nodes, see the Interactive Jobs recipe. For sacct and seff in depth, see Intermediate Slurm Patterns.

Right-Sizing Your Requests¶

Getting resource requests right is an iterative process, not a guessing game. Here's the workflow experienced users follow:

1. Start with a reasonable estimate¶

For a typical single-process research script (Python, R, MATLAB), these are sensible starting points:

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=01:00:00

Adjust upward if you know your data is large (more memory) or your computation is long (more time). If your code can use multiple cores, increase --cpus-per-task.

2. Run the job¶

Submit it and let it finish, or fail. Both outcomes are informative.

3. Check actual usage¶

Once the job completes, run:

seff JOBID

This shows you what you actually used versus what you requested:

Job ID: 12345
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:42:00
CPU Efficiency: 70.00% of 01:00:00 core-walltime
Memory Utilized: 2.80 GB
Memory Efficiency: 70.00% of 4.00 GB

4. Adjust for next time¶

In this example, the job used 2.8 GB of the 4 GB requested and ran for 42 minutes of the 1-hour limit. Both are comfortably within bounds. Good requests. If memory efficiency were 10%, you'd cut the request in half. If the job timed out, you'd increase --time.

What's Next¶

You now understand the concepts behind job scheduling. Here's where to go to put them into practice:

• Interactive Jobs — Get a shell on a compute node for testing and development
• Job Arrays — Run the same analysis on many inputs simultaneously
• Intermediate Slurm Patterns — Job dependencies, advanced diagnostics, and pipeline automation
• GPU Computing — When and how to request GPUs for your jobs