Parallel Python on a Single Node¶

Python's GIL problem¶

Most languages let you spin up threads and run them on separate CPU cores simultaneously. Python doesn't, at least not for regular Python code. The Global Interpreter Lock (GIL) is a mutex inside CPython that allows only one thread to execute Python bytecode at a time, even on a 64-core machine.

For I/O-bound work (waiting on network, disk, APIs), threads are fine. The GIL is released while waiting, so other threads can run. For CPU-bound work (number crunching, data processing, simulations), threads give you the illusion of parallelism through interleaving, but no actual speedup. Your 8 threads take turns on one core rather than running on 8 cores simultaneously.

The fix: use processes instead of threads. Each process gets its own Python interpreter with its own GIL. Eight processes can run on eight cores with true simultaneous execution.

multiprocessing.Pool¶

The multiprocessing module is Python's built-in tool for process-based parallelism. Its Pool class is the quickest way to parallelize a loop: you give it a function and a list of inputs, and it fans the work out across worker processes.

Here's a complete, runnable example. The task is counting prime numbers in successive ranges, a pure CPU-bound computation that benefits from multiple cores.

Save this as parallel_primes.py:

"""Count primes in ranges using a multiprocessing Pool."""

import math
import multiprocessing
import os
import time


def is_prime(n):
    """Check whether a single integer is prime."""
    if n < 2:
        return False
    if n < 4:
        return True
    if n % 2 == 0 or n % 3 == 0:
        return False
    for i in range(5, int(math.sqrt(n)) + 1, 6):
        if n % i == 0 or n % (i + 2) == 0:
            return False
    return True


def count_primes_in_range(args):  # (1)!
    """Count how many primes exist in [start, end)."""
    start, end = args
    return sum(1 for n in range(start, end) if is_prime(n))


def main():
    # Split 0..2_000_000 into 20 equal chunks
    total = 2_000_000
    num_chunks = 20
    chunk_size = total // num_chunks
    chunks = [(i * chunk_size, (i + 1) * chunk_size) for i in range(num_chunks)]

    # Use SLURM_CPUS_PER_TASK if available, otherwise fall back to cpu_count
    num_workers = int(os.environ.get("SLURM_CPUS_PER_TASK", os.cpu_count()))  # (2)!

    # --- Sequential baseline ---
    t0 = time.perf_counter()
    sequential_total = sum(count_primes_in_range(c) for c in chunks)
    t_sequential = time.perf_counter() - t0

    # --- Parallel with multiprocessing ---
    t0 = time.perf_counter()
    with multiprocessing.Pool(processes=num_workers) as pool:  # (3)!
        results = pool.map(count_primes_in_range, chunks)  # (4)!
    parallel_total = sum(results)
    t_parallel = time.perf_counter() - t0

    print(f"Primes below {total:,}: {sequential_total:,}")
    print(f"Sequential:  {t_sequential:.2f}s")
    print(f"Parallel ({num_workers} workers): {t_parallel:.2f}s")
    print(f"Speedup: {t_sequential / t_parallel:.1f}x")

    assert sequential_total == parallel_total, "Results don't match!"


if __name__ == "__main__":  # (5)!
    main()

1. Pool.map sends each item to a worker. The function must accept a single argument, so we pack start and end into a tuple.
2. On a Slurm job, this reads the number of CPUs you requested. On your laptop, it falls back to all available cores.
3. The with statement ensures worker processes are cleaned up when the block exits.
4. pool.map(func, iterable) is the parallel equivalent of [func(x) for x in iterable]. It blocks until all results are ready.
5. This guard is required. Without it, each spawned worker process re-imports the module and tries to spawn its own workers, causing an infinite fork bomb.

Run it locally to verify:

uv run python parallel_primes.py

Expected output (timing varies by hardware):

Primes below 2,000,000: 148,933
Sequential:  1.82s
Parallel (8 workers): 0.31s
Speedup: 5.9x

concurrent.futures — the modern API¶

Python 3.2 introduced concurrent.futures as a higher-level interface over multiprocessing and threading. The ProcessPoolExecutor class does the same thing as multiprocessing.Pool with a cleaner API and better integration with Python's exception handling.

Here's the same prime-counting example rewritten:

"""Count primes in ranges using concurrent.futures."""

import math
import os
import time
from concurrent.futures import ProcessPoolExecutor, as_completed


def is_prime(n):
    if n < 2:
        return False
    if n < 4:
        return True
    if n % 2 == 0 or n % 3 == 0:
        return False
    for i in range(5, int(math.sqrt(n)) + 1, 6):
        if n % i == 0 or n % (i + 2) == 0:
            return False
    return True


def count_primes_in_range(start, end):  # (1)!
    """Count how many primes exist in [start, end)."""
    return sum(1 for n in range(start, end) if is_prime(n))


def main():
    total = 2_000_000
    num_chunks = 20
    chunk_size = total // num_chunks
    chunks = [(i * chunk_size, (i + 1) * chunk_size) for i in range(num_chunks)]

    num_workers = int(os.environ.get("SLURM_CPUS_PER_TASK", os.cpu_count()))

    # --- Using executor.map (ordered results) ---
    t0 = time.perf_counter()
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        results = list(executor.map(count_primes_in_range, *zip(*chunks)))  # (2)!
    t_map = time.perf_counter() - t0

    print(f"Primes below {total:,}: {sum(results):,}")
    print(f"Parallel ({num_workers} workers, executor.map): {t_map:.2f}s")

    # --- Using submit + as_completed (results as they finish) ---
    t0 = time.perf_counter()
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        futures = {
            executor.submit(count_primes_in_range, start, end): (start, end)  # (3)!
            for start, end in chunks
        }
        total_primes = 0
        for future in as_completed(futures):  # (4)!
            total_primes += future.result()
    t_submit = time.perf_counter() - t0

    print(f"Parallel ({num_workers} workers, as_completed): {t_submit:.2f}s")


if __name__ == "__main__":
    main()

1. With concurrent.futures, the function can take multiple arguments directly. No tuple packing needed.
2. executor.map works like the built-in map() but runs in parallel. Results come back in input order.
3. executor.submit() schedules a single call and returns a Future object. You can attach metadata (here, the chunk range) to track which result is which.
4. as_completed() yields futures as they finish, not in submission order. This is useful when tasks have very different durations and you want to process results immediately.

Joblib — one-liner parallelism¶

Joblib is a third-party library that wraps process-based parallelism in a single readable expression. If concurrent.futures feels like too much boilerplate for a simple parallel loop, Joblib's Parallel + delayed pattern is worth knowing.

Unlike the previous two options, Joblib is not part of the standard library. Install it in your project:

uv add joblib

If you already have scikit-learn installed, Joblib comes bundled with it.

Here's the same prime-counting example:

"""Count primes in ranges using Joblib."""

import math
import os
from joblib import Parallel, delayed  # (1)!


def is_prime(n):
    if n < 2:
        return False
    if n < 4:
        return True
    if n % 2 == 0 or n % 3 == 0:
        return False
    for i in range(5, int(math.sqrt(n)) + 1, 6):
        if n % i == 0 or n % (i + 2) == 0:
            return False
    return True


def count_primes_in_range(start, end):
    return sum(1 for n in range(start, end) if is_prime(n))


def main():
    total = 2_000_000
    num_chunks = 20
    chunk_size = total // num_chunks
    chunks = [(i * chunk_size, (i + 1) * chunk_size) for i in range(num_chunks)]

    num_workers = int(os.environ.get("SLURM_CPUS_PER_TASK", os.cpu_count()))

    results = Parallel(n_jobs=num_workers)(  # (2)!
        delayed(count_primes_in_range)(start, end)  # (3)!
        for start, end in chunks
    )

    print(f"Primes below {total:,}: {sum(results):,}")


if __name__ == "__main__":
    main()

1. joblib is not in the standard library. Install it with uv add joblib or get it automatically when you install scikit-learn.
2. Parallel(n_jobs=N) creates a parallel executor. The parentheses after it accept a generator of delayed calls.
3. delayed(func)(args) captures a function call without executing it. Joblib collects all of them, then dispatches them to worker processes.

The Parallel(n_jobs=...)(delayed(func)(args) for ...) pattern reads almost like a list comprehension, which makes it appealing for quick parallelization. Joblib also handles progress bars (verbose=10), automatic memory-mapping of large NumPy arrays, and configurable backends (processes or threads).

Which one should you use?¶

Quick decision guide:

• Joblib is the fastest path from a sequential loop to a parallel one. One expression, minimal boilerplate, good defaults. Start here if you just need to parallelize independent function calls.
• concurrent.futures gives you more control: as_completed() for results as they arrive, Future objects for cancellation and callbacks, and clean exception handling. Use it when you need to react to individual results or manage task lifecycle.
• multiprocessing is the power tool. Reach for it when you need shared memory, Queue, Pipe, or other low-level IPC primitives that the higher-level APIs don't expose.

Submitting to Slurm¶

The key Slurm directive for single-node parallel Python is --cpus-per-task. You're running one task (one Python process) that internally spawns worker processes across multiple cores.

#!/bin/bash
#SBATCH --job-name=parallel-primes
#SBATCH --partition=cpu
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=8 (1)
#SBATCH --mem=4G
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err

# Load required modules
module load uv/latest

# Run your work
uv run python parallel_primes.py

1. Match this to the number of workers your script uses. The script reads SLURM_CPUS_PER_TASK automatically.

The script reads SLURM_CPUS_PER_TASK and adjusts its worker count to match:

num_workers = int(os.environ.get("SLURM_CPUS_PER_TASK", os.cpu_count()))

This pattern means you change parallelism in one place (the sbatch --cpus-per-task line) and the Python code adapts.

How many workers?¶

There's no universal answer, but here are practical guidelines:

Start with SLURM_CPUS_PER_TASK. One worker per allocated core is the right default for CPU-bound work.

Budget memory per worker. Each worker process gets its own copy of your program's memory. If your script loads a 2 GB dataset and you spawn 8 workers, you need at least 16 GB of RAM. Use --mem-per-cpu instead of --mem to scale memory with core count:

#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=2G   # 8 CPUs × 2 GB = 16 GB total

Watch for diminishing returns. Spawning processes and shipping data between them has overhead. For very small tasks (milliseconds each), that overhead can dominate. If doubling your workers from 8 to 16 only gives a 1.3x speedup instead of 2x, you've hit the point of diminishing returns. Profile before scaling beyond ~32 workers.

Don't exceed a node's core count. Unity compute nodes typically have 32 to 128 cores depending on the partition. You can check with sinfo -p <partition> -o "%n %c". Requesting more CPUs than a node has will make your job impossible to schedule.

Common pitfalls¶

# This fails:
pool.map(lambda x: x ** 2, range(100))

# This works:
def square(x):
    return x ** 2

pool.map(square, range(100))

When to reach for something else¶

This recipe covers the case where you're writing your own parallel logic in Python. But that's not always the right tool:

• Your tasks are completely independent (no shared code needed)? A Slurm job array is simpler. One sbatch submission fans out N copies of your script, each with a different input. No multiprocessing code needed.

• Your bottleneck is I/O, not CPU? Threads or asyncio are better suited. See the Concurrent Programming fundamentals.

• You need more cores than a single node provides? You need distributed computing. See the mpi4py recipe or the Distributed Computing fundamentals.

• You're using NumPy, scikit-learn, or other scientific libraries? Many of these already parallelize internally via native threads that bypass the GIL. Wrapping them in multiprocessing can actually make things slower by oversubscribing cores. See Parallelism in the Scientific Python Stack for how to control library-level parallelism on Unity.