Storage Fundamentals¶

When you work on your laptop, you have one hard drive that holds everything: your operating system, your documents, your datasets, and your results. HPC clusters are different. They typically offer multiple storage tiers, each designed for a different purpose. Using the right tier at the right time can dramatically improve your job performance and help you avoid common pitfalls like running out of space mid-job or bottlenecking on slow I/O.

What storage tiers are for¶

HPC workloads generate and consume data at scales that a single file system can't efficiently handle. A genomics pipeline might need to read terabytes of sequencing data, write hundreds of intermediate files during processing, and then store a few gigabytes of final results. Each of those stages has different requirements:

• Reading input data needs reliable, always-available storage.
• Writing intermediate files during a job needs speed. The faster your storage, the less time your CPUs spend waiting on disk.
• Keeping final results needs persistence and enough capacity that you're not constantly deleting old work.

That's why clusters separate storage into tiers.

High-Speed Scratch Storage¶

Scratch storage is designed for speed. It sits on fast disks (often SSDs or parallel file systems like Lustre) and is optimized for the heavy read/write patterns that jobs produce. This is where your jobs should write temporary and intermediate files.

The tradeoff is that scratch storage is not permanent. Files on scratch are typically subject to automatic purge policies: if a file hasn't been accessed in a certain number of days, it may be deleted to free space for other users. Scratch is working space, not archival space.

Long-Term Storage¶

Long-term (or persistent) storage is where you keep datasets, code, and results that need to stick around. It's typically larger in capacity than scratch but slower to access, since it's optimized for durability and availability rather than raw I/O speed.

This is the right place for:

• Input datasets you'll reuse across multiple jobs
• Final results and publications-ready data
• Code repositories and environments
• Anything you can't afford to lose

A Typical Storage Workflow¶

A common pattern for HPC jobs looks like this:

1. Before the job — Your input data lives on long-term storage.
2. Job starts — Your job script copies input data from long-term storage to scratch.
3. Job runs — All intermediate I/O happens on fast scratch storage.
4. Job finishes — Your job script copies results from scratch back to long-term storage.
5. After the job — Scratch files are eventually purged; your results are safe on long-term storage.

flowchart LR
    A["Long-Term Storage"] -->|"1. Copy input"| B["Scratch Storage"]
    B -->|"2. Job reads/writes"| C["Compute Nodes"]
    C -->|"3. Job writes results"| B
    B -->|"4. Copy results back"| A

This approach gives you the best of both worlds: fast I/O where it matters and safe, persistent storage for everything else.

Home Directories¶

Most clusters also give each user a home directory. Your home directory is persistent and backed up, but it usually has a small quota (often just a few gigabytes). It's meant for configuration files, scripts, and small personal files, not for large datasets or job I/O.

Storage on Unity¶

Unity provides a tiered storage system. Here's how each tier maps to the concepts described above:

Work Storage (/work)¶

Your PI group's work directory is the primary location for research data and job I/O. Each PI group gets 1 TB of shared storage by default. This is where you should keep datasets you're actively using, job input files, and results.

Scratch Storage (/scratch)¶

Scratch provides high-speed temporary storage for running jobs. Files on scratch are not backed up and are subject to cleanup policies. Use scratch for intermediate results and temporary files during computation, then copy important results back to your work directory.

Scratch space is managed through the HPC Workspace system. Here's the minimum you need to get started:

# Create a scratch workspace (max 30 days)
ws_allocate myproject 30

This prints the path to your new workspace, something like:

/scratch/workspace/yourname-myproject

Use that path anywhere a recipe says /path/to/your/scratch.

A few other commands you'll reach for:

# List your workspaces and their expiration dates
ws_list

# Extend a workspace before it expires
ws_extend myproject 30

# Release a workspace when you're done
ws_release myproject

export SCRATCH=$(ws_list -s myproject)

For the full set of options (shared workspaces, email reminders, etc.), see the HPC Workspace documentation.

Curated Datasets (/datasets)¶

Unity provides 120+ pre-staged AI/ML models and bioinformatics databases at /datasets. These include popular models (Llama, Whisper, DINO), databases (AlphaFold, BLAST, UniProt), and benchmark datasets (ImageNet, COCO). Using these saves you from downloading large files and counting them against your storage quota.

Which Storage Should I Use?¶

For full storage documentation, see the Unity storage overview.

Key Takeaways¶

Understanding which storage tier to use is one of the simplest ways to improve your HPC experience. The general rule is straightforward: use scratch for speed during jobs, and long-term storage for everything you want to keep. Your home directory is for small personal files and configuration, not for heavy workloads.

What's Next?¶

• What is HPC? if you haven't read the overview yet
• Submit your first job to put storage concepts into practice