Create a container image for HPC

Overview   Requirements   Create a Python file with demo code   Run the Python file to test it   Create a Dockerfile   Build the image locally   Run a container locally   Export the image   Transfer the image to Discovery   Build an Apptainer image from the Docker image   Run an Apptainer container   Create Slurm job script   Submit the job to Discovery  

Overview  

This KB outlines how to do the following:

  • Create a Docker image on your laptop
  • Export the image as a file and transfer it to a HPC server
  • Convert the image to an Apptainer (formerly Singularity) image
  • Run a container

Requirements  

Create a Python file with demo code   

cat << 'EOF' > fibonacci.py
import os, pickle, sys, time

CHECKPOINT_FILE = "fibonacci.pickle"

class Fibonacci:

  def __init__(self, n):
    self.checkpoint_i = 1
    self.n = n
    self.a = 0
    self.b = 1

  def run(self):
    print(f"Fibonacci: {self.n} (starting at calc {self.checkpoint_i})")
    self.checkpoint_time = time.time()
    if self.n == 0:
      return 0
    elif self.n == 1:
      return self.b
    else:
      for i in range(self.checkpoint_i, self.n):
        self.c = self.a + self.b
        self.a = self.b
        self.b = self.c
        if time.time() - self.checkpoint_time > 10:
          print(f"Checkpoint: {i}")
          self.checkpoint_i = i + 1
          with open(CHECKPOINT_FILE, 'wb') as f:
            pickle.dump(self, f)
          self.checkpoint_time = time.time()
      return self.b

# Handle calling Fibonacci from a script
if __name__ == "__main__":
  # allow for working with large numbers
  sys.set_int_max_str_digits(0)
  if os.path.exists(CHECKPOINT_FILE):
    with open(CHECKPOINT_FILE, 'rb') as f:
      fib = pickle.load(f)
  else:
    fib = Fibonacci(int(sys.argv[1]))
  # start/continue calculating
  result = fib.run()
  print(f"Result: {result}")
  # clean-up checkpoint file if exists
  if os.path.exists(CHECKPOINT_FILE):
    os.remove(CHECKPOINT_FILE)
EOF

Run the Python file to test it   

python ./fibonacci.py 30

Create a Dockerfile   

cat << 'EOF' > Dockerfile
FROM python:3.10-alpine

COPY fibonacci.py /src/
EOF

Build the image locally  

If not on a Mac with a M1 chip or later

docker build -t fibonacci:latest .

If on a Mac with a M1 chip or later, build the image for x86_64 architecture

docker buildx build --platform linux/amd64 -t fibonacci:latest .

Run a container locally   

docker run -it --rm fibonacci:latest python /src/fibonacci.py 30

Export the image   

docker save fibonacci:latest -o fibonacci.tar

Transfer the image to Discovery   

scp fibonacci.tar netid@discovery:

Build an Apptainer image from the Docker image   

ssh netid@discovery

apptainer build fibonacci docker-archive://fibonacci.tar

Run an Apptainer container   

apptainer exec fibonacci python /src/fibonacci.py 40

Create Slurm job script   

cat << 'EOF' > fibonacci-apptainer.sh
#!/bin/bash
#SBATCH --job-name=fibonacci-apptainer
#SBATCH --error=%x-%J.err
#SBATCH --output=%x-%J.out
#SBATCH --requeue
#SBATCH --time=00:01:00
#SBATCH --mem=1G
##SBATCH --account=free
##SBATCH --partition=standard

scontrol show job $SLURM_JOBID > fibonacci-info.$SLURM_JOBID

stdbuf -i0 -o0 -e0 apptainer exec fibonacci.tar.sif python -u /src/fibonacci.py 3000001
EOF

Submit the job to Discovery   

sbatch fibonacci-apptainer.sh

NOTE: You can use the squeue command to see the status of your job.

On this page: