Containers¶

On this Slinky (Slurm-on-Kubernetes) cluster, the supported way to run containers is Apptainer (formerly Singularity) from inside a Slurm step. Every srun/sbatch step runs in a per-job shared jail, and Apptainer works inside that jail.

Info

singularity is an alias of apptainer (the singularity binary is a symlink to apptainer), so either name works. enroot is also installed for manual use.

Run a container in a Slurm step¶

Pull and run a public image directly. Apptainer converts the OCI image to a SIF on the fly:

$ srun apptainer exec docker://alpine:3 cat /etc/os-release
NAME="Alpine Linux"
PRETTY_NAME="Alpine Linux v3.24"

See the Slurm documentation for the full srun/sbatch reference, including --time, --mem and partition selection.

GPU containers¶

Request GPUs with --gpus=N on the step and pass --nv to Apptainer so the host NVIDIA driver userspace is exposed inside the container. Exactly the requested number of GPUs appears:

$ srun --gpus=1 apptainer exec --nv docker://nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)

Warning

The step is GPU-cgroup fenced. A step launched with no --gpus sees no GPUs — nvidia-smi -L prints No devices found. from inside the container. Always request --gpus=N for GPU work.

Tip

Pick an image whose CUDA / framework build matches the GPU architecture. These are Blackwell B300 GPUs (compute_cap 10.3), so use CUDA 13.x and a recent framework build (e.g. a current PyTorch/NGC release). CUDA 12.x images predate B300 support and may fail to run kernels even though nvidia-smi lists the device.

Pull once, reuse many times¶

Converting an OCI image on every step is wasteful for large images. Pull to a SIF in the shared /home filesystem once, then point every step at that file:

# Pull once, inside a job
srun --time=00:10:00 bash -c '
  export APPTAINER_CACHEDIR=/local/apptainer-cache APPTAINER_TMPDIR=/tmp/apptainer-tmp
  mkdir -p "$APPTAINER_CACHEDIR" "$APPTAINER_TMPDIR"
  apptainer pull /home/ubuntu/cuda.sif docker://nvidia/cuda:13.0.0-base-ubuntu24.04'

# In any step — reuse the local SIF (no re-download, no conversion)
srun --gpus=1 apptainer exec --nv /home/ubuntu/cuda.sif nvidia-smi -L

Because /home is shared (and bind-mounted into the jail), the same SIF is usable from any worker, including multi-node jobs.

Cache and temp directories

Set APPTAINER_CACHEDIR and APPTAINER_TMPDIR to node-local paths as above. The default cache lives under $HOME, which puts multi-gigabyte layers on the shared filesystem. The cache is not reclaimed automatically — clear it with apptainer cache clean when you are done.

What is not available¶

pyxis / `srun --container-image`¶

NVIDIA pyxis is not enabled on this cluster, so srun --container-image=... does not work:

$ srun --container-image=alpine:3 true
srun: unrecognized option '--container-image=alpine:3'

The shared jail owns the step's mount namespace, and pyxis would need to own the same namespace — the two are mutually exclusive. Use Apptainer instead. See Shared jail for details on how the jail works.

`docker run`¶

The docker client binary exists in the jail, but there is no in-job Docker daemon, so docker run (and any other daemon-backed command) fails inside a step:

$ srun docker run --rm alpine:3 true
failed to connect to the docker API at unix:///var/run/docker.sock ...

To run a Docker image, reference it with Apptainer (docker://...) as shown above, or pre-pull it to a SIF.