Use case: workshop users¶

The scenario:

I am hosting a hackathon and want to add non-admin users. Each user should only be allowed to run a single Slurm job (max 8 GPUs) at a time.

This takes three steps: create the users, create a limiting QoS, and make sure Slurm actually enforces it. The steps differ between Slinky and native Slurm clusters.

On Slinky clusters¶

1. Create the users¶

Use slinky-user-add on the login node. One command per user provisions the Kanidm identity, POSIX account, Slurm association and SSH access:

sudo slinky-user-add alice --display-name "Alice Example" \
  --ssh-key "ssh-ed25519 AAAA... alice@laptop" --enable-seamless --test

--enable-seamless opens the seamless SSH port (once, for the whole cluster) so users land in the Slurm login pod as themselves. --test runs a smoke test that confirms the user can start a Slurm step.

Users then connect with:

ssh alice@<cluster-ip> -p 2222

Tip

The first SSH attempt as a freshly created user can fail with Permission denied while the nodes' kanidm caches warm up. Retry after a few seconds.

2. Limit each user to one job¶

Create the QoS once, then assign it to every workshop user (run as admin on the login node; -i commits immediately instead of prompting):

sudo sacctmgr -i add qos limited_qos MaxTRESPerUser=gres/gpu=8 MaxJobsPerUser=1
sudo sacctmgr -i modify user alice set DefaultQOS=limited_qos QOS=limited_qos

Note

If the QoS already exists, add qos prints Nothing new added. and exits 1, and it never changes limits on an existing QoS. Adjust an existing one with modify:

sudo sacctmgr -i modify qos limited_qos set MaxTRESPerUser=gres/gpu=8 MaxJobsPerUser=1

3. Enable enforcement¶

Slinky clusters ship with AccountingStorageEnforce unset, so QoS limits are not enforced until you add this via the controller's extraConf (see Changing the Slurm configuration):

AccountingStorageEnforce=associations,limits,qos

Note

With associations enforced, every submitting identity needs a Slurm association. Admin submissions through the login-node wrappers run as root, which keeps working because the root association (account root) is created automatically with the cluster. After enabling enforcement, verify the admin path with sudo srun true.

4. Verify¶

First confirm the controller loaded the limits:

scontrol show assoc_mgr qos=limited_qos   # shows MaxJobsPU=1 and usage counters

Then, as a workshop user (over seamless SSH), confirm the identity and the limit:

srun id                      # shows the user's own uid, not root
sbatch --wrap "sleep 300" && sbatch --wrap "sleep 300"
squeue --me -O jobid,state,reason   # second job pends with QOSMaxJobsPerUserLimit

Tip

By default an over-limit job pends until capacity frees up. To reject it at submit time instead (clearer for workshop users), set:

sudo sacctmgr -i modify qos limited_qos set Flags=DenyOnLimit

On native Slurm clusters¶

Use the helper script, which creates the Kanidm users, SSH keys, Slurm account and per-user QoS in one run:

cluster-make-users.sh

Download it, review it, and edit the USERS map at the top (one line per user: username, display name, gid, limited or unlimited QoS, SSH keys). Then run it from your laptop against the jumphost:

bash cluster-make-users.sh <jumphost-public-ip>

The final summary lists which users were created and whether limits are enforced. AccountingStorageEnforce: OK means the per-user QoS is active; on MISSING, add AccountingStorageEnforce=associations,limits,qos to /etc/slurm/slurm.conf and run scontrol reconfigure.

Users connect with plain SSH to the jumphost (ssh alice@<cluster-ip>) and can reach worker nodes by name.

Removing users afterwards¶

On Slinky:

sudo slinky-user-remove alice --yes --archive-home

On native Slurm, remove the Slurm association and the Kanidm person:

sudo sacctmgr remove user alice
kanidm person delete alice   # on the service node, as idm_admin