Configure Slurm

Andromeda generates a working Slurm configuration for every cluster. You can extend it with your own partitions, slurm.conf settings, prolog and epilog scripts, and packages installed on login and compute nodes. Some changes restart parts of the cluster, and a few can stop Slurm scheduling if they are wrong. Changing the configuration needs the clusters.configure permission. Treat it as root on every node: prolog, epilog, and package scripts run as root. See Permissions.

What you can set

Setting	Type	What it does
`slurmPartitions`	string	Partition lines for `slurm.conf`, replacing the generated partitions.
`slurmExtraConfig`	string	Lines appended to `slurm.conf`.
`cgroupExtraConfig`	string	Lines appended to `cgroup.conf`.
`gresConf`	string	The whole of `gres.conf`, replacing the generated file.
`prolog`	map of file name to script	Scripts run before each job on its nodes.
`epilog`	map of file name to script	Scripts run after each job on its nodes.
`controllerEtc`	map of file name to contents	Extra files in the controller’s `/etc/slurm`, such as `job_submit.lua`.
`loginS6Packages`	string	A script run when a login node starts, for installing packages.
`computeS6Packages`	string	A script run when a compute node starts, for installing packages.

By default a cluster has one partition per GPU type, named like compute-h200, and a default partition all with every node. Setting slurmPartitions replaces all of them. Changes take effect in different ways:

Change	Effect
`prolog`, `epilog`	Updated in place. No restart.
`controllerEtc`	Restarts `slurmctld`.
`loginS6Packages`	Restarts login nodes.
`computeS6Packages`	Restarts every compute node, which ends running jobs.

Any change closes open SSH sessions on the cluster; see Connect over SSH.

Read the configuration

andromeda call platform GET "/clusters/$CLUSTER_ID/scheduler-configuration?businessOrgId=$ORG_ID" > slurm.json
jq . slurm.json

{}

Settings you have never set are omitted, so a cluster that runs only the generated configuration returns an empty object. Keep this file. If a change breaks Slurm, sending it back with PUT restores the configuration you started from.

Change the configuration

PUT replaces all nine settings at once: a setting you leave out is cleared. Always start from the current configuration and change only what you mean to. This adds a prolog that keeps jobs off a node that has lost a GPU, and keeps everything else:

jq --arg script '#!/bin/sh
[ "$(nvidia-smi -L | wc -l)" -eq 8 ]
' '.prolog["10-gpu-check.sh"] = $script' slurm.json > slurm.new.json

andromeda call --data @slurm.new.json \
  platform PUT "/clusters/$CLUSTER_ID/scheduler-configuration?businessOrgId=$ORG_ID"

The response is the configuration as stored. To confirm the cluster received the change, watch spec.schedulerGeneration on the cluster. It increases with each PUT that changes something. Sending the same configuration again changes nothing and does not increase it. When a prolog exits with an error, Slurm drains the node, so it takes no new jobs, and requeues the job. See Prolog in the slurm.conf documentation. To add a setting to slurm.conf:

jq '.slurmExtraConfig = "MaxJobCount=20000\n"' slurm.json > slurm.new.json

slurmExtraConfig replaces what was there before, so include any lines you set earlier.

Validation

Platform checks file names, sizes, and shebang lines before they reach the cluster, and returns 400 with code invalid_scheduler_config and a message saying what is wrong. The validation rules list every check.

Nothing checks the text settings. slurmPartitions, slurmExtraConfig, cgroupExtraConfig, and gresConf go into Slurm’s configuration as written. A typo can stop slurmctld, and then no jobs are scheduled until you fix it. The content of scripts is not checked either: a prolog that always fails drains every node it runs on. After a change, run a short job to confirm Slurm is healthy, and keep the previous configuration ready to restore.

Set defaults for new clusters

An organization can hold a default configuration that every new cluster starts with. It takes the same settings except slurmPartitions and gresConf, which describe one cluster’s hardware.

andromeda call platform GET "/organizations/$ORG_ID/scheduler-configuration"

jq -n '{loginS6Packages: "#!/bin/sh\napt-get install -y tmux htop\n"}' |
  andromeda call --data @- platform PUT "/organizations/$ORG_ID/scheduler-configuration"

PUT replaces the whole default. The default applies only to clusters created after you set it. Existing clusters keep their configuration.

Reference

Slurm configuration endpoints

Get started

Guides

What you can set

Read the configuration

Change the configuration

Validation

Set defaults for new clusters

Reference

​What you can set

​Read the configuration

​Change the configuration

​Validation

​Set defaults for new clusters

​Reference

What you can set

Read the configuration

Change the configuration

Validation

Set defaults for new clusters

Reference