> ## Documentation Index > Fetch the complete documentation index at: https://docs.andromeda.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Cluster API calls are made as a signed-in person with an OIDC bearer token. There are no API keys, and service-account tokens are refused (Business returns 401, Platform returns 403). > Platform takes the organization as the query parameter businessOrgId on most cluster routes, but as organizationId on /clusters/{clusterId}/ssh-access and /clusters/{clusterId}/kubeconfig. > The Business API uses snake_case. The Platform API uses camelCase, except /ssh-access and /machines, which use snake_case. > Business write requests need Idempotency-Key and Audit-Reason headers. > Storage capacity can grow but cannot shrink, including disk and PVC resizing. Each purchase adds capacity. > A cluster's storage can be Ceph, WEKA, or both, or NFS on its own. The Ceph quota, the purchased bytes, and Slurm's default 100 GiB shared-volume request are three different quantities. > Organization permissions cover every cluster the organization owns. A Kubernetes resource grant shares one cluster or namespace, possibly with someone outside the organization, and lets them read that cluster and fetch its kubeconfig. A grant does not give Slurm SSH, Slurm configuration, or organization storage inventory access. Slurm SSH requires organization membership and clusters.ssh. > The SSH login name is .. The gateway supports shells, commands, SFTP (including modern scp), and forwarding to loopback ports on the login node. Legacy scp and rsync need their programs on the login node. Remote forwarding and non-loopback destinations are refused. > Use POST /me/ssh-keys for personal key registration. POST /organizations/{orgID}/ssh-keys is a deprecated alias and requires no membership. Read the preferred shell from GET /me as user.login_shell; update it with PUT /me/login-preferences. > GET /clusters/{clusterId}/ssh-access returns 200 with ready and reason, even when access is not ready. ready: true does not mean compute nodes or Slurm accounting are ready. To confirm a key or shell change reached a cluster, require observation_current: true and compare ssh_key_set_generation with generation from GET /me/ssh-keys, and shell with user.login_shell from GET /me. > Adding or removing a login account restarts the cluster's login nodes, Slurm controller, and accounting service, then its compute nodes one at a time; jobs running on a restarting node end. An account is added or removed when a person with an active key gains or loses SSH access, or a person with SSH access registers their first key or revokes their last. Other key changes and shell changes restart no nodes. # Configure Slurm > Partitions, slurm.conf additions, prolog and epilog scripts, what each change restarts, and organization defaults. Andromeda generates a working Slurm configuration for every cluster. You can extend it with your own partitions, `slurm.conf` settings, prolog and epilog scripts, and packages installed on login and compute nodes. Some changes restart parts of the cluster, and a few can stop Slurm scheduling if they are wrong. Changing the configuration needs the `clusters.configure` permission. Treat it as root on every node: prolog, epilog, and package scripts run as root. See [Permissions](/clusters/permissions). ## What you can set | Setting | Type | What it does | | - | - | - | | `slurmPartitions` | string | Partition lines for `slurm.conf`, replacing the generated partitions. | | `slurmExtraConfig` | string | Lines appended to `slurm.conf`. | | `cgroupExtraConfig` | string | Lines appended to `cgroup.conf`. | | `gresConf` | string | The whole of `gres.conf`, replacing the generated file. | | `prolog` | map of file name to script | Scripts run before each job on its nodes. | | `epilog` | map of file name to script | Scripts run after each job on its nodes. | | `controllerEtc` | map of file name to contents | Extra files in the controller's `/etc/slurm`, such as `job_submit.lua`. | | `loginS6Packages` | string | A script run when a login node starts, for installing packages. | | `computeS6Packages` | string | A script run when a compute node starts, for installing packages. | By default a cluster has one partition per GPU type, named like `compute-h200`, and a default partition `all` with every node. Setting `slurmPartitions` replaces all of them. Changes take effect in different ways: | Change | Effect | | - | - | | `prolog`, `epilog` | Updated in place. No restart. | | `controllerEtc` | Restarts `slurmctld`. | | `loginS6Packages` | Restarts login nodes. | | `computeS6Packages` | Restarts every compute node, which **ends running jobs**. | Any change closes open SSH sessions on the cluster; see [Connect over SSH](/clusters/ssh#sessions-end-without-warning). ## Read the configuration ```sh theme={null} andromeda call platform GET "/clusters/$CLUSTER_ID/scheduler-configuration?businessOrgId=$ORG_ID" > slurm.json jq . slurm.json ``` ```json theme={null} {} ``` Settings you have never set are omitted, so a cluster that runs only the generated configuration returns an empty object. Keep this file. If a change breaks Slurm, sending it back with `PUT` restores the configuration you started from. ## Change the configuration `PUT` replaces all nine settings at once: a setting you leave out is cleared. Always start from the current configuration and change only what you mean to. This adds a prolog that keeps jobs off a node that has lost a GPU, and keeps everything else: ```sh theme={null} jq --arg script '#!/bin/sh [ "$(nvidia-smi -L | wc -l)" -eq 8 ] ' '.prolog["10-gpu-check.sh"] = $script' slurm.json > slurm.new.json andromeda call --data @slurm.new.json \ platform PUT "/clusters/$CLUSTER_ID/scheduler-configuration?businessOrgId=$ORG_ID" ``` The response is the configuration as stored. To confirm the cluster received the change, watch `spec.schedulerGeneration` on the [cluster](/api-reference/platform/clusters#read-a-cluster). It increases with each `PUT` that changes something. Sending the same configuration again changes nothing and does not increase it. When a prolog exits with an error, Slurm drains the node, so it takes no new jobs, and requeues the job. See `Prolog` in the [slurm.conf documentation](https://slurm.schedmd.com/slurm.conf.html). To add a setting to `slurm.conf`: ```sh theme={null} jq '.slurmExtraConfig = "MaxJobCount=20000\n"' slurm.json > slurm.new.json ``` `slurmExtraConfig` replaces what was there before, so include any lines you set earlier. ## Validation Platform checks file names, sizes, and shebang lines before they reach the cluster, and returns `400` with code `invalid_scheduler_config` and a message saying what is wrong. The [validation rules](/api-reference/platform/scheduler#validation) list every check. Nothing checks the text settings. `slurmPartitions`, `slurmExtraConfig`, `cgroupExtraConfig`, and `gresConf` go into Slurm's configuration as written. A typo can stop `slurmctld`, and then no jobs are scheduled until you fix it. The content of scripts is not checked either: a prolog that always fails drains every node it runs on. After a change, run a short job to confirm Slurm is healthy, and keep the previous configuration ready to restore. ## Set defaults for new clusters An organization can hold a default configuration that every new cluster starts with. It takes the same settings except `slurmPartitions` and `gresConf`, which describe one cluster's hardware. ```sh theme={null} andromeda call platform GET "/organizations/$ORG_ID/scheduler-configuration" jq -n '{loginS6Packages: "#!/bin/sh\napt-get install -y tmux htop\n"}' | andromeda call --data @- platform PUT "/organizations/$ORG_ID/scheduler-configuration" ``` `PUT` replaces the whole default. The default applies only to clusters created after you set it. Existing clusters keep their configuration. ## Reference * [Slurm configuration endpoints](/api-reference/platform/scheduler) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.