> ## Documentation Index
> Fetch the complete documentation index at: https://docs.andromeda.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Cluster API calls are made as a signed-in person with an OIDC bearer token. There are no API keys, and service-account tokens are refused (Business returns 401, Platform returns 403).
> Platform takes the organization as the query parameter businessOrgId on most cluster routes, but as organizationId on /clusters/{clusterId}/ssh-access and /clusters/{clusterId}/kubeconfig.
> The Business API uses snake_case. The Platform API uses camelCase, except /ssh-access and /machines, which use snake_case.
> Business write requests need Idempotency-Key and Audit-Reason headers.
> Storage capacity can grow but cannot shrink, including disk and PVC resizing. Each purchase adds capacity.
> A cluster's storage can be Ceph, WEKA, or both, or NFS on its own. The Ceph quota, the purchased bytes, and Slurm's default 100 GiB shared-volume request are three different quantities.
> Organization permissions cover every cluster the organization owns. A Kubernetes resource grant shares one cluster or namespace, possibly with someone outside the organization, and lets them read that cluster and fetch its kubeconfig. A grant does not give Slurm SSH, Slurm configuration, or organization storage inventory access. Slurm SSH requires organization membership and clusters.ssh.
> The SSH login name is <cluster-id>.<username>. The gateway supports shells, commands, SFTP (including modern scp), and forwarding to loopback ports on the login node. Legacy scp and rsync need their programs on the login node. Remote forwarding and non-loopback destinations are refused.
> Use POST /me/ssh-keys for personal key registration. POST /organizations/{orgID}/ssh-keys is a deprecated alias and requires no membership. Read the preferred shell from GET /me as user.login_shell; update it with PUT /me/login-preferences.
> GET /clusters/{clusterId}/ssh-access returns 200 with ready and reason, even when access is not ready. ready: true does not mean compute nodes or Slurm accounting are ready. To confirm a key or shell change reached a cluster, require observation_current: true and compare ssh_key_set_generation with generation from GET /me/ssh-keys, and shell with user.login_shell from GET /me.
> Adding or removing a login account restarts the cluster's login nodes, Slurm controller, and accounting service, then its compute nodes one at a time; jobs running on a restarting node end. An account is added or removed when a person with an active key gains or loses SSH access, or a person with SSH access registers their first key or revokes their last. Other key changes and shell changes restart no nodes.

# Connect over SSH

> Manage SSH keys and your login shell, check access, transfer files, forward local ports, and troubleshoot logins.

On a Slurm cluster you work on a login node: you submit jobs, edit files, and read job output there. You reach it
through the Andromeda SSH gateway, with a key you register once.

SSH is available on clusters that run Slurm. A cluster that runs Kubernetes only has no login node; use
[kubectl](/clusters/kubernetes) instead.

[Inside a Slurm cluster](/clusters/concepts#inside-a-slurm-cluster) diagrams the gateway, the login node, and the
compute nodes.

## Before you start

You need:

* The `clusters.ssh` or `clusters.ssh.admin` permission in the organization that owns the cluster. A Kubernetes
  resource grant does not give SSH access. See [Permissions](/clusters/permissions).
* `ORG_ID` and `CLUSTER_ID` set as in the [quickstart](/clusters/quickstart).
* An Ed25519 SSH key. Other key types are refused. To create one:

  ```sh theme={null}
  ssh-keygen -t ed25519
  ```

## Register your key

Register a key once, and every cluster in every organization where you hold `clusters.ssh` accepts it. You do not
need to be a member of any organization to register it.

```sh theme={null}
jq -n --arg key "$(cat ~/.ssh/id_ed25519.pub)" '{public_key: $key}' |
  andromeda call \
    --header "Idempotency-Key: $(uuidgen)" \
    --header 'Audit-Reason: Register SSH key' \
    --data @- \
    business POST /me/ssh-keys |
  jq '{id: .ssh_key.id, fingerprint: .ssh_key.fingerprint}'
```

```json theme={null}
{
  "id": "ssh_key_01J8Z3Q4R5S6T7V8W9X0Y1Z2A5",
  "fingerprint": "SHA256:Qm9vbGVhbiBhbGdlYnJhIGlzIHRoZSBrZXkgdG8gaXQ"
}
```

The fingerprint should match `ssh-keygen -lf ~/.ssh/id_ed25519.pub`. The deprecated
`POST /organizations/{orgID}/ssh-keys` route does the same thing; use `POST /me/ssh-keys`.

A key can be registered once across all of Andromeda. Registering the same key again, with a new idempotency key,
returns `409` with code `invalid_state`. You can register more than one key.

In the console, **SSH keys** in the organization settings lists your keys, with when each was added or revoked, and
lets you register or revoke one and choose your shell. You do not need a running cluster. A Slurm cluster's
**Manage SSH settings** shows the same controls next to the connection details.

## List and revoke keys

List your keys, including revoked ones. You do not need to be a member of any organization:

```sh theme={null}
andromeda call business GET /me/ssh-keys |
  jq '.keys[] | {id, fingerprint, lifecycle_state}'
```

Revoke a key you no longer trust:

```sh theme={null}
andromeda call \
  --header "Idempotency-Key: $(uuidgen)" \
  --header 'Audit-Reason: Laptop lost' \
  business POST "/me/ssh-keys/$KEY_ID/revoke"
```

Revocation is permanent and applies in every organization at once. Revoking a key that is already revoked succeeds and
changes nothing. A revoked key cannot be registered again; create a new key instead. The gateway closes sessions and
forwarded connections that used the revoked key within seconds of the cluster applying the change. Until
`ssh-access` confirms the new key-set generation, treat the key as still usable. See [Wait for access](#wait-for-access).

## Choose your login shell

Your login shell is `/bin/bash` unless you choose another. You can choose any absolute path, such as `/bin/zsh` or
`/usr/bin/fish`. It applies on every cluster where you can log in, in every organization:

```sh theme={null}
andromeda call \
  --header "Idempotency-Key: $(uuidgen)" \
  --header 'Audit-Reason: Prefer zsh' \
  --data '{"shell": "/bin/zsh"}' \
  business PUT /me/login-preferences
```

Read the current choice from `user.login_shell` in [`GET /me`](/api-reference/business/identity#read-the-signed-in-user):

```sh theme={null}
andromeda call business GET /me | jq -r .user.login_shell
```

Your username, uid, gid, and home directory do not change.

Changing your shell restarts no nodes and does not close open sessions. Your next session uses the new shell once the
cluster has applied it. If the shell is missing or not executable on the login node, the session prints a warning and
falls back to `/bin/bash`; if Bash is also missing, the session fails.

The shell applies only to SSH sessions through the gateway, so compute nodes do not need it installed.

## Wait for access

Registering or revoking a key, changing your shell, or being granted `clusters.ssh` starts an access update on the
clusters where you can log in. The update is applied asynchronously, and new logins can fail until it is. Open
sessions continue while their user and key stay authorized and their login node keeps running.

Poll the connection details until `ready` is `true`:

```sh theme={null}
for attempt in $(seq 60); do
  andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" > ssh-access.json || break
  jq -e .ready ssh-access.json > /dev/null && { echo 'SSH access is ready'; break; }
  jq -r .reason ssh-access.json
  sleep 10
done
```

While access is not ready the endpoint still returns `200`, with `ready: false` and a `reason`; see
[`ready` stays `false`](#ready-stays-false).

`ready: true` can still describe an earlier change. To confirm that a particular key or shell change has reached the
cluster, compare what the cluster reports with what Business holds:

```sh theme={null}
andromeda call business GET /me/ssh-keys | jq .generation
andromeda call business GET /me | jq -r .user.login_shell
andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" |
  jq '{observation_current, ssh_key_set_generation, shell}'
```

The change has reached the cluster when `observation_current` is `true`, `ssh_key_set_generation` equals the Business
`generation`, and `shell` matches. Compare `ssh_key_set_generation`, not `manifest_generation`, which counts versions
of the whole cluster's access list. After you revoke your last key, `ready` stays `false` with `reason: no_keys`; use
the same check to confirm the revocation.

`ready` turns `true` once the login nodes and the Slurm controller have your account; it does not wait for compute nodes to
restart or for your Slurm accounting association to be created (see
[Access changes restart nodes](#access-changes-restart-nodes)). Right after you are granted access, SSH can work while
`srun` and batch jobs still fail. If they do, wait for the compute nodes and accounting to catch up and try again.

## Access changes restart nodes

Nodes read the list of users once, when they start. To change that list, the cluster restarts its login nodes, Slurm
controller, and accounting service, and then its compute nodes one at a time. The list changes when:

* a person with an active key gains SSH access, or loses their last grant of it
* a person with SSH access registers their first active key, or revokes their last one

The gateway reads keys and shells directly, so other changes restart no nodes: adding or revoking a key while you keep
another active one, changing your login shell, or moving between `clusters.ssh` and `clusters.ssh.admin`.

<Warning>
  A compute node does not wait for its running jobs before it restarts. Slurm marks the node down, and jobs running on
  it end. Make access changes when the cluster is idle, or be ready to resubmit jobs. Open SSH sessions on a restarting
  login node also close.
</Warning>

## Connect

Read the connection details:

```sh theme={null}
andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" > ssh-access.json
jq . ssh-access.json
```

```json theme={null}
{
  "hostname": "ssh.k8s.prod-nyc3-z001.tenants.andromedacluster.xyz",
  "port": 22,
  "username": "ada",
  "host_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGF0ZXN0a2V5Zm9yZG9jdW1lbnRhdGlvbm9ubHk",
  "host_key_fingerprint": "SHA256:dGhpcyBpcyBhbiBleGFtcGxlIGZpbmdlcnByaW50IG9ubHk",
  "login_mode": "Shared",
  "ready": true,
  "reason": "ready",
  "manifest_generation": 7,
  "observed_manifest_generation": 7,
  "ssh_key_set_generation": 3,
  "key_fingerprints": ["SHA256:Qm9vbGVhbiBhbGdlYnJhIGlzIHRoZSBrZXkgdG8gaXQ"],
  "shell": "/bin/bash",
  "observation_current": true,
  "observed_at": "2026-10-06T12:00:00Z"
}
```

Your SSH login name is the cluster ID and your username, joined by a dot. This is how the gateway knows which
cluster you want:

```sh theme={null}
HOST=$(jq -r .hostname ssh-access.json)
LOGIN="$CLUSTER_ID.$(jq -r .username ssh-access.json)"
```

Trust the gateway's host key before the first connection, so SSH does not prompt you for it:

```sh theme={null}
echo "$HOST $(jq -r .host_key ssh-access.json)" >> ~/.ssh/known_hosts
```

Then connect:

```sh theme={null}
ssh -l "$LOGIN" "$HOST"
```

```text theme={null}
ada@slurm-login-0:~$
```

One gateway serves many clusters, so its host key identifies the gateway, not your cluster. Every cluster on the same
gateway shows the same key.

To connect with a short name, add an entry to `~/.ssh/config`:

```text theme={null}
Host training
  HostName ssh.k8s.prod-nyc3-z001.tenants.andromedacluster.xyz
  User 0199a1b2-c3d4-7e5f-8a9b-0c1d2e3f4a5b.ada
```

```sh theme={null}
ssh training
```

## What the gateway supports

The SSH gateway runs your session on the login node. It supports:

* interactive shells, with terminal resizing
* running a command, `ssh training 'squeue --me'`, which returns the command's exit status
* piping data through standard input and output
* SFTP, including modern `scp` transfers
* remote commands used by legacy `scp` and `rsync`, when those programs are installed on the login node
* local port forwarding to loopback addresses on the login node

It refuses forwarding to non-loopback destinations, remote forwarding (`-R`), agent forwarding, X11 forwarding, and
passing environment variables. It cannot be used as a jump host to other machines.

### Transfer files

Using the `training` entry in your SSH config:

```sh theme={null}
scp train.py training:~/train.py
sftp training
```

You can also pipe files through a command:

```sh theme={null}
# Upload one file
ssh training 'cat > train.py' < train.py

# Upload a directory
tar czf - ./configs | ssh training 'tar xzf -'

# Download a file
ssh training 'cat results/metrics.json' > metrics.json
```

For code, cloning the repository on the login node is usually simpler.

### Forward a local port

To reach a service listening on port 8888 on the login node:

```sh theme={null}
ssh -N -L 8888:127.0.0.1:8888 training
```

Open `http://localhost:8888` on your computer. The destination is the login node, not a compute node. Accepted
destinations are `localhost` and loopback IP addresses. Forwarded connections close if the SSH connection ends or
its key or access is revoked.

## Sessions end without warning

The gateway closes your SSH session when any of these happen:

* The login node your session runs on restarts. That happens for a change to the user list (see
  [Access changes restart nodes](#access-changes-restart-nodes)) and for some configuration changes, including the
  [Slurm configuration](/clusters/slurm-configuration).
* Your access is revoked, or the key the session used is revoked.
* The gateway cannot confirm your access.

Another member adding or revoking a key, when that does not change the user list, does not close your session.

It does not wait for commands in the session to finish. Submit long-running work as a Slurm batch job with `sbatch`:
the job runs on compute nodes and keeps running after you disconnect. A foreground `srun`, or a process left running
on the login node, can end with the session.

## Login modes

`login_mode` says how the cluster runs login nodes:

| Mode | Behavior |
| - | - |
| `Shared` | Everyone logs in to the same login node, `slurm-login-0`. |
| `PerUser` | You get a login node of your own. It starts on your first connection, which can take up to three minutes, and is removed about ten minutes after your last session ends. |

If the cluster has shared storage, your home directory is on it in both modes, so it survives login nodes being
replaced. See [Files and storage](#files-and-storage).

## Files and storage

| Path | What it is |
| - | - |
| `/home/<username>` | Your home directory. Shared by login and compute nodes when shared storage is configured. |
| `/shared` | Space shared by everyone on the cluster, visible on login and compute nodes when shared storage is configured. |
| `/tmp`, `/dev/shm` on compute nodes | Local to one node. Use for scratch data only. |

Ceph or NFS can back the shared filesystem. To see which one a cluster uses and its capacity, read
[cluster options](/api-reference/platform/clusters#read-cluster-options) or
[`GET /organizations/{orgId}/storage`](/api-reference/platform/storage). Slurm's shared volume requests 100 GiB by
default, which is not the same as the cluster's purchased capacity.

A cluster without shared storage gives each node its own `/home`. Files there are not shared with compute nodes
and do not survive the node being replaced.

<Warning>
  Storage capacity can grow but cannot shrink, and deleting files does not reduce it. See
  [Increase storage capacity](/api-reference/platform/storage#increase-storage-capacity).
</Warning>

## Run something

Your default Slurm account is set for you. Unless your organization has changed the partitions, `all` is the default
partition and contains every node, and each GPU type has its own partition, such as `compute-h200`. Each compute node
has 8 GPUs.

```sh theme={null}
sinfo
srun --nodes=1 --gres=gpu:8 nvidia-smi -L
```

For work longer than a few minutes, use `sbatch`. The [nanoGPT speedrun example](/clusters/examples/nanogpt-speedrun) is a
complete batch job.

## Troubleshooting

| Symptom | Cause and fix |
| - | - |
| `ssh-access` returns `403` | You lack `clusters.ssh`. Ask an owner or admin; see [Permissions](/clusters/permissions). |
| `ready` stays `false` | Check `reason`; see [`ready` stays `false`](#ready-stays-false) below. |
| `Permission denied (publickey)` | Check that `ready` is `true`, that the login name is `<cluster-id>.<username>`, and that SSH offers the key you registered: `ssh -v` lists the keys it tries. |
| `Host key verification failed` | The `known_hosts` entry does not match `host_key` from `ssh-access`. Compare fingerprints with `ssh-keygen -lf ~/.ssh/known_hosts`. |
| The session closes by itself | See [Sessions end without warning](#sessions-end-without-warning). |

### `ready` stays `false`

`reason` says why:

| `reason` | What to do |
| - | - |
| `no_keys` | Register a key; see [Register your key](#register-your-key). |
| `access_not_projected` | You hold `clusters.ssh`, but the cluster's newest access list does not include you yet. Wait. |
| `projection_pending` | The cluster has not applied your current keys, shell, or permission yet. Wait. |
| `gateway_unavailable` | The cluster has not reported a ready SSH gateway. Wait. |
| `observation_stale` | The cluster has not reported recently, so current access is unknown. Wait. |
| `cluster_inactive` | Read the cluster with [`GET /clusters/{clusterId}`](/api-reference/platform/clusters#read-a-cluster). If `status.lifecycleState` is not `active`, wait for it or use another cluster. |
| `named_user_required` | You called with a service credential. Call as yourself. |

If `access_not_projected` or `projection_pending` lasts more than ten minutes, look for yourself in the
[access view](/api-reference/business/clusters#read-a-clusters-access). Until your entry has `allocation`, the cluster
has not accepted an access list that includes you. If it has one and `ready` still stays `false`, contact
[Andromeda Support](/support) with the cluster ID.

## Reference

* [`POST /me/ssh-keys`](/api-reference/business/ssh-keys#register-an-ssh-key)
* [`GET /me/ssh-keys`](/api-reference/business/ssh-keys#list-your-ssh-keys)
* [`POST /me/ssh-keys/{keyID}/revoke`](/api-reference/business/ssh-keys#revoke-an-ssh-key)
* [`GET /me`](/api-reference/business/identity#read-the-signed-in-user) and [`PUT /me/login-preferences`](/api-reference/business/ssh-keys#login-shell)
* [`GET /clusters/{clusterId}/ssh-access`](/api-reference/platform/access#read-ssh-access)
* [`GET /organizations/{orgID}/clusters/{clusterID}/access`](/api-reference/business/clusters#read-a-clusters-access)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.