Connect over SSH

On a Slurm cluster you work on a login node: you submit jobs, edit files, and read job output there. You reach it through the Andromeda SSH gateway, with a key you register once. SSH is available on clusters that run Slurm. A cluster that runs Kubernetes only has no login node; use kubectl instead. Inside a Slurm cluster diagrams the gateway, the login node, and the compute nodes.

Before you start

You need:

The clusters.ssh or clusters.ssh.admin permission in the organization that owns the cluster. A Kubernetes resource grant does not give SSH access. See Permissions.
ORG_ID and CLUSTER_ID set as in the quickstart.
An Ed25519 SSH key. Other key types are refused. To create one:
ssh-keygen -t ed25519

Register your key

Register a key once, and every cluster in every organization where you hold clusters.ssh accepts it. You do not need to be a member of any organization to register it.

jq -n --arg key "$(cat ~/.ssh/id_ed25519.pub)" '{public_key: $key}' |
  andromeda call \
    --header "Idempotency-Key: $(uuidgen)" \
    --header 'Audit-Reason: Register SSH key' \
    --data @- \
    business POST /me/ssh-keys |
  jq '{id: .ssh_key.id, fingerprint: .ssh_key.fingerprint}'

{
  "id": "ssh_key_01J8Z3Q4R5S6T7V8W9X0Y1Z2A5",
  "fingerprint": "SHA256:Qm9vbGVhbiBhbGdlYnJhIGlzIHRoZSBrZXkgdG8gaXQ"
}

The fingerprint should match ssh-keygen -lf ~/.ssh/id_ed25519.pub. The deprecated POST /organizations/{orgID}/ssh-keys route does the same thing; use POST /me/ssh-keys. A key can be registered once across all of Andromeda. Registering the same key again, with a new idempotency key, returns 409 with code invalid_state. You can register more than one key. In the console, SSH keys in the organization settings lists your keys, with when each was added or revoked, and lets you register or revoke one and choose your shell. You do not need a running cluster. A Slurm cluster’s Manage SSH settings shows the same controls next to the connection details.

List and revoke keys

List your keys, including revoked ones. You do not need to be a member of any organization:

andromeda call business GET /me/ssh-keys |
  jq '.keys[] | {id, fingerprint, lifecycle_state}'

Revoke a key you no longer trust:

andromeda call \
  --header "Idempotency-Key: $(uuidgen)" \
  --header 'Audit-Reason: Laptop lost' \
  business POST "/me/ssh-keys/$KEY_ID/revoke"

Revocation is permanent and applies in every organization at once. Revoking a key that is already revoked succeeds and changes nothing. A revoked key cannot be registered again; create a new key instead. The gateway closes sessions and forwarded connections that used the revoked key within seconds of the cluster applying the change. Until ssh-access confirms the new key-set generation, treat the key as still usable. See Wait for access. Your login shell is /bin/bash unless you choose another. You can choose any absolute path, such as /bin/zsh or /usr/bin/fish. It applies on every cluster where you can log in, in every organization:

andromeda call \
  --header "Idempotency-Key: $(uuidgen)" \
  --header 'Audit-Reason: Prefer zsh' \
  --data '{"shell": "/bin/zsh"}' \
  business PUT /me/login-preferences

Read the current choice from user.login_shell in GET /me:

andromeda call business GET /me | jq -r .user.login_shell

Your username, uid, gid, and home directory do not change. Changing your shell restarts no nodes and does not close open sessions. Your next session uses the new shell once the cluster has applied it. If the shell is missing or not executable on the login node, the session prints a warning and falls back to /bin/bash; if Bash is also missing, the session fails. The shell applies only to SSH sessions through the gateway, so compute nodes do not need it installed.

Wait for access

Registering or revoking a key, changing your shell, or being granted clusters.ssh starts an access update on the clusters where you can log in. The update is applied asynchronously, and new logins can fail until it is. Open sessions continue while their user and key stay authorized and their login node keeps running. Poll the connection details until ready is true:

for attempt in $(seq 60); do
  andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" > ssh-access.json || break
  jq -e .ready ssh-access.json > /dev/null && { echo 'SSH access is ready'; break; }
  jq -r .reason ssh-access.json
  sleep 10
done

While access is not ready the endpoint still returns 200, with ready: false and a reason; see ready stays false. ready: true can still describe an earlier change. To confirm that a particular key or shell change has reached the cluster, compare what the cluster reports with what Business holds:

andromeda call business GET /me/ssh-keys | jq .generation
andromeda call business GET /me | jq -r .user.login_shell
andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" |
  jq '{observation_current, ssh_key_set_generation, shell}'

The change has reached the cluster when observation_current is true, ssh_key_set_generation equals the Business generation, and shell matches. Compare ssh_key_set_generation, not manifest_generation, which counts versions of the whole cluster’s access list. After you revoke your last key, ready stays false with reason: no_keys; use the same check to confirm the revocation. ready turns true once the login nodes and the Slurm controller have your account; it does not wait for compute nodes to restart or for your Slurm accounting association to be created (see Access changes restart nodes). Right after you are granted access, SSH can work while srun and batch jobs still fail. If they do, wait for the compute nodes and accounting to catch up and try again.

Access changes restart nodes

Nodes read the list of users once, when they start. To change that list, the cluster restarts its login nodes, Slurm controller, and accounting service, and then its compute nodes one at a time. The list changes when:

a person with an active key gains SSH access, or loses their last grant of it
a person with SSH access registers their first active key, or revokes their last one

The gateway reads keys and shells directly, so other changes restart no nodes: adding or revoking a key while you keep another active one, changing your login shell, or moving between clusters.ssh and clusters.ssh.admin.

A compute node does not wait for its running jobs before it restarts. Slurm marks the node down, and jobs running on it end. Make access changes when the cluster is idle, or be ready to resubmit jobs. Open SSH sessions on a restarting login node also close.

Connect

Read the connection details:

andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" > ssh-access.json
jq . ssh-access.json

{
  "hostname": "ssh.k8s.prod-nyc3-z001.tenants.andromedacluster.xyz",
  "port": 22,
  "username": "ada",
  "host_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGF0ZXN0a2V5Zm9yZG9jdW1lbnRhdGlvbm9ubHk",
  "host_key_fingerprint": "SHA256:dGhpcyBpcyBhbiBleGFtcGxlIGZpbmdlcnByaW50IG9ubHk",
  "login_mode": "Shared",
  "ready": true,
  "reason": "ready",
  "manifest_generation": 7,
  "observed_manifest_generation": 7,
  "ssh_key_set_generation": 3,
  "key_fingerprints": ["SHA256:Qm9vbGVhbiBhbGdlYnJhIGlzIHRoZSBrZXkgdG8gaXQ"],
  "shell": "/bin/bash",
  "observation_current": true,
  "observed_at": "2026-10-06T12:00:00Z"
}

Your SSH login name is the cluster ID and your username, joined by a dot. This is how the gateway knows which cluster you want:

HOST=$(jq -r .hostname ssh-access.json)
LOGIN="$CLUSTER_ID.$(jq -r .username ssh-access.json)"

Trust the gateway’s host key before the first connection, so SSH does not prompt you for it:

echo "$HOST $(jq -r .host_key ssh-access.json)" >> ~/.ssh/known_hosts

Then connect:

ssh -l "$LOGIN" "$HOST"

ada@slurm-login-0:~$

One gateway serves many clusters, so its host key identifies the gateway, not your cluster. Every cluster on the same gateway shows the same key. To connect with a short name, add an entry to ~/.ssh/config:

Host training
  HostName ssh.k8s.prod-nyc3-z001.tenants.andromedacluster.xyz
  User 0199a1b2-c3d4-7e5f-8a9b-0c1d2e3f4a5b.ada

ssh training

What the gateway supports

The SSH gateway runs your session on the login node. It supports:

interactive shells, with terminal resizing
running a command, ssh training 'squeue --me', which returns the command’s exit status
piping data through standard input and output
SFTP, including modern scp transfers
remote commands used by legacy scp and rsync, when those programs are installed on the login node
local port forwarding to loopback addresses on the login node

It refuses forwarding to non-loopback destinations, remote forwarding (-R), agent forwarding, X11 forwarding, and passing environment variables. It cannot be used as a jump host to other machines.

Transfer files

Using the training entry in your SSH config:

scp train.py training:~/train.py
sftp training

You can also pipe files through a command:

# Upload one file
ssh training 'cat > train.py' < train.py

# Upload a directory
tar czf - ./configs | ssh training 'tar xzf -'

# Download a file
ssh training 'cat results/metrics.json' > metrics.json

For code, cloning the repository on the login node is usually simpler.

Forward a local port

To reach a service listening on port 8888 on the login node:

ssh -N -L 8888:127.0.0.1:8888 training

Open http://localhost:8888 on your computer. The destination is the login node, not a compute node. Accepted destinations are localhost and loopback IP addresses. Forwarded connections close if the SSH connection ends or its key or access is revoked.

Sessions end without warning

The gateway closes your SSH session when any of these happen:

The login node your session runs on restarts. That happens for a change to the user list (see Access changes restart nodes) and for some configuration changes, including the Slurm configuration.
Your access is revoked, or the key the session used is revoked.
The gateway cannot confirm your access.

Another member adding or revoking a key, when that does not change the user list, does not close your session. It does not wait for commands in the session to finish. Submit long-running work as a Slurm batch job with sbatch: the job runs on compute nodes and keeps running after you disconnect. A foreground srun, or a process left running on the login node, can end with the session. login_mode says how the cluster runs login nodes:

Mode	Behavior
`Shared`	Everyone logs in to the same login node, `slurm-login-0`.
`PerUser`	You get a login node of your own. It starts on your first connection, which can take up to three minutes, and is removed about ten minutes after your last session ends.

If the cluster has shared storage, your home directory is on it in both modes, so it survives login nodes being replaced. See Files and storage.

Files and storage

Path	What it is
`/home/<username>`	Your home directory. Shared by login and compute nodes when shared storage is configured.
`/shared`	Space shared by everyone on the cluster, visible on login and compute nodes when shared storage is configured.
`/tmp`, `/dev/shm` on compute nodes	Local to one node. Use for scratch data only.

Ceph or NFS can back the shared filesystem. To see which one a cluster uses and its capacity, read cluster options or GET /organizations/{orgId}/storage. Slurm’s shared volume requests 100 GiB by default, which is not the same as the cluster’s purchased capacity. A cluster without shared storage gives each node its own /home. Files there are not shared with compute nodes and do not survive the node being replaced.

Storage capacity can grow but cannot shrink, and deleting files does not reduce it. See Increase storage capacity.

Run something

Your default Slurm account is set for you. Unless your organization has changed the partitions, all is the default partition and contains every node, and each GPU type has its own partition, such as compute-h200. Each compute node has 8 GPUs.

sinfo
srun --nodes=1 --gres=gpu:8 nvidia-smi -L

For work longer than a few minutes, use sbatch. The nanoGPT speedrun example is a complete batch job.

Troubleshooting

Symptom	Cause and fix
`ssh-access` returns `403`	You lack `clusters.ssh`. Ask an owner or admin; see Permissions.
`ready` stays `false`	Check `reason`; see `ready` stays `false` below.
`Permission denied (publickey)`	Check that `ready` is `true`, that the login name is `<cluster-id>.<username>`, and that SSH offers the key you registered: `ssh -v` lists the keys it tries.
`Host key verification failed`	The `known_hosts` entry does not match `host_key` from `ssh-access`. Compare fingerprints with `ssh-keygen -lf ~/.ssh/known_hosts`.
The session closes by itself	See Sessions end without warning.

`ready` stays `false`

reason says why:

`reason`	What to do
`no_keys`	Register a key; see Register your key.
`access_not_projected`	You hold `clusters.ssh`, but the cluster’s newest access list does not include you yet. Wait.
`projection_pending`	The cluster has not applied your current keys, shell, or permission yet. Wait.
`gateway_unavailable`	The cluster has not reported a ready SSH gateway. Wait.
`observation_stale`	The cluster has not reported recently, so current access is unknown. Wait.
`cluster_inactive`	Read the cluster with `GET /clusters/{clusterId}`. If `status.lifecycleState` is not `active`, wait for it or use another cluster.
`named_user_required`	You called with a service credential. Call as yourself.

If access_not_projected or projection_pending lasts more than ten minutes, look for yourself in the access view. Until your entry has allocation, the cluster has not accepted an access list that includes you. If it has one and ready still stays false, contact Andromeda Support with the cluster ID.

Get started

Guides

Before you start

Register your key

List and revoke keys

Wait for access

Access changes restart nodes

Connect

What the gateway supports

Transfer files

Forward a local port

Sessions end without warning

Files and storage

Run something

Troubleshooting

`ready` stays `false`

Reference

​Before you start

​Register your key

​List and revoke keys

​Choose your login shell

​Wait for access

​Access changes restart nodes

​Connect

​What the gateway supports

​Transfer files

​Forward a local port

​Sessions end without warning

​Login modes

​Files and storage

​Run something

​Troubleshooting

​ready stays false

​Reference

Before you start

Register your key

List and revoke keys

Choose your login shell

Wait for access

Access changes restart nodes

Connect

What the gateway supports

Transfer files

Forward a local port

Sessions end without warning

Login modes

Files and storage

Run something

Troubleshooting

`ready` stays `false`

Reference