Quickstart

This walkthrough is for organization members with Slurm SSH access. If someone shared a Kubernetes cluster or namespace with you, start with Access to a shared cluster.

Before you start

You need:

An Andromeda account in an organization that has a running Slurm cluster.
The clusters.read and clusters.ssh permissions in that organization. The Engineer profile includes both. If you’re not sure, continue: step 2 shows how to check.
The andromeda CLI, jq, and OpenSSH. Authentication explains how to get the CLI.

andromeda auth login

Open the URL the CLI prints, sign in, and enter the code. See Authentication for details.

Find your organization

andromeda call business GET /me > me.json
jq -c '.memberships[] | {org_id, role}' me.json

{"org_id":"org_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3","role":"member"}

Save the organization ID:

export ORG_ID=$(jq -r '.default_org_id // empty' me.json)
echo "$ORG_ID"

If this prints nothing, you belong to more than one organization. Set ORG_ID to the org_id of the one that owns the cluster, from the list above.Check which permissions you have in it:

USER_ID=$(jq -r .user.id me.json)
andromeda call business GET "/organizations/$ORG_ID/members/$USER_ID/permissions" | jq -c .permissions

["clusters.read","clusters.ssh"]

organization.owner or organization.admin also covers everything on this page. If you have neither, and clusters.read or clusters.ssh is missing, ask an owner or admin of the organization to grant it. See Permissions.

Find your cluster

andromeda call platform GET "/clusters?businessOrgId=$ORG_ID" > clusters.json
jq -r '.items[] | [.id, .spec.name, .status.lifecycleState] | @tsv' clusters.json

0199a1b2-c3d4-7e5f-8a9b-0c1d2e3f4a5b	training-h200	active

Save the ID of the first active cluster. If the organization has several, set CLUSTER_ID to the one you want instead.

export CLUSTER_ID=$(jq -r 'first(.items[] | select(.status.lifecycleState == "active")) | .id' clusters.json)

Check that the cluster runs Slurm and that Slurm is ready:

andromeda call platform GET "/clusters/$CLUSTER_ID/options?businessOrgId=$ORG_ID" |
  jq -c '{scheduler: .scheduler.configured.value, ready: .scheduler.ready.status}'

{"scheduler":"sunk","ready":"True"}

sunk means the cluster runs Slurm. If it shows kubernetes, the cluster has no Slurm login node; use kubectl instead.

The key must be Ed25519. Create one with ssh-keygen -t ed25519 if you don’t have one.

Your first key adds you to the user list of every cluster where you have SSH access, which restarts their compute nodes and ends the jobs running on them. If others are running jobs, register your key when they are done; see Access changes restart nodes.

jq -n --arg key "$(cat ~/.ssh/id_ed25519.pub)" '{public_key: $key}' |
  andromeda call \
    --header "Idempotency-Key: $(uuidgen)" \
    --header 'Audit-Reason: Register SSH key' \
    --data @- \
    business POST /me/ssh-keys |
  jq -r .ssh_key.fingerprint

SHA256:Qm9vbGVhbiBhbGdlYnJhIGlzIHRoZSBrZXkgdG8gaXQ

The cluster now sets up your login. Wait until it is ready:

for attempt in $(seq 60); do
  andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" > ssh-access.json || break
  jq -e .ready ssh-access.json > /dev/null && { echo 'SSH access is ready'; break; }
  jq -r .reason ssh-access.json
  sleep 10
done

While your login is being set up, the endpoint returns 200 with ready: false and a reason, which the loop prints. The loop gives up after ten minutes, or at the first HTTP error. If it ends without printing SSH access is ready, see Troubleshooting.srun can still fail for a few minutes after this, while compute nodes and Slurm accounting catch up; see Wait for access.

Connect

andromeda call platform GET "/clusters/$CLUSTER_ID/ssh-access?organizationId=$ORG_ID" > ssh-access.json
HOST=$(jq -r .hostname ssh-access.json)
LOGIN="$CLUSTER_ID.$(jq -r .username ssh-access.json)"

echo "$HOST $(jq -r .host_key ssh-access.json)" >> ~/.ssh/known_hosts
ssh -l "$LOGIN" "$HOST"

The login name is the cluster ID, a dot, and your username. You land in your home directory on the cluster’s login node.

Run on a GPU

On the login node, sinfo lists the partitions and nodes. Run nvidia-smi on one node, with all 8 of its GPUs:

sinfo
srun --nodes=1 --gres=gpu:8 nvidia-smi -L

GPU 0: NVIDIA H200 (UUID: GPU-...)
GPU 1: NVIDIA H200 (UUID: GPU-...)
...
GPU 7: NVIDIA H200 (UUID: GPU-...)

Next steps

Train nanoGPT in a batch job, a complete example on one 8-GPU node.
Connect over SSH: moving files, ~/.ssh/config, and why long work belongs in sbatch.
Configure Slurm: partitions, prolog scripts, packages.
Concepts: organizations, the two cluster APIs, and how access reaches a cluster.
API reference.

Get started

Guides

Before you start

Next steps

​Before you start

​Next steps

Before you start

Next steps