> ## Documentation Index
> Fetch the complete documentation index at: https://docs.andromeda.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Cluster API calls are made as a signed-in person with an OIDC bearer token. There are no API keys, and service-account tokens are refused (Business returns 401, Platform returns 403).
> Platform takes the organization as the query parameter businessOrgId on most cluster routes, but as organizationId on /clusters/{clusterId}/ssh-access and /clusters/{clusterId}/kubeconfig.
> The Business API uses snake_case. The Platform API uses camelCase, except /ssh-access and /machines, which use snake_case.
> Business write requests need Idempotency-Key and Audit-Reason headers.
> Storage capacity can grow but cannot shrink, including disk and PVC resizing. Each purchase adds capacity.
> A cluster's storage can be Ceph, WEKA, or both, or NFS on its own. The Ceph quota, the purchased bytes, and Slurm's default 100 GiB shared-volume request are three different quantities.
> Organization permissions cover every cluster the organization owns. A Kubernetes resource grant shares one cluster or namespace, possibly with someone outside the organization, and lets them read that cluster and fetch its kubeconfig. A grant does not give Slurm SSH, Slurm configuration, or organization storage inventory access. Slurm SSH requires organization membership and clusters.ssh.
> The SSH login name is <cluster-id>.<username>. The gateway supports shells, commands, SFTP (including modern scp), and forwarding to loopback ports on the login node. Legacy scp and rsync need their programs on the login node. Remote forwarding and non-loopback destinations are refused.
> Use POST /me/ssh-keys for personal key registration. POST /organizations/{orgID}/ssh-keys is a deprecated alias and requires no membership. Read the preferred shell from GET /me as user.login_shell; update it with PUT /me/login-preferences.
> GET /clusters/{clusterId}/ssh-access returns 200 with ready and reason, even when access is not ready. ready: true does not mean compute nodes or Slurm accounting are ready. To confirm a key or shell change reached a cluster, require observation_current: true and compare ssh_key_set_generation with generation from GET /me/ssh-keys, and shell with user.login_shell from GET /me.
> Adding or removing a login account restarts the cluster's login nodes, Slurm controller, and accounting service, then its compute nodes one at a time; jobs running on a restarting node end. An account is added or removed when a person with an active key gains or loses SSH access, or a person with SSH access registers their first key or revokes their last. Other key changes and shell changes restart no nodes.

# Serve GLM-5.3-Flash with SGLang

> Serve the 320B-parameter GLM-5.3-Flash on one 8-GPU H100 node of an Andromeda SUNK cluster with SGLang, as a Slurm batch job, and call its OpenAI-compatible API from the login node.

[GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) is an open-weight model from Z.ai, under the MIT
license. It is a mixture of experts with 320 billion parameters, of which 18 billion are active for each token. It
reads text and images, and it reasons before it answers. The official checkpoint is stored in FP8 and takes 328 GB,
more than the 320 GB of four H100s, so the model needs a whole 8-GPU node.

In this example you serve it with [SGLang](https://github.com/sgl-project/sglang) on one H100 node of a SUNK cluster.
A first job downloads the weights and the SGLang container to your home directory without using any GPUs. A second
job runs the server on a whole node, and you call its OpenAI-compatible API from the login node.

## What you need

* An H100 SUNK cluster and the `training` host alias. Rent and connect as in the first three steps of the
  [nanoGPT speedrun](/clusters/examples/nanogpt-speedrun). One H100 node runs the server, and H200 nodes work too.
* About 400 GB free in your home directory, for the weights and the container. Check with
  `ssh training 'df -h ~'`. If there is less, buy more storage before you start; see
  [Increase storage capacity](/api-reference/platform/storage#increase-storage-capacity).

## Set up the directory

Create a directory for the model, and an API key for the server in it:

```sh theme={null}
ssh training 'mkdir -p glm && umask 077 && openssl rand -hex 24 > glm/api-key'
```

The server listens on a port that everyone on the cluster can reach, so it accepts only requests that carry this key.
`umask 077` makes the file readable only by you.

## The download script

```bash theme={null}
#SBATCH --job-name=glm-download
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=06:00:00
#SBATCH --output=%x-%j.out

if [ ! -f "$IMAGE" ]; then
  srun --container-image=docker://lmsysorg/sglang:v0.5.21 --container-save="$IMAGE" true
fi

srun --container-image="$IMAGE" hf download zai-org/GLM-5.3-Flash --local-dir "$MODEL"
```

The download asks for no GPUs and no whole node, so it runs next to other jobs, and the GPU node is not held while
328 GB arrives. It saves the SGLang container to `~/glm/sglang.sqsh` first, then downloads the model with the
Hugging Face CLI from inside that container.

The image is pinned to SGLang v0.5.21. GLM-5.3-Flash needs v0.5.20 or later, and those releases are built for
CUDA 13 only. SUNK compute nodes run NVIDIA driver 580, which runs CUDA 13 containers.

`hf download` checks what is already in `~/glm/GLM-5.3-Flash` and fetches only what is missing, so if the job fails
or hits its time limit, submitting it again continues where it stopped.

## The serve script

### Resources

```bash theme={null}
#SBATCH --job-name=glm-serve
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out
```

The server takes one whole node: all 8 GPUs, every CPU, and all memory. It runs until the time limit and then
stops, so set `--time` to how long you want to serve. On an H200 cluster, set `--partition=compute-h200`.

### The server

```bash theme={null}
srun --container-image="$IMAGE" \
  --ntasks=1 --gres=gpu:8 --cpus-per-task="$SLURM_CPUS_ON_NODE" \
  bash -c 'unset NCCL_NET; exec python3 -m sglang.launch_server \
    --model-path "$MODEL" \
    --served-model-name glm-5.3-flash \
    --tp-size 8 \
    --ep-size 8 \
    --mem-fraction-static 0.75 \
    --kv-cache-dtype bfloat16 \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --moe-runner-backend deep_gemm \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --api-key "$GLM_API_KEY" \
    --host 0.0.0.0 \
    --port 30000'
```

These are the settings SGLang's [GLM-5.3-Flash cookbook](https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash)
verifies on 8 H100s for throughput, plus the parsers, the key, and the address:

| Flag | What it does |
| - | - |
| `--tp-size 8`, `--ep-size 8` | Splits each layer across the 8 GPUs, and gives each GPU its own share of the 288 experts. |
| `--mem-fraction-static 0.75` | Uses 75% of each GPU's memory for the weights and the KV cache. The rest is working memory. |
| `--kv-cache-dtype bfloat16` | Stores the KV cache in BF16. On Hopper GPUs SGLang has no kernel for this model with an FP8 KV cache. |
| `--dsa-prefill-backend tilelang`, `--dsa-decode-backend tilelang` | The sparse-attention kernels that run on Hopper. |
| `--moe-runner-backend deep_gemm` | The FP8 expert kernels for Hopper. The faster FlashInfer kernels need Blackwell GPUs. |
| `--reasoning-parser glm45` | Moves the model's reasoning out of the answer into `reasoning_content`. |
| `--tool-call-parser glm47` | Returns tool calls as structured `tool_calls` instead of raw text. |
| `--api-key` | Refuses requests without `Authorization: Bearer <key>`. |
| `--host 0.0.0.0`, `--port 30000` | Listens on the node's address, so the login node can reach it. |

SGLang starts one worker per GPU from a single Slurm task, so `--cpus-per-task` gives that task every CPU on the
node. `unset NCCL_NET` keeps NCCL on NVLink inside the node; the
[nanoGPT example](/clusters/examples/nanogpt-speedrun#training) explains why.

## The full scripts

Save both files on your machine.

```bash download.sbatch theme={null}
#!/bin/bash
# Download GLM-5.3-Flash and the SGLang container to ~/glm, on the cluster's Ceph filesystem. Uses no GPUs.
# Submit from ~/glm: sbatch download.sbatch
#SBATCH --job-name=glm-download
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=06:00:00
#SBATCH --output=%x-%j.out

set -euo pipefail

WORK=$HOME/glm
IMAGE=$WORK/sglang.sqsh
MODEL=$WORK/GLM-5.3-Flash

# SGLang v0.5.20 is the first release that runs GLM-5.3-Flash. Save the container once, next to the model.
if [ ! -f "$IMAGE" ]; then
  srun --container-image=docker://lmsysorg/sglang:v0.5.21 --container-save="$IMAGE" true
fi

# 328 GB of FP8 weights. Files already downloaded are skipped, so a second submission resumes.
srun --container-image="$IMAGE" hf download zai-org/GLM-5.3-Flash --local-dir "$MODEL"
```

```bash serve.sbatch theme={null}
#!/bin/bash
# Serve GLM-5.3-Flash with SGLang on one 8-GPU H100 node, on port 30000 of that node.
# Submit from ~/glm once download.sbatch has finished: sbatch serve.sbatch
#SBATCH --job-name=glm-serve
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out

set -euo pipefail

WORK=$HOME/glm
IMAGE=$WORK/sglang.sqsh
export MODEL=$WORK/GLM-5.3-Flash
export GLM_API_KEY=$(cat "$WORK/api-key")

# SGLang's verified H100 settings: tensor and expert parallel over 8 GPUs, a BF16 KV cache, and the Hopper
# sparse-attention and MoE kernels. One task starts a worker per GPU, so give the task every CPU on the node.
# NCCL_NET is unset so that NCCL uses NVLink inside the node, even on RoCE clusters where SUNK sets NCCL_NET=IB.
srun --container-image="$IMAGE" \
  --ntasks=1 --gres=gpu:8 --cpus-per-task="$SLURM_CPUS_ON_NODE" \
  bash -c 'unset NCCL_NET; exec python3 -m sglang.launch_server \
    --model-path "$MODEL" \
    --served-model-name glm-5.3-flash \
    --tp-size 8 \
    --ep-size 8 \
    --mem-fraction-static 0.75 \
    --kv-cache-dtype bfloat16 \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --moe-runner-backend deep_gemm \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --api-key "$GLM_API_KEY" \
    --host 0.0.0.0 \
    --port 30000'
```

## Run it

Copy the scripts to the cluster:

```sh theme={null}
scp download.sbatch serve.sbatch training:glm/
```

Then submit both from the login node. `--dependency=afterok` holds the server until the download has succeeded:

```sh theme={null}
ssh training
cd ~/glm
DOWNLOAD=$(sbatch --parsable download.sbatch)
sbatch --dependency=afterok:$DOWNLOAD serve.sbatch
```

```text theme={null}
Submitted batch job 1043
```

`squeue --me` shows the server as `PD` with reason `Dependency` until the download finishes. Once the model is on
disk, later runs only need `sbatch serve.sbatch`.

## Watch it

```sh theme={null}
tail -f ~/glm/glm-serve-1043.out
```

Startup takes a while. The server reads 328 GB from the shared filesystem, compiles its GPU kernels, and captures
CUDA graphs. Along the way it logs the size of its KV cache:

```text theme={null}
max_total_num_tokens=<tokens>, chunked_prefill_size=<tokens>, max_prefill_tokens=<tokens>, max_running_requests=<n>, context_len=<tokens>
```

`max_total_num_tokens` is how many tokens all running requests can hold at once, so no single request can be
longer. The server is ready when it prints:

```text theme={null}
The server is fired up and ready to roll!
```

## Call it

Run these on the login node, in the `ssh training` session. Find the server's node and read the key:

```sh theme={null}
NODE=$(squeue --me --name=glm-serve -h -o %N)
KEY=$(cat ~/glm/api-key)
curl -s -H "Authorization: Bearer $KEY" "http://$NODE:30000/v1/models" | jq -r '.data[].id'
```

```text theme={null}
glm-5.3-flash
```

Send a chat request:

```sh theme={null}
curl -s "http://$NODE:30000/v1/chat/completions" \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.3-flash",
    "messages": [{"role": "user", "content": "How many times does the letter r appear in strawberry?"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "max_tokens": 4096,
    "chat_template_kwargs": {"reasoning_effort": "low", "clear_thinking": true}
  }' |
  jq -r '.choices[0].message | "reasoning:\n\(.reasoning_content)\n\nanswer:\n\(.content)"'
```

The model always reasons first. `reasoning_effort` sets how much: `low`, `high`, or `max`, the default, which
the model card recommends for benchmarks. `clear_thinking: true` drops the reasoning of earlier turns from the
prompt, which the model card recommends for chat. Z.ai recommends `temperature` 1.0 and `top_p` 0.95.

Any OpenAI-compatible client works, with `http://<node>:30000/v1` as the base URL and the key as the API key.
To send an image, use a message `content` list with an `image_url` part, as with OpenAI models. The URL must be
reachable from the compute node, or be a `data:` URL.

The SSH gateway forwards ports only to the login node, so you cannot reach the server from your own computer.
Run clients on the cluster: on the login node, or in other jobs.

## Lower latency for one user

The settings above favor many requests at once. For one user at a time, SGLang can draft several tokens ahead with
the model's built-in multi-token prediction layer. In `serve.sbatch`, change `--mem-fraction-static` to `0.70` and
add these lines before `--host`:

```bash theme={null}
    --speculative-algorithm EAGLE \
    --speculative-num-steps 5 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 6 \
```

The SGLang cookbook lists these settings for H100 but has not finished verifying them there.

## When you're done

Stop the server:

```sh theme={null}
ssh training 'scancel --me --name=glm-serve'
```

The weights stay in `~/glm`, so the next `sbatch serve.sbatch` skips the download. On-demand clusters are billed
until they end; to stop paying, end the cluster in the console as in the
[nanoGPT example](/clusters/examples/nanogpt-speedrun#when-youre-done).

## Troubleshooting

| Symptom | Cause and fix |
| - | - |
| `glm-serve` stays `PD` with reason `DependencyNeverSatisfied` | The download failed. Read `glm-download-<job ID>.out`, cancel the server job, and submit both again. |
| The download job ends at the time limit | Submit `download.sbatch` again. It resumes. |
| The download fails with `No space left on device` | Your home directory has less than about 400 GB free. Delete the partial download or buy more storage, then submit again. |
| `glm-serve` stays `PD` with reason `Resources` | No H100 node is free. It starts when one is. |
| `sbatch` rejects the partition | The cluster's GPUs are not H100s. Run `sinfo` and use the `compute-` partition it lists. |
| The server runs out of GPU memory while starting | Lower `--mem-fraction-static` to `0.70`. |
| An error about the FP8 KV cache | `--kv-cache-dtype` is set to an FP8 type. Hopper GPUs need `bfloat16` for this model. |
| `curl` reports `Connection refused` | The server is still starting. Wait for `The server is fired up and ready to roll!` in the log. |
| `curl` returns `401` | The request has no key or the wrong one. Send `Authorization: Bearer $(cat ~/glm/api-key)`. |
| Reasoning turns into repeated `!` on requests with many tools | A known SGLang issue with long system prompts, many tools, and `tool_choice: auto` on Hopper GPUs; see [sglang#36669](https://github.com/sgl-project/sglang/issues/36669). |

## Next

* The [SGLang GLM-5.3-Flash cookbook](https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash) has settings
  for other GPUs, more nodes, and benchmark results.
* The [model card](https://huggingface.co/zai-org/GLM-5.3-Flash) describes the model's architecture, benchmarks, and
  chat template.
* [Configure Slurm](/clusters/slurm-configuration), for example to add a prolog that checks GPU health before
  every job.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.