Serve GLM-5.3-Flash with SGLang

GLM-5.3-Flash is an open-weight model from Z.ai, under the MIT license. It is a mixture of experts with 320 billion parameters, of which 18 billion are active for each token. It reads text and images, and it reasons before it answers. The official checkpoint is stored in FP8 and takes 328 GB, more than the 320 GB of four H100s, so the model needs a whole 8-GPU node. In this example you serve it with SGLang on one H100 node of a SUNK cluster. A first job downloads the weights and the SGLang container to your home directory without using any GPUs. A second job runs the server on a whole node, and you call its OpenAI-compatible API from the login node.

What you need

An H100 SUNK cluster and the training host alias. Rent and connect as in the first three steps of the nanoGPT speedrun. One H100 node runs the server, and H200 nodes work too.
About 400 GB free in your home directory, for the weights and the container. Check with ssh training 'df -h ~'. If there is less, buy more storage before you start; see Increase storage capacity.

Set up the directory

Create a directory for the model, and an API key for the server in it:

ssh training 'mkdir -p glm && umask 077 && openssl rand -hex 24 > glm/api-key'

The server listens on a port that everyone on the cluster can reach, so it accepts only requests that carry this key. umask 077 makes the file readable only by you.

The download script

#SBATCH --job-name=glm-download
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=06:00:00
#SBATCH --output=%x-%j.out

if [ ! -f "$IMAGE" ]; then
  srun --container-image=docker://lmsysorg/sglang:v0.5.21 --container-save="$IMAGE" true
fi

srun --container-image="$IMAGE" hf download zai-org/GLM-5.3-Flash --local-dir "$MODEL"

The download asks for no GPUs and no whole node, so it runs next to other jobs, and the GPU node is not held while 328 GB arrives. It saves the SGLang container to ~/glm/sglang.sqsh first, then downloads the model with the Hugging Face CLI from inside that container. The image is pinned to SGLang v0.5.21. GLM-5.3-Flash needs v0.5.20 or later, and those releases are built for CUDA 13 only. SUNK compute nodes run NVIDIA driver 580, which runs CUDA 13 containers. hf download checks what is already in ~/glm/GLM-5.3-Flash and fetches only what is missing, so if the job fails or hits its time limit, submitting it again continues where it stopped.

The serve script

Resources

#SBATCH --job-name=glm-serve
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out

The server takes one whole node: all 8 GPUs, every CPU, and all memory. It runs until the time limit and then stops, so set --time to how long you want to serve. On an H200 cluster, set --partition=compute-h200.

The server

srun --container-image="$IMAGE" \
  --ntasks=1 --gres=gpu:8 --cpus-per-task="$SLURM_CPUS_ON_NODE" \
  bash -c 'unset NCCL_NET; exec python3 -m sglang.launch_server \
    --model-path "$MODEL" \
    --served-model-name glm-5.3-flash \
    --tp-size 8 \
    --ep-size 8 \
    --mem-fraction-static 0.75 \
    --kv-cache-dtype bfloat16 \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --moe-runner-backend deep_gemm \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --api-key "$GLM_API_KEY" \
    --host 0.0.0.0 \
    --port 30000'

These are the settings SGLang’s GLM-5.3-Flash cookbook verifies on 8 H100s for throughput, plus the parsers, the key, and the address:

Flag	What it does
`--tp-size 8`, `--ep-size 8`	Splits each layer across the 8 GPUs, and gives each GPU its own share of the 288 experts.
`--mem-fraction-static 0.75`	Uses 75% of each GPU’s memory for the weights and the KV cache. The rest is working memory.
`--kv-cache-dtype bfloat16`	Stores the KV cache in BF16. On Hopper GPUs SGLang has no kernel for this model with an FP8 KV cache.
`--dsa-prefill-backend tilelang`, `--dsa-decode-backend tilelang`	The sparse-attention kernels that run on Hopper.
`--moe-runner-backend deep_gemm`	The FP8 expert kernels for Hopper. The faster FlashInfer kernels need Blackwell GPUs.
`--reasoning-parser glm45`	Moves the model’s reasoning out of the answer into `reasoning_content`.
`--tool-call-parser glm47`	Returns tool calls as structured `tool_calls` instead of raw text.
`--api-key`	Refuses requests without `Authorization: Bearer <key>`.
`--host 0.0.0.0`, `--port 30000`	Listens on the node’s address, so the login node can reach it.

SGLang starts one worker per GPU from a single Slurm task, so --cpus-per-task gives that task every CPU on the node. unset NCCL_NET keeps NCCL on NVLink inside the node; the nanoGPT example explains why.

The full scripts

Save both files on your machine.

download.sbatch

#!/bin/bash
# Download GLM-5.3-Flash and the SGLang container to ~/glm, on the cluster's Ceph filesystem. Uses no GPUs.
# Submit from ~/glm: sbatch download.sbatch
#SBATCH --job-name=glm-download
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=06:00:00
#SBATCH --output=%x-%j.out

set -euo pipefail

WORK=$HOME/glm
IMAGE=$WORK/sglang.sqsh
MODEL=$WORK/GLM-5.3-Flash

# SGLang v0.5.20 is the first release that runs GLM-5.3-Flash. Save the container once, next to the model.
if [ ! -f "$IMAGE" ]; then
  srun --container-image=docker://lmsysorg/sglang:v0.5.21 --container-save="$IMAGE" true
fi

# 328 GB of FP8 weights. Files already downloaded are skipped, so a second submission resumes.
srun --container-image="$IMAGE" hf download zai-org/GLM-5.3-Flash --local-dir "$MODEL"

serve.sbatch

#!/bin/bash
# Serve GLM-5.3-Flash with SGLang on one 8-GPU H100 node, on port 30000 of that node.
# Submit from ~/glm once download.sbatch has finished: sbatch serve.sbatch
#SBATCH --job-name=glm-serve
#SBATCH --partition=compute-h100
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out

set -euo pipefail

WORK=$HOME/glm
IMAGE=$WORK/sglang.sqsh
export MODEL=$WORK/GLM-5.3-Flash
export GLM_API_KEY=$(cat "$WORK/api-key")

# SGLang's verified H100 settings: tensor and expert parallel over 8 GPUs, a BF16 KV cache, and the Hopper
# sparse-attention and MoE kernels. One task starts a worker per GPU, so give the task every CPU on the node.
# NCCL_NET is unset so that NCCL uses NVLink inside the node, even on RoCE clusters where SUNK sets NCCL_NET=IB.
srun --container-image="$IMAGE" \
  --ntasks=1 --gres=gpu:8 --cpus-per-task="$SLURM_CPUS_ON_NODE" \
  bash -c 'unset NCCL_NET; exec python3 -m sglang.launch_server \
    --model-path "$MODEL" \
    --served-model-name glm-5.3-flash \
    --tp-size 8 \
    --ep-size 8 \
    --mem-fraction-static 0.75 \
    --kv-cache-dtype bfloat16 \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --moe-runner-backend deep_gemm \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --api-key "$GLM_API_KEY" \
    --host 0.0.0.0 \
    --port 30000'

Run it

Copy the scripts to the cluster:

scp download.sbatch serve.sbatch training:glm/

Then submit both from the login node. --dependency=afterok holds the server until the download has succeeded:

ssh training
cd ~/glm
DOWNLOAD=$(sbatch --parsable download.sbatch)
sbatch --dependency=afterok:$DOWNLOAD serve.sbatch

Submitted batch job 1043

squeue --me shows the server as PD with reason Dependency until the download finishes. Once the model is on disk, later runs only need sbatch serve.sbatch.

Watch it

tail -f ~/glm/glm-serve-1043.out

Startup takes a while. The server reads 328 GB from the shared filesystem, compiles its GPU kernels, and captures CUDA graphs. Along the way it logs the size of its KV cache:

max_total_num_tokens=<tokens>, chunked_prefill_size=<tokens>, max_prefill_tokens=<tokens>, max_running_requests=<n>, context_len=<tokens>

max_total_num_tokens is how many tokens all running requests can hold at once, so no single request can be longer. The server is ready when it prints:

The server is fired up and ready to roll!

Call it

Run these on the login node, in the ssh training session. Find the server’s node and read the key:

NODE=$(squeue --me --name=glm-serve -h -o %N)
KEY=$(cat ~/glm/api-key)
curl -s -H "Authorization: Bearer $KEY" "http://$NODE:30000/v1/models" | jq -r '.data[].id'

glm-5.3-flash

Send a chat request:

curl -s "http://$NODE:30000/v1/chat/completions" \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.3-flash",
    "messages": [{"role": "user", "content": "How many times does the letter r appear in strawberry?"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "max_tokens": 4096,
    "chat_template_kwargs": {"reasoning_effort": "low", "clear_thinking": true}
  }' |
  jq -r '.choices[0].message | "reasoning:\n\(.reasoning_content)\n\nanswer:\n\(.content)"'

The model always reasons first. reasoning_effort sets how much: low, high, or max, the default, which the model card recommends for benchmarks. clear_thinking: true drops the reasoning of earlier turns from the prompt, which the model card recommends for chat. Z.ai recommends temperature 1.0 and top_p 0.95. Any OpenAI-compatible client works, with http://<node>:30000/v1 as the base URL and the key as the API key. To send an image, use a message content list with an image_url part, as with OpenAI models. The URL must be reachable from the compute node, or be a data: URL. The SSH gateway forwards ports only to the login node, so you cannot reach the server from your own computer. Run clients on the cluster: on the login node, or in other jobs.

Lower latency for one user

The settings above favor many requests at once. For one user at a time, SGLang can draft several tokens ahead with the model’s built-in multi-token prediction layer. In serve.sbatch, change --mem-fraction-static to 0.70 and add these lines before --host:

    --speculative-algorithm EAGLE \
    --speculative-num-steps 5 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 6 \

The SGLang cookbook lists these settings for H100 but has not finished verifying them there.

When you’re done

Stop the server:

ssh training 'scancel --me --name=glm-serve'

The weights stay in ~/glm, so the next sbatch serve.sbatch skips the download. On-demand clusters are billed until they end; to stop paying, end the cluster in the console as in the nanoGPT example.

Troubleshooting

Symptom	Cause and fix
`glm-serve` stays `PD` with reason `DependencyNeverSatisfied`	The download failed. Read `glm-download-<job ID>.out`, cancel the server job, and submit both again.
The download job ends at the time limit	Submit `download.sbatch` again. It resumes.
The download fails with `No space left on device`	Your home directory has less than about 400 GB free. Delete the partial download or buy more storage, then submit again.
`glm-serve` stays `PD` with reason `Resources`	No H100 node is free. It starts when one is.
`sbatch` rejects the partition	The cluster’s GPUs are not H100s. Run `sinfo` and use the `compute-` partition it lists.
The server runs out of GPU memory while starting	Lower `--mem-fraction-static` to `0.70`.
An error about the FP8 KV cache	`--kv-cache-dtype` is set to an FP8 type. Hopper GPUs need `bfloat16` for this model.
`curl` reports `Connection refused`	The server is still starting. Wait for `The server is fired up and ready to roll!` in the log.
`curl` returns `401`	The request has no key or the wrong one. Send `Authorization: Bearer $(cat ~/glm/api-key)`.
Reasoning turns into repeated `!` on requests with many tools	A known SGLang issue with long system prompts, many tools, and `tool_choice: auto` on Hopper GPUs; see sglang#36669.

The SGLang GLM-5.3-Flash cookbook has settings for other GPUs, more nodes, and benchmark results.
The model card describes the model’s architecture, benchmarks, and chat template.
Configure Slurm, for example to add a prolog that checks GPU health before every job.

Andromeda Reference

Guides

What you need

Set up the directory

The download script

The serve script

Resources

The server

The full scripts

Run it

Watch it

Call it

Lower latency for one user

When you’re done

Troubleshooting

Next

​What you need

​Set up the directory

​The download script

​The serve script

​Resources

​The server

​The full scripts

​Run it

​Watch it

​Call it

​Lower latency for one user

​When you’re done

​Troubleshooting

​Next

What you need

Set up the directory

The download script

The serve script

Resources

The server

The full scripts

Run it

Watch it

Call it

Lower latency for one user

When you’re done

Troubleshooting

Next