GLM-5.2 (W4A8C8)
glm_moe_dsa (GlmMoeDsaForCausalLM) is a 256-expert MoE with MLA attention and
DeepSeek-style sparse attention (DSA), served here with a 1,048,576-token context.
It was validated on 8 × Atlas 800 A2 = 64 × Ascend 910B3 64 GB as a
Prefill/Decode-disaggregated service — 4 prefill nodes and 4 decode nodes — with a
shared mooncake KV pool, through the product MaaS gateway.
Load is a replay of real coding-agent sessions (Exgentic v2, claude_code working
SWE-bench tasks) swept over concurrent sessions, with no other load on the service.
In short — this deployment served 64 concurrent coding-agent sessions with time to first token (TTFT) P90 under 10 s; 128 did not.
At 64 sessions TTFT P90 was 9.25 s and the service produced 510 output tokens/s (111k tokens/s counting input). At 128 sessions TTFT P90 rose to 24.8 s while completed requests per second grew only 1 % (output tokens/s 17 %, as responses were longer): the extra requests queued at prefill. Nothing between 64 and 128 was measured.
TOC
Model identityValidated hardware × stackModel configurationRequest pathDeployBenchmark methodBenchmark resultsWhere the 128-session rung saturatesCaveatsModel identity
Validated hardware × stack
Routing and P/D handoff: hermes-router 26.6.0 (EPP, sha256:8cb00e74…) and the InferNex P→D proxy
proxy-server:26.6.0, both from build-harbor.alauda.cn/3rdparty/openfuyao.
Controller: Alauda Build of InferNex Bridge v26.6.0.
Model configuration
Why gpu-memory-utilization is 0.88, not the 0.93 this deployment started from. Deep-context
prefill needs more temporary memory per layer than the start-up profiling run reserves.
At 0.93 prefill slowed sharply once the context passed ~350k tokens and at 0.90 past
~800k, as the allocator synchronised and re-requested device memory layer by layer. At
0.88 a context ladder up to 1M showed no cliff, and a cold 970k-token prefill showed no
allocator stalls in its stack samples: 119 steps of 8,192 tokens, about 2 s each and slightly
longer as the context grew (381 s in total, see Caveats). Decode uses 0.88 as well: at 0.93 one decode node ran out
of temporary memory at 30 concurrent 128k-token requests.
Request path
MaaS → KServe ingress gateway → InferencePool → hermes-router EPP (Endpoint Picker: picks one
prefill node and one decode node) → P→D proxy → prefill (max_tokens=1) → decode, streamed back.
- Decode is picked by load: the lowest of 2 × in-flight + 1 × KV usage + 1 × queue, with in-flight and queue divided by their maximum over the four nodes and KV usage taken as is (0–1). In-flight is booked by request length (characters; there is no tokenizer in the EPP) when the EPP routes a request and released when its response stream ends (or after 5 minutes without a response chunk), so a decode node that is still waiting for a long prefill is already counted. Prefill is picked at random.
- Picking prefill by load was tried and reverted after a synthetic sweep of three 1M sessions plus 128k-token requests: because in-flight is booked by request length, one 1M request outweighs about nine 128k-token requests, and those piled onto the nodes without a 1M request.
- The proxy is deployed by the manifest, not by the InferNex-Bridge (the controller that
turns the
LLMInferenceServiceinto the running components): the Bridge-managed proxy carries nopdRolelabel, so the EPP cannot address it. Two start-up patches fix the proxy itself (see the comments in the manifest).
Deploy
After the first request, the proxy log should show Using specified prefiller (the EPP's
routing headers arrived). A full chat request in a prefill engine's log means the gateway
bypassed the proxy.
Known issue: publish this service on MaaS by hand.
The console flow that publishes an inference service on MaaS points MaaS at
<llmisvc>-kserve-workload-svc, which skips the EPP, so no prefill or decode node is picked.
Pointing MaaS at the InferencePool directly does not work either: request bodies above 4 MB
fail with HTTP 500 (the EPP's gRPC message limit), and 1M-token bodies are 2–4 MB.
Until this is fixed, glm-5.2-w4a8c8-maas-route.yaml registers the service as an external
model whose endpoint is its route on the KServe ingress gateway; that gateway streams the body
to the EPP, so the 4 MB limit does not apply. On this path MaaS shows the model as ready
whenever the AIServiceBackend exists, whatever the state of the engines.
On InferNex Bridge v26.6.0, run only one mooncake-enabled LLMInferenceService per namespace:
the Bridge keeps a single mooncake-master-service / redis-service pair per namespace and
points it at whichever service reconciled last.
The published manifest differs from the one that was benchmarked only outside the engine flags:
Benchmark method
aiperf 0.12.0, replaying real coding-agent sessions with
timingMode: agentic_replay. Load is set by the number of concurrent sessions: each
session sends its next request after the gap recorded in the trace (the agent's tool time),
and a finished session is replaced by the next one from the pool. Nothing else ran on the
service during the sweep.
Pass criterion (set before the run). Every figure in the result tables is measured over the second half of each rung — its timed requests sorted by start time, from the first start in that half to the last end. A rung passes when, in that window, the TTFT P90 is under 10 s and the window is stable (median inter-token latency of its two halves within 25 %). The result is the highest passing rung, without interpolation.
TTFT is the time to the first streamed token. GLM-5.2 reasons before answering, so that token may be reasoning; first answer token is the first non-reasoning token. ISL / OSL are each request's input / output tokens (OSL includes reasoning). Output TPS is the output tokens of the window's requests divided by the window length; total TPS adds their input tokens, the definition used for TPS on the overview page.
Benchmark results
Verdict per rung (measured window)
Inputs, outputs and throughput (measured window)
Achieved sessions is the time-averaged number of sessions with a request in flight; it is below the nominal count because a replayed session waits out its recorded tool time between turns. Per-session output is 1,000 / average ITL, the rate one user sees while a response streams. It is not output TPS divided by achieved sessions: a request spends part of its time in flight before the first token (queueing and prefill), when it outputs nothing. Over the whole sweep 12,639 timed requests were sent and 10 failed (0.08 %). With MTP, decode emitted 3.7–3.8 tokens per step on average over each whole rung (5 drafted, 55–57 % of drafts accepted).
Where the 128-session rung saturates
Engine counters over each measured window:
Decode running per node is the range of the four nodes' averages, with the highest peak on any one node in brackets.
- Decode is not the limit. The four decode nodes carried even loads (at most 21
requests on one node against its
max-num-seqsof 32), KV peaked at 16 % and nothing was preempted. - Requests queue at prefill, 83 % of the time at 128 sessions, yet while requests were
queued each prefill node was computing only about 4 (its
max-num-seqsis 8) with 13–14 % of its KV in use. Why they queue was not located — no operator-level profiling was run. Candidates include the 8,192-token step budget, the time to load cached prefixes from the KV pool, and the four prefill nodes stepping together (with experts spread over all 32 cards, every node runs each step at the same time, so the slowest sets the pace). - Doubling sessions from 64 to 128 added 1 % completed requests per second (3.82 → 3.87; output TPS rose 17 % because responses were longer, OSL 134 → 154): by 128 sessions the extra load had turned into prefill queueing.
Caveats
- These sessions are shorter than long-context agent sessions of 128k tokens and more. Inputs reached 86k tokens in the measured windows (the corpus maximum is 121k). Longer sessions make both prefill and decode heavier; do not carry the 64-session figure over to 128k-token sessions.
- The 64-session pass is narrow and measured once. Its TTFT P90 is 0.75 s under the line, and its first answer token P90 (10.6 s) is above it.
- Concurrent 1M requests shift decode routing. In a separate sweep of this replay with three
1M sessions running continuously alongside, coding requests concentrated on the one decode
node without a 1M request: at 64 sessions it averaged 25 running requests and reached its
max-num-seqsof 32 for 9 % of the time, ITL rose from 39 to 55 ms and TTFT P90 from 9.25 to 10.1 s — above this page's 10 s line, so 64 sessions would not pass with that load. The likely cause (inferred, not tested by changing the routing) is that in-flight is booked by request length, so a node serving a 1M request looks about 30 coding requests busy. Prefill queueing at 128 sessions was about the same with and without the 1M load (27.8 vs 28.3 queued on average, 83 % vs 89 % of the time). - A 1M request whose prefix is not cached is slow. A ~970k-token request with nothing cached took 381 s to the first token (measured once, sent straight to a prefill node). One request is computed on a single prefill node; the other three stay mostly idle. With their prefixes cached, 1M requests running alone (three sessions) measured TTFT P50 7.6 s / P90 11.7 s through MaaS. Prefixes stay in the KV pool until it fills up and evicts the least recently read ones; it evicted four times during this sweep.
- fastokens matters for 1M TTFT. vLLM-Ascend v0.23.0 images do not include it; a cached
1M request measured 20.8 s TTFT through MaaS without it and 10.7 s with it, mostly from
tokenization (the whole prompt is tokenized once for prefill and again for decode). The
coding-agent sweep was not measured without it; at its input lengths that would add an
estimated ~0.25 s to TTFT. Community images from
v0.28.0.rc1on pinfastokens==0.2.0inrequirements.txt(vllm-ascend #15351); on those images the pip step can be dropped, but 0.2.0 is not the version validated here. - Accuracy was not evaluated. The vLLM-Ascend v0.23.0 GLM-5.2 guide lists the w4a8c8 weights as experimental with known accuracy issues in P/D-disaggregated deployments, and lists 1M context as validated on Atlas 800 A3 only. This page reports performance; evaluate accuracy for your own workload before production use.
- Errors seen: all 10 failed requests carried reasoning but no answer content within the replayed token limit. Nearly all warm-up requests (one per session slot, sent by aiperf at the start of each rung: 223 of 240) ended the same way; warm-ups are not counted.