Qwen3.8-27B (NVFP4)
qwen3_5-architecture dense model — a Gated DeltaNet linear-attention hybrid (48 of 64
layers are linear attention) with a native MTP head, 27B parameters, served as an
NVFP4 checkpoint (~25 GB) with DFlash2 speculative decoding. Validated on a single
NVIDIA RTX PRO 6000 Blackwell with community vLLM v0.31.0, deployed as a plain
KServe LLMInferenceService behind the Alauda AI Platform MaaS gateway. The GPU memory budget
was capped at 84 GB to emulate an RTX PRO 6000D. Context length 128k.
TOC
Model identityValidated hardware × stackModel configurationDeployFunction checksBenchmark resultsModel identity
Validated hardware × stack
The speed numbers were measured on the standard 96 GB RTX PRO 6000 with the memory
budget scaled down to 84 GB (--gpu-memory-utilization=0.805). This reproduces the
KV capacity of an RTX PRO 6000D, not its compute or bandwidth, which are lower —
re-measure on a real 6000D before quoting 6000D speeds. Alauda AI Platform's bundled vLLM (0.23.0)
was not used: the community v0.31.0 image is brought in by an LLMInferenceServiceConfig
that overrides only the container image.
Model configuration
Deploy
The weights come from two ModelCars on Docker Hub: the checkpoint is the service's
model.uri, and an init container copies the draft model out of its own ModelCar into an
emptyDir (an LLMInferenceService takes a single model.uri).
With the weights on a local disk the first start took about 9.5 minutes (weights ~3 min,
then kernel compilation, CUDA-graph capture and FlashInfer tuning); with the ModelCars add
the first image pull (~16.3 GiB compressed for the checkpoint, ~2.8 GiB for the draft
model). The compile cache on qwen38-27b-cache makes later starts skip compilation.
The function checks and benchmarks on this page were run with the same files served from a PVC. The ModelCars carry byte-identical files (the sha256 of every file was checked against the pinned Hugging Face revisions), but this ModelCar manifest — including the init-container copy of the draft model — has not yet been run on the GPU.
- With speculative decoding on, vLLM rejects
min_pandlogit_bias(HTTP 400). - The answer text starts with
"\n\n"after the reasoning parser splits the thinking off; strip leading whitespace in the client. nvidia-smireached 82,013 MiB under full KV load — about 3.1 GiB above the 84 GB plan (vLLM does not budget some GDN scratch buffers, #54775). An RTX PRO 6000D still has ~3.5 GiB left; to stay strictly inside 84 GB, lower--gpu-memory-utilizationto 0.785.
Function checks
All through the MaaS gateway, on the deployed manifest above:
Benchmark results
Workload. The benchmark replays real agent trajectories whose per-call input and output
lengths (ISL / OSL) are close to how Alauda Hyperflux uses this model: it makes 3–4
model calls per question, and the Exgentic v2 subset openai_solo × swebench has the
closest input lengths of the corpus.
The corpus outputs are shorter than Hyperflux's final answers, so the decode side is somewhat lighter than production.
Method. aiperf 0.12.0 (NVIDIA image) through the MaaS gateway, timingMode: agentic_replay (recorded gaps between calls, sessions start at 25–75% of their trajectory,
per-session cache-bust), 15 minutes per tier, token counts as reported by the engine, 0
errors. The number of requests actually in flight varies a lot between sessions, so every
request is grouped by the time-averaged number of requests in flight during its lifetime
(the first 120 s of each tier excluded).
One card processes up to 8 requests at the same time with TTFT p90 ≤ 0.7 s and ~50 tok/s per request (slowest 10% ~35 tok/s). Treat 8 as the ceiling for this workload: above it the 515k-token KV cache fills and cached history is evicted between turns. Quote the result as concurrent requests, not users or sessions.