Gemma-4-31B-it (NVFP4)
gemma4 dense multimodal model, 31B parameters, served text-only as an NVFP4
checkpoint (~23 GB) with Google's official MTP assistant for speculative decoding.
Validated on a single NVIDIA RTX PRO 6000 Blackwell with community vLLM v0.31.0,
deployed as a plain KServe LLMInferenceService behind the Alauda AI Platform MaaS gateway.
The GPU memory budget was capped at 84 GB to emulate an RTX PRO 6000D. Context length
128k, thinking on by default. For the BF16 model on Ascend see
Gemma-4-31B-it.
TOC
Model identityValidated hardware × stackModel configurationDeployFunction checksBenchmark resultsModel identity
Validated hardware × stack
The speed numbers were measured on the standard 96 GB RTX PRO 6000 with the memory
budget scaled down to 84 GB (--gpu-memory-utilization=0.805). This reproduces the
KV capacity of an RTX PRO 6000D, not its compute or bandwidth, which are lower —
re-measure on a real 6000D before quoting 6000D speeds. Alauda AI Platform's bundled vLLM (0.23.0)
was not used: the community v0.31.0 image is brought in by an LLMInferenceServiceConfig
that overrides only the container image.
Model configuration
Deploy
The weights come from two ModelCars on Docker Hub: the checkpoint is the service's
model.uri, and an init container copies the MTP assistant out of its own ModelCar into
an emptyDir (an LLMInferenceService takes a single model.uri).
With the weights on a local disk the first start took about 6–7 minutes (weights ~3 min,
then kernel compilation and CUDA-graph capture); later starts reuse the compile cache on
gemma4-31b-it-cache and took about 5.5 minutes. With the ModelCars add the first image
pull (~19.5 GiB compressed for the checkpoint, ~0.7 GiB compressed for the assistant).
The function checks and benchmarks on this page were run with the same files served from a PVC. The ModelCars carry byte-identical files (the sha256 of every file was checked against the pinned Hugging Face revisions), but this ModelCar manifest — including the init-container copy of the MTP assistant — has not yet been run on the GPU.
- With speculative decoding on, vLLM rejects
min_pandlogit_bias(HTTP 400). - Long prompts are slow to prefill on this card: time to first token is ~7.6 s for a 32k prompt and ~84 s for a 127k prompt (Triton attention). Agent turns that hit the prefix cache prefill only the new part (see the benchmark below).
nvidia-smireached 80,941 MiB under the agent replay — about 2.1 GiB above the 84 GB plan (PyTorch memory that vLLM does not budget). An RTX PRO 6000D still has ~4.5 GiB left.
Function checks
All through the MaaS gateway, on the deployed manifest above:
Benchmark results
Workload. The benchmark replays real agent trajectories whose per-call input and output
lengths (ISL / OSL) are close to how Alauda Hyperflux uses this model: it makes 3–4
model calls per question, and the Exgentic v2 subset openai_solo × swebench has the
closest input lengths of the corpus.
The corpus outputs are shorter than Hyperflux's final answers, so the decode side is somewhat lighter than production.
Method. aiperf 0.12.0 (NVIDIA image) through the MaaS gateway, timingMode: agentic_replay (recorded gaps between calls, sessions start at 25–75% of their trajectory,
per-session cache-bust), 15 minutes per tier, token counts as reported by the engine; one
of 9,043 requests came back empty (HTTP 200), none failed. The number of requests actually
in flight varies a lot between sessions, so every request is grouped by the time-averaged
number of requests in flight during its lifetime (the first 120 s of each tier excluded).
One card processes up to 8 requests at the same time with TTFT p90 ≈ 1.0 s and ~46 tok/s per request (slowest 10% ~31 tok/s). Treat 8 as the ceiling for this workload: above it the ~300k-token KV cache fills and cached history is evicted between turns. Quote the result as concurrent requests, not users or sessions.