Run Multi-NPU vLLM Ascend Inference as Root
TOC
OverviewWhen to Use Root ModeConfiguration ScopePrerequisitesConfigure the Namespace Security PolicySet the Modelcar UID in AmlClusterCreate a Root-Specific ClusterServingRuntimeCreate the InferenceServiceGrant Access to Ascend Device FilesDeploy and VerifyRestore Non-Root ModeReport the Compatibility IssueRelated DocumentationOverview
Some vLLM-ascend images cannot initialize multi-NPU inference reliably when the Modelcar and serving container run as UID 1000. The failure is commonly seen with tensor parallelism greater than one, for example Qwen3.6-27B with --tensor-parallel-size=2 on two Ascend NPUs.
When testing confirms that the same service starts with the Modelcar and serving container running as root (UID 0), root mode can be used as a compatibility workaround. This guide shows how to configure that mode for a multi-NPU InferenceService.
Root mode is a compatibility workaround for images or driver integrations that do not yet work with UID 1000. It changes the UID used by Modelcar workloads at the cluster level. Use a dedicated runtime, apply the smallest namespace security policy that allows the workload, and restore the non-root configuration when root mode is no longer needed.
This guide does not replace tensor parallel configuration. The service must still request the required number of NPUs and pass a matching --tensor-parallel-size value to vLLM.
When to Use Root Mode
Use root mode only when all of the following conditions apply:
- The service runs on Ascend NPUs and uses multi-NPU inference (
TP > 1). - The selected
vLLM-ascendimage has been tested with the required model and hardware. - The service fails with UID
1000during distributed initialization, while the same workload starts with UID0, and the failure remains after you provide a passwd entry for UID1000. - The service owner accepts the cluster-level effect of changing the Modelcar UID.
Typical symptoms include:
These messages indicate a distributed initialization or device-access failure. They do not prove that root is the correct long-term fix. Record the runtime image, driver, CANN, torch_npu, vLLM, and vLLM Ascend versions so the issue can be reported to the image or device integration owner.
Configuration Scope
Root mode requires aligned settings in the following places:
uidModelcar is a cluster-level setting. It affects Modelcar workloads in the cluster, not only the inference service described in this guide. Plan the change with the owners of other Modelcar workloads.
Prerequisites
- A namespace where you can create or update an
InferenceService. - An Ascend node with the number of NPUs required by the model. For TP=2, two NPUs must be visible to the same Pod.
- A
vLLM-ascendruntime image that supports the target model and Ascend software stack. - An existing
ClusterServingRuntimethat you can copy into a root-specific runtime. - Cluster administrator access to update
AmlClusterand the namespace Pod Security Admission settings.
Configure the Namespace Security Policy
Set the namespace's Pod Security Admission Enforce level to baseline before creating the root-mode service. In the platform console, open Project > Namespace, select the target namespace, and update its Pod Security Admission settings.
Use privileged only when the generated Pod contains a requirement that baseline rejects, such as host namespaces or other explicitly privileged settings. Running as UID 0 alone does not require setting the namespace to privileged.
The Audit and Warn levels do not decide whether the Pod is admitted. Check the Enforce level when a Pod is rejected by Pod Security Admission.
Set the Modelcar UID in AmlCluster
Update the AmlCluster resource so KServe Modelcar uses UID 0:
Apply the change to the existing default AmlCluster using your cluster's
normal administration workflow. Do not create a second AmlCluster. Verify
that the controller has reconciled the new value before rolling out the
inference service:
Do not change this value casually on a shared cluster. The platform default is UID 1000; changing it to 0 also changes newly generated Modelcar workloads for other services.
Create a Root-Specific ClusterServingRuntime
Copy the existing Ascend runtime and give the copy a distinct name. Keep the model path handling, Ascend toolkit initialization, shared-memory volume, probes, and image version from the validated runtime. Merge the following fragment into the complete copied runtime; it is not a standalone manifest. Preserve supportedModelFormats, protocolVersions, and the other runtime fields. For a starting configuration, see the custom Ascend runtime example, and choose an image that supports your model. Change the container security context as follows:
When adapting an existing non-root runtime:
- Remove
runAsNonRoot: true, or set it tofalse. - Replace
runAsUser: 1000withrunAsUser: 0. - Container-level security context fields take precedence over the corresponding Pod-level fields. If a container omits
runAsNonRoot, it inherits the Pod-level setting. Inspect every generated container, including Modelcar, for an inherited non-root restriction that conflicts with UID0. - Keep
HOMEpointed to a writable path such as/tmpwhen the image writes caches or temporary files. - Do not remove
seccompProfile, dropped capabilities, or other restrictions unless the validated image requires them and the namespace policy permits the change.
Keep the runtime command's "$@" forwarding so service-specific vLLM arguments
reach the server. Set the TP value in the InferenceService model.args field
shown below. Do not define conflicting tensor parallel values in both the
runtime command and the service.
Create the InferenceService
Select the root-specific runtime and request two NPUs for a TP=2 deployment. The exact Ascend resource key depends on the cluster:
Root mode does not need fsGroup for device access. Keep fsGroup only when the storage backend or another workload requirement needs it. Whether the service needs supplementalGroups depends on the ownership of the Ascend device files on the node; see Grant Access to Ascend Device Files.
The --gpu-memory-utilization value is an example. It controls the per-instance device-memory budget for model weights, activations, and KV cache; it does not replace --tensor-parallel-size=2 and does not resolve a UID or HCCL initialization failure.
Grant Access to Ascend Device Files
Running the container as UID 0 does not by itself grant access to the Ascend device files. The root-specific runtime drops all Linux capabilities, including CAP_DAC_OVERRIDE, so UID 0 is subject to the normal owner, group, and mode checks on those files.
On many Ascend nodes, the NPU driver creates the device files with the HwHiAiUser owner and group and mode crw-rw----. Check the ownership on the node that runs the service:
Example output:
When the device files grant no access to other users, as in this example, add the device group ID to the Pod-level supplementalGroups of the InferenceService:
Without the device group, the vLLM worker processes cannot open the NPU devices and exit during initialization. For the log messages and the diagnosis procedure, see vLLM Ascend Workers Fail to Open NPU Devices.
Prefer the device group over adding CAP_DAC_OVERRIDE to the runtime. The capability bypasses permission checks for every file in the container, while the device group grants access only to files owned by that group. Do not remove drop: [ALL] or set privileged: true to work around this failure.
Deploy and Verify
Save the complete merged runtime as vllm-ascend-root-runtime.yaml and the service as qwen3-6-27b.yaml. Replace the image tag, namespace, model URI, and resource key with values validated for your environment, then apply:
Inspect the generated Pod and verify the effective settings:
Confirm all of the following:
- The Pod requests and receives two NPUs on the same node.
- The serving container runs as UID
0(id -ureturns0). - When the node's device files are group-owned,
id -Gin the serving container includes the device group ID. - The generated vLLM arguments contain exactly
--tensor-parallel-size=2. - The logs show successful HCCL/distributed initialization and model loading.
- The
InferenceServicereachesReadyand serves a test request.
If the Pod is rejected before startup, inspect the namespace events for a PSA violation. If the Pod starts but HCCL still fails, collect the first worker error and the complete software-version matrix before changing more security settings.
Restore Non-Root Mode
When the image or vLLM Ascend integration supports UID 1000, restore the safer default:
- Set
uidModelcarback to1000inAmlCluster. - Wait for the Modelcar configuration to reconcile. Switch affected services to the standard non-root
ClusterServingRuntimewithrunAsNonRoot: trueandrunAsUser: 1000, restoring required device and storage group settings from the non-root example. - Roll out the affected services and verify the effective UID of their replacement containers, service readiness, and a test request.
- Restore the namespace PSA level required by your workloads after confirming that replacement Pods comply with it.
- Remove the root-specific runtime after no service uses it.
Report the Compatibility Issue
When reporting a UID 1000 failure, include the model, TP size, NPU model and count, runtime image digest, CANN and driver versions, vLLM and vLLM Ascend versions, effective Pod security context, and the first HCCL/DIST error. This information distinguishes a device-permission problem from a model, memory, or distributed-topology problem.