vLLM Ascend Workers Fail to Open NPU Devices
TOC
Problem DescriptionRoot CauseDiagnose the Cause1. Check Whether Another Process Holds the NPUs2. Check That the NPUs Are Injected3. Rule Out a Memory Cgroup OOM Kill4. Compare Device File Permissions With the Container IdentitySolutionFix the Permission MismatchRelease an NPU Held by Another ContainerFix Device InjectionVerify the FixRelated DocumentationProblem Description
A vLLM inference service on Ascend NPUs fails during startup. The Pod is scheduled and receives its NPUs, but the serving container exits and restarts repeatedly. The InferenceService status can report ModelLoadFailed with exit code 1.
The container logs contain the following message from the Ascend software stack:
vLLM then reports that the worker processes failed and the engine could not start:
The worker processes often exit before they log a Python traceback of their own, so the logs may not contain an aclrt error code or an NPU out of memory error.
Root Cause
The vLLM worker processes cannot open the NPU device files. The three causes listed in the message map to the following conditions in a Kubernetes cluster:
A container that runs as UID 0 can bypass file permissions only when it has the CAP_DAC_OVERRIDE capability. The Ascend runtime examples set capabilities.drop: [ALL], which removes that capability. UID 0 is then neither the owner of the device files nor a member of their group, and the device files cannot be opened. A container ends up in this state in either of the following configurations:
- The runtime does not set
runAsUser. The container runs as the default user of the image, which is root for manyvLLM-ascendimages, even though the runtime is not meant to run as root. The non-root runtime example setsrunAsUser: 1000andrunAsNonRoot: trueto prevent this. - The runtime runs as root on purpose, as in root mode, and the
InferenceServicedoes not add the device group.
Diagnose the Cause
Run the following checks in order. The node-level checks require root access on the node that runs the Pod, with npu-smi and crictl available.
1. Check Whether Another Process Holds the NPUs
On the node, run:
The failing container restarts repeatedly, so its own worker processes are usually absent. If the process table lists a process on an NPU that is allocated to the failing Pod, find the Pod that owns the process:
If every NPU reports No running processes found, no other container holds the NPUs.
2. Check That the NPUs Are Injected
On the node, inspect the most recent serving container of the failing Pod:
ASCEND_VISIBLE_DEVICES must be set and must list as many NPUs as the Pod requests.
3. Rule Out a Memory Cgroup OOM Kill
A worker process that the kernel kills for exceeding the container memory limit also exits without a traceback. On the node, run:
If a python process of the serving container appears at the time of the failure, increase the container memory limit instead.
4. Compare Device File Permissions With the Container Identity
On the node, check the device file ownership and the device group:
Then check the identity and capabilities of the serving container:
If you have no node access, read the effective security context from the Pod instead:
The container cannot open the devices when all of the following apply:
- The device files grant no access to other users, for example
crw-rw---- HwHiAiUser HwHiAiUser. - The container UID is not the owner of the device files, and its groups (
gidandadditionalGids) do not include the device group. - The container has no
CAP_DAC_OVERRIDE. For example, the container configuration showsdrop_capabilities: ["ALL"]and the effectivecapabilitiesare empty.
Solution
Fix the Permission Mismatch
When step 4 shows a permission mismatch, choose the fix that matches the runtime.
- The runtime does not need root: set
runAsUser: 1000andrunAsNonRoot: truein the runtime container, as in the non-root example. The change applies to every service that uses the runtime. - The runtime runs as root on purpose: add the device group to the
InferenceService, as described in Grant Access to Ascend Device Files.
To fix a single service immediately while you correct the runtime, add the device group with a merge patch, which keeps the existing seccompProfile and other security context fields. For example, when HwHiAiUser has group ID 1000:
Release an NPU Held by Another Container
When step 1 shows a process from another Pod, resolve the conflict with the owner of that Pod. When the process belongs to a container that has already exited, wait for the NPU to be released, or stop the process after you confirm that its container no longer exists.
Fix Device Injection
When step 2 shows that ASCEND_VISIBLE_DEVICES is missing or lists the wrong number of NPUs, check that the Pod requests the Ascend resource of the device plugin that runs on the node, that the Pod uses the Ascend runtime class configured by the cluster, and that the device plugin logs show a successful allocation for the Pod.
Verify the Fix
The InferenceService rolls out a new Pod. With the RollingUpdate strategy, the previous Pod remains until the new Pod becomes ready, so inspect the most recently created Pod:
Confirm all of the following:
- The serving container runs as the UID that owns the device files, or its groups include the device group. Check
idin the container, oruidandadditionalGidsincrictl inspect. - The logs no longer contain
Failed to obtain the console log level, and vLLM proceeds to load the model weights. - The
InferenceServicereportsReadyand serves a test request.