vLLM Ascend Workers Fail to Open NPU Devices

Problem Description

A vLLM inference service on Ascend NPUs fails during startup. The Pod is scheduled and receives its NPUs, but the serving container exits and restarts repeatedly. The InferenceService status can report ModelLoadFailed with exit code 1.

The container logs contain the following message from the Ascend software stack:

Failed to obtain the console log level. The possible causes are as follows:
1. Different containers share the same device;
2. Failed to open the character device when interacting with the OS kernel;
3. Failed to obtain the number of devices.

vLLM then reports that the worker processes failed and the engine could not start:

Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The worker processes often exit before they log a Python traceback of their own, so the logs may not contain an aclrt error code or an NPU out of memory error.

Root Cause

The vLLM worker processes cannot open the NPU device files. The three causes listed in the message map to the following conditions in a Kubernetes cluster:

MessageCondition in the cluster
Different containers share the same deviceAnother container already holds the NPU, for example a process from a previous container that has not released the NPU yet.
Failed to open the character deviceThe container process has no permission on /dev/davinci*. A common case is a container that runs as UID 0 with all capabilities dropped, while the device files are owned by HwHiAiUser with mode crw-rw----.
Failed to obtain the number of devicesThe NPU devices are not injected into the container.

A container that runs as UID 0 can bypass file permissions only when it has the CAP_DAC_OVERRIDE capability. The Ascend runtime examples set capabilities.drop: [ALL], which removes that capability. UID 0 is then neither the owner of the device files nor a member of their group, and the device files cannot be opened. A container ends up in this state in either of the following configurations:

  • The runtime does not set runAsUser. The container runs as the default user of the image, which is root for many vLLM-ascend images, even though the runtime is not meant to run as root. The non-root runtime example sets runAsUser: 1000 and runAsNonRoot: true to prevent this.
  • The runtime runs as root on purpose, as in root mode, and the InferenceService does not add the device group.

Diagnose the Cause

Run the following checks in order. The node-level checks require root access on the node that runs the Pod, with npu-smi and crictl available.

1. Check Whether Another Process Holds the NPUs

On the node, run:

npu-smi info

The failing container restarts repeatedly, so its own worker processes are usually absent. If the process table lists a process on an NPU that is allocated to the failing Pod, find the Pod that owns the process:

grep -oE 'pod[0-9a-f]{8}[-_][0-9a-f]{4}[-_][0-9a-f]{4}[-_][0-9a-f]{4}[-_][0-9a-f]{12}' /proc/<pid>/cgroup | head -1
# Replace underscores with hyphens in the output to get the Pod UID, then:
kubectl get pod -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid | grep <pod-uid>

If every NPU reports No running processes found, no other container holds the NPUs.

2. Check That the NPUs Are Injected

On the node, inspect the most recent serving container of the failing Pod:

POD_ID=$(crictl pods --namespace <namespace> --name <pod-name> --latest -q)
CONTAINER_ID=$(crictl ps -a --pod "$POD_ID" --name kserve-container -q | head -1)
crictl inspect "$CONTAINER_ID" | grep -A1 'ASCEND_VISIBLE_DEVICES'

ASCEND_VISIBLE_DEVICES must be set and must list as many NPUs as the Pod requests.

3. Rule Out a Memory Cgroup OOM Kill

A worker process that the kernel kills for exceeding the container memory limit also exits without a traceback. On the node, run:

dmesg -T | grep -iE 'oom-kill|killed process' | tail -10

If a python process of the serving container appears at the time of the failure, increase the container memory limit instead.

4. Compare Device File Permissions With the Container Identity

On the node, check the device file ownership and the device group:

ls -l /dev/davinci* /dev/davinci_manager /dev/devmm_svm /dev/hisi_hdc
id HwHiAiUser

Then check the identity and capabilities of the serving container:

crictl inspect "$CONTAINER_ID" | grep -A8 '"user":'
crictl inspect "$CONTAINER_ID" | grep -A3 '"capabilities"'

If you have no node access, read the effective security context from the Pod instead:

kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.spec.securityContext}{"\n"}{.spec.containers[?(@.name=="kserve-container")].securityContext}{"\n"}'

The container cannot open the devices when all of the following apply:

  • The device files grant no access to other users, for example crw-rw---- HwHiAiUser HwHiAiUser.
  • The container UID is not the owner of the device files, and its groups (gid and additionalGids) do not include the device group.
  • The container has no CAP_DAC_OVERRIDE. For example, the container configuration shows drop_capabilities: ["ALL"] and the effective capabilities are empty.

Solution

Fix the Permission Mismatch

When step 4 shows a permission mismatch, choose the fix that matches the runtime.

  • The runtime does not need root: set runAsUser: 1000 and runAsNonRoot: true in the runtime container, as in the non-root example. The change applies to every service that uses the runtime.
  • The runtime runs as root on purpose: add the device group to the InferenceService, as described in Grant Access to Ascend Device Files.

To fix a single service immediately while you correct the runtime, add the device group with a merge patch, which keeps the existing seccompProfile and other security context fields. For example, when HwHiAiUser has group ID 1000:

kubectl patch inferenceservice <inference-service-name> -n <namespace> --type merge \
  -p '{"spec":{"predictor":{"securityContext":{"supplementalGroups":[1000]}}}}'

Release an NPU Held by Another Container

When step 1 shows a process from another Pod, resolve the conflict with the owner of that Pod. When the process belongs to a container that has already exited, wait for the NPU to be released, or stop the process after you confirm that its container no longer exists.

Fix Device Injection

When step 2 shows that ASCEND_VISIBLE_DEVICES is missing or lists the wrong number of NPUs, check that the Pod requests the Ascend resource of the device plugin that runs on the node, that the Pod uses the Ascend runtime class configured by the cluster, and that the device plugin logs show a successful allocation for the Pod.

Verify the Fix

The InferenceService rolls out a new Pod. With the RollingUpdate strategy, the previous Pod remains until the new Pod becomes ready, so inspect the most recently created Pod:

kubectl get pod -n <namespace> -l serving.kserve.io/inferenceservice=<inference-service-name> -o wide

Confirm all of the following:

  • The serving container runs as the UID that owns the device files, or its groups include the device group. Check id in the container, or uid and additionalGids in crictl inspect.
  • The logs no longer contain Failed to obtain the console log level, and vLLM proceeds to load the model weights.
  • The InferenceService reports Ready and serves a test request.