How it works

Two detection paths.
One incident.
One decision.

A fast path reads kernel XIDs on the node within seconds. A slow path catches trends through dcgm-exporter and Alertmanager. Both converge on a single incident per target and failure class, so a GPU falling off the bus never opens three tickets.

Fast path
kubeneuron-agent

Watches kernel events on each GPU node. XID observed in seconds.

mTLS + Pod-bound token

Events reported over the agent's existing outbound channel.

kubeneuron-controller

Correlates, deduplicates, applies safety gates, and drives the workflow.

  • One incident per (target, class)
  • Transactional audit on every transition
  • Durable action queue, HA via Postgres
Slow path
dcgm-exporter

ECC counters, row remap state, thermals, NVLink and PCIe health.

vmalert → Alertmanager

Threshold and trend rules, each carrying a runbook URL.

01

See the signal

Every incident carries the evidence that opened it — the raw XID, the counter, the alert, the node.

02

Review the path

The escalation ladder is a versioned GPUPlaybook resource, reviewed in a pull request like any other config.

03

Earn automation

Run in dry-run, compare what it would have done against what you did, then decide which rungs deserve to act on their own.

The escalation ladder

Every rung is a decision
someone can defend.

Seven rungs, climbed one at a time and only as far as the evidence justifies. The tread darkens as the step gets destructive, and the climb stops at rung five until a person says otherwise.

  1. 01Observe

    Count, correlate, threshold.

    Most signals stop here.

  2. 02Restart workload

    Contained fault — evict just the affected pod.

  3. 03GPU reset

    Device-level recovery, optionally deferred until idle.

  4. 04Drain & reset

    Move work off first, then reset the device.

  5. Approval gate
  6. 05Guarded reboot

    Approval-gated. The rung most teams keep human.

  7. 06Driver remediation

    Reinstall or roll back when the fault follows the driver.

  8. 07Hardware escalation

    Quarantine, collect a diagnostic bundle, prepare the RMA evidence.

Proven on real hardware

One injected fault ran the whole ladder
and ended with a real machine destroyed.

On an EKS node carrying an NVIDIA Tesla T4, an injected XID 79 opened an incident, cordoned the node, drained its workloads with PodDisruptionBudgets respected, waited for a named human to approve, terminated the actual EC2 instance through a run-scoped IAM role, let the node group replace it, and closed the incident only after verifying the replacement — with the approver's identity on the record.

  1. AWS EKS · g4dn.xlarge
  2. NVIDIA Tesla T4
  3. XID 79 injected
  4. Workloads drained
  5. Human approved
  6. EC2 instance replaced
  7. Recovery verified

The fault was injected. The recovery was not. What this does not prove: per-device GPU reset, which a virtualised instance cannot perform and the agent refuses on measured evidence rather than pretending; and anything at all on AMD silicon, where detection ships but has never run on an AMD device. Read the release evidence →

Signal catalog

Not every XID deserves
the same reaction.

A row remap failure will never be fixed by a reset — that GPU needs to leave the fleet. A contained ECC error only ruins one workload's context. Encoding that difference is the entire point.

NVIDIA XID codes and the KubeNeuron response for each
XIDMeaningResponseWhy
79 GPU has fallen off the bus drain → reboot The device vanished from PCIe. A reset can't reach it — only a reboot or power cycle can.
94 Contained ECC error workload-restart Contained to one context. Evict that workload, keep the GPU serving everything else.
64 Row remap failed drain → reset → re-check The remap could not be recorded. Drain and reset, then re-check remap state; a recurrence after a clean reset is what escalates to quarantine and RMA.

Three of the classes the catalog knows. Each maps a vendor-native code to a neutral (vendor, code) pair, so a second accelerator vendor describes the same problem in the same vocabulary instead of forking the policy. See the full failure catalog →

Build the safe path first

Run a calmer GPU fleet.

Read the source, run the dry-run ladder, and tell us where the policy model breaks.

Explore KubeNeuron Join the Discord