DetectionScattered across three tools
XIDs live in kernel logs, ECC counters live in DCGM, and node death shows up somewhere else entirely. Teams end up correlating them by hand, across tools.
Open source — for Kubernetes GPU fleets
A GPU fleet reliability control plane.
Vendor-neutral by construction: KubeNeuron detects degradation, protects the workloads running on the device, automates safe recovery through a policy ladder with approval gates and a complete audit trail — and measures recovered capacity, so what the fleet got back is a number rather than a feeling.
curl -sfL https://github.com/kubeneuron/kubeneuron/releases/latest/download/install.sh | bash -s -- --version latest
Dry-run by default: the workflow runs end to end and records exactly what it would do, so you see the decision before anything touches a node.
The 3am problem
A GPU dies mid-run.
Now what, exactly?
One uncorrectable GPU memory error can kill hours or days of training work. Today that means a human paging through kernel logs and DCGM dashboards, then guessing whether to reset the device, drain the node, or file an RMA — under time pressure, without a record.
ResponseTribal, and only partly written down
The right answer for XID 64 is not the right answer for XID 94. That knowledge sits in a few engineers' heads and a stale runbook.
AutomationToo dangerous to switch on
Scripts that reboot nodes on a metric threshold are exactly how one bad signal takes out a fleet. So most teams never trust them enough to turn them on.
The rest of the site
Four verbs, four pages.
The product definition is a sentence with four claims in it. Each one has a page that shows the work behind it, including where the work stops.
-
Detects
How it works →
Two detection paths converging on one incident, the escalation ladder, and the signal catalog that decides which failure gets which response.
-
Protects
The safety model →
Dry-run by default, a blast radius you declare, human authority on the rungs that deserve it, and unknown state stopping the action rather than guessing.
-
Recovers
The console →
Every decision visible and every actor named — the approval gate, the audit trail, the fleet map, and the dashboard that ships with it.
-
Measures
Run it yourself →
The whole ladder against a synthetic XID, no cluster and no hardware, in about fifteen minutes.
What recovery gave back
“Resolved” is not
a capacity number.
An incident closing tells you the workflow finished. It does not tell you how much accelerator capacity came back, how long it took, or how often the fleet healed without waking anybody. KubeNeuron measures all three from its own incident store — exact, not sampled, and reproducible from a database snapshot.
366.5GPU-hours recovered, of 412.8 degraded — 88.8%
27 of 31recoveries that finished without a human — 87.1%
12m22smedian time to resolution (p50, n=31; p90 52m0s)
kubeneuronctl report --since 30d
degraded GPU-hours 412.8
recovered GPU-hours 366.5 88.8% of degraded
incidents recovered 31 of 37 83.8%
without a human 27 of 31 87.1% of recovered
MTTR (resolved, n=31) p50 12m22s p90 52m0s- Recovered means resolved. An incident parked for a human keeps accruing degraded time until somebody closes it. Nothing is credited to automation that automation did not finish.
- Degraded, not lost. A degraded GPU may still have been serving. The number is honest about what it counts.
- Dry-run recovers nothing. A pilot still gets the numbers — reported separately, labelled as the simulation it is, so the headline figure cannot flatter a fleet nothing has touched.
Build the safe path first
Run a calmer GPU fleet.
Read the source, run the dry-run ladder, and tell us where the policy model breaks.