New The SREYUN website is live: five in-house products and reference pricing Learn more →

From alert to self-healing: building 24/7 AI on-call

AIOps For: SRE and on-call teams Products: Ruyi OpsBuddy, openaiops

AI groups incoming alerts and analyzes the root cause first, then acts by risk level with approval and self-healing, and verifies the result; on-call engineers only confirm and decide.

The challenge

  • In an alert storm one fault triggers dozens of alerts, and on-call engineers struggle to find the root cause quickly.
  • Night shifts depend on the experience of a few senior engineers, and their troubleshooting is hard to capture.
  • Automation scripts lack approval and verification, so nobody dares let them make real changes.

The solution

openaiops collects metrics, logs and alerts; Ruyi OpsBuddy analyzes and acts:

  • Noise reduction: Alertmanager / webhook alerts are grouped by topology and time window, with the likely root cause and propagation chain.
  • Diagnosis loop: detect → hypothesize → gather evidence → conclude → act (with approval) → verify → learn, with evidence at every step.
  • Graded actions: write operations need approval by risk level (L0–L3), with multi-level approval chains and on-call schedules; results are verified automatically, and failures roll back or become tickets for people.
  • Knowledge capture: feedback scores, knowledge gaps and experience distillation turn frequent troubleshooting paths into draft skills.

The value

  • On-call engineers move from reading alerts one by one to confirming conclusions and approving actions.
  • Investigations and conclusions are backed by evidence and can become tickets, group messages and diagnosis reports in one click.
  • Without an LLM, the offline engine still handles skill search, alert grouping and root cause, inspections and verification.