Guarded Autonomy for Alert-Driven Remediation: Context-Aware Decisions and Crash-Safe Execution in Multi-Cloud Kubernetes
DOI:
https://doi.org/10.15662/IJEETR.2026.0801026Keywords:
AIOps, autonomous remediation, self-healing systems, Kubernetes, closed-loop control, site reliability engineering, multi-cloud operationsAbstract
Alert-driven remediation can resolve common infrastructure failures faster than an on-call engineer, but production teams hesitate to let automation act unattended. Scripted runbooks act without checking whether an action is safe in the current context, repeat themselves when alerts flap, race with one another, stop at the action without confirming its effect, and leave little record of what they did. This paper presents the execution model of CURE (Cluster Uptime and Resiliency Engine), an alert-driven remediation framework designed by the author for a multi-tenant enterprise SaaS platform on Microsoft Azure, Amazon Web Services, and Google Cloud. The model rests on three ideas. First, every remediation follows an Analyze–Decide–Act–Verify–Record contract, under which an action is complete only when its effect has been verified and recorded. Second, decisions are context-aware but deterministic: recipes read live allocation, capacity, and feasibility signals and choose among explicit branches, including escalation. Third, execution is protected by cooldown windows, per-resource locks, concurrency limits, bounds, and a dry-run mode, with remediation state stored as annotations on the affected Kubernetes object so that coordination survives engine restarts and spans engine replicas. We describe a staged rollout in which six of eleven planned use cases reached production on all three clouds, report lessons from that rollout, and define a methodology for evaluating guarded autonomy in production.
References
[1] Kubernetes Authors, “Kubernetes self-healing,” Kubernetes documentation. [Online]. Available: https://kubernetes.io/docs/concepts/architecture/self-healing/
[2] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds., Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA, USA: O’Reilly Media, 2016.
[3] R. Ewaschuk, “Monitoring distributed systems,” in Site Reliability Engineering: How Google Runs Production Systems, B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds. Sebastopol, CA, USA: O’Reilly Media, 2016.
[4] P. Notaro, J. Cardoso, and M. Gerndt, “A survey of AIOps methods for failure management,” ACM Trans. Intell. Syst. Technol., vol. 12, no. 6, Art. no. 81, 2021.
[5] Y. Dang, Q. Lin, and P. Huang, “AIOps: Real-world challenges and research innovations,” in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng.: Companion Proc. (ICSE-Companion), 2019, pp. 4–5.
[6] Kubernetes Authors, “Annotations,” Kubernetes documentation. [Online]. Available: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
[7] J. O. Kephart and D. M. Chess, “The vision of autonomic computing,” Computer, vol. 36, no. 1, pp. 41–50, 2003, doi: 10.1109/MC.2003.1160055.
[8] J. L. Hellerstein, Y. Diao, S. Parekh, and D. M. Tilbury, Feedback Control of Computing Systems. Hoboken, NJ, USA: Wiley-IEEE Press, 2004.
[9] H. Psaier and S. Dustdar, “A survey on self-healing systems: Approaches and systems,” Computing, vol. 91, no. 1, pp. 43–73, 2011.
[10] E. Tabassi, “Artificial Intelligence Risk Management Framework (AI RMF 1.0),” National Institute of Standards and Technology, Gaithersburg, MD, USA, NIST AI 100-1, 2023, doi: 10.6028/NIST.AI.100-1.
[11] Kubernetes Authors, “Operator pattern,” Kubernetes documentation. [Online]. Available: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
[12] Kubernetes Authors, “Resize CPU and memory resources assigned to containers,” Kubernetes documentation. [Online]. Available: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/
[13] Kubernetes Authors, “Persistent volumes,” Kubernetes documentation. [Online]. Available: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
[14] Kubernetes Authors, “Safely drain a node,” Kubernetes documentation. [Online]. Available: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
[15] StackStorm Contributors, “StackStorm documentation,” StackStorm Project. [Online]. Available: https://docs.stackstorm.com/
[16] Prometheus Authors, “Alerting rules,” Prometheus documentation. [Online]. Available: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
[17] Prometheus Authors, “Alertmanager configuration,” Prometheus documentation. [Online]. Available: https://prometheus.io/docs/alerting/latest/configuration/
[18] OpenTelemetry Authors, “OpenTelemetry documentation,” OpenTelemetry Project. [Online]. Available: https://opentelemetry.io/docs/
[19] Google Cloud, “Use Four Keys metrics like change failure rate to measure your DevOps performance,” Google Cloud Blog, 2020. [Online]. Available: https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance
