Automating chaos experiments in production
This work addresses the problem of ensuring system reliability for companies operating distributed systems, but it is incremental as it applies existing chaos engineering concepts to a specific production environment.
The paper tackles the challenge of verifying distributed system health during component failures by building an automated chaos experiment platform at Netflix, which generates and executes tests to check system resilience in production.
Distributed systems often face transient errors and localized component degradation and failure. Verifying that the overall system remains healthy in the face of such failures is challenging. At Netflix, we have built a platform for automatically generating and executing chaos experiments, which check how well the production system can handle component failures and slowdowns. This paper describes the platform and our experiences operating it.