Configuration management is an integral part of modern DevOps-based cloud system management. Many critical operations are done by updating configurations to dynamically change system behavior in production. Today, large-scale cloud and Internet services evolve rapidly, with hundreds to thou-sands of configuration changes deployed daily. At Facebook, thousands of configuration changes are committed every day, outpacing the frequency of code changes. It is not surprising to hear that “cloud feels more about configuration management than software engineering". With the high velocity of changes, faulty configurations inevitably have become major causes of system failures and service outages. For example, faulty configurations are reported as the second largest cause of service disruptions in a main Google production service.

Many configuration-induced failures led to catastrophic impacts. In March 2019, a misconfiguration led to Facebook’s largest outage in terms of duration (14 hours); in June 2021, a seemingly-valid configuration change at Fastly triggered an undiscovered software bug and broke the Internet for an hour.We argue that continuous testing is a key missing piece of today’s configuration management practice. Despite the “configuration-as-code” movement, there is no widely-used, systematic con-figuration testing technique and thus configuration changes are not unit-tested—imagining a world where code changes only go through manual review and static analysis, without regression testing.We will introduce the idea of configuration testing, a new testing technique that enables configuration changes to be unit-tested in DevOps-based continuous integration/deployment. The basic idea of configuration testing is connecting system configurations to software tests so that configuration changes can be tested in the context of code affected by the changes. We will introduce a new type of tests, termed Ctests, to fill the critical need of configuration testing. Ctests complement static validation (the de facto protection), analogous to how testing complements static analysis.