Configuration management is an integral part of modern DevOps-based cloud system management. Many critical operations are done by updating configurations to dynamically change system behavior in production. Today, large-scale cloud and Internet services evolve rapidly, with hundreds to thou-sands of configuration changes deployed daily. At Facebook, thousands of configuration changes are committed every day, outpacing the frequency of code changes. It is not surprising to hear that “cloud feels more about configuration management than software engineering". With the high velocity of changes, faulty configurations inevitably have become major causes of system failures and service outages. For example, faulty configurations are reported as the second largest cause of service disruptions in a main Google production service.
Many configuration-induced failures led to catastrophic impacts. In March 2019, a misconfiguration led to Facebook’s largest outage in terms of duration (14 hours); in June 2021, a seemingly-valid configuration change at Fastly triggered an undiscovered software bug and broke the Internet for an hour.We argue that continuous testing is a key missing piece of today’s configuration management practice. Despite the “configuration-as-code” movement, there is no widely-used, systematic con-figuration testing technique and thus configuration changes are not unit-tested—imagining a world where code changes only go through manual review and static analysis, without regression testing.We will introduce the idea of configuration testing, a new testing technique that enables configuration changes to be unit-tested in DevOps-based continuous integration/deployment. The basic idea of configuration testing is connecting system configurations to software tests so that configuration changes can be tested in the context of code affected by the changes. We will introduce a new type of tests, termed Ctests, to fill the critical need of configuration testing. Ctests complement static validation (the de facto protection), analogous to how testing complements static analysis.
Mon 18 OctDisplayed time zone: Central Time (US & Canada) change
10:50 - 12:10
|Analyzing Infrastructure as Code to Prevent Intra-update Sniping Vulnerabilities
Julien Lepiller Yale UniversityMedia Attached
|Continuous Configuration Testing
Tianyin Xu University of Illinois at Urbana-ChampaignMedia Attached
|Intra-update Sniping Vulnerabilities in Smart Contracts
Mark Santolucito Barnard College, Columbia University, USA, Shmuel Berman Columbia University, Brennen Yu Columbia University, USA, Stella LesslerMedia Attached
|Local Expectation Testing for Terraform
|Scuemata: A Framework for Evolvable, Composable Data Schema
Sam Boyer Grafana LabsMedia Attached
|Configuration management: Q&A and discussion