There is something counterintuitive about paying an engineering team to deliberately bring down systems that are already working. And yet, that is exactly what mature reliability-focused companies do, in a controlled and frequent manner. Rolando Bonaccorsi, a computer engineer with an executive MBA, explains that the logic behind this is simple to state but difficult to accept emotionally: it is better to discover a weakness through a planned experiment than during a real incident, at three in the morning, under maximum pressure.
The practice, known as chaos engineering, emerged within technology companies operating at massive scale that realized traditional tests, performed in controlled environments, simply do not capture the real-world complexity of distributed systems in production. The only reliable way to understand how a system responds to a specific failure is to deliberately trigger that failure and observe the outcome.
Failures Will Happen Anyway; the Question Is When
Servers go down, networks fail, and external dependencies become unavailable; these are inevitable facts in any infrastructure operating at a significant scale, not remote hypotheses. The real question has never been whether a failure will happen, but whether the company will discover its vulnerability through a planned experiment or through an angry customer reporting an outage in production.
In this context, Rolando Bonaccorsi summarizes the issue by pointing out that passively waiting to discover where a system is fragile means accepting that the timing of the discovery will be determined by chance rather than by the company. Chaos experiments restore that control, allowing teams to choose when and how a specific weakness will be exposed, usually during business hours, with a team on standby and the ability to quickly roll back if necessary.
Starting Small Without Breaking Everything at Once
Companies that are new to chaos engineering often make the mistake of starting with experiments that are too ambitious, testing catastrophic scenarios before validating the system’s basic resilience to simpler and more likely failures that can occur in everyday operations.
Rolando Bonaccorsi highlights an important detail: shutting down a single server instance and observing whether the system automatically redirects traffic is a more valuable experiment at the beginning of this journey than simulating the complete failure of an entire data center region, a scenario that is rare compared with isolated failures that occur far more frequently in everyday operations.
Blast Radius: Limiting the Damage Caused by the Experiment
Every chaos experiment must define its blast radius in advance, ensuring that even if something behaves differently than expected, the damage remains contained within a small, reversible scope, with no real risk of compromising the experience of actual customers during the resilience test itself.
Based on this principle, a chaos experiment without a defined blast radius is not engineering; it is recklessness disguised as discipline. As Director of Operations at Vert Analytics, Rolando Bonaccorsi recommends always starting in test environments that mirror production, moving to real production environments only after the team has validated basic hypotheses in lower-risk contexts.
What Does Chaos Engineering Reveal About Company Culture?
An organization’s willingness to deliberately trigger failures indirectly reveals its maturity in dealing with mistakes in general. Companies that punish every failure, even within the context of a planned and authorized experiment, are unlikely to sustain a genuine chaos engineering culture over time.
Finally, Rolando Bonaccorsi emphasizes a specific point: if a team is afraid to deliberately trigger a controlled failure, it is probably also afraid to report real failures as soon as they occur. This pattern undermines the company’s ability to learn quickly from any type of incident, whether planned or unplanned.
Chaos engineering is not about destroying systems for the sake of it, but about replacing blind confidence with concrete evidence of resilience. Systems that have never been tested under real failure conditions remain, until proven otherwise, only theoretically reliable—a distinction that becomes apparent at the worst possible moment, when a genuine incident finally occurs without warning.
