Chaos engineering

What it means

Chaos engineering is the practice of running controlled experiments on a system to learn how it behaves when parts fail. A team might switch off a server, slow a network link or cut a dependency, and watch whether the service stays up and whether alerts fire.

The published principles describe it as experimenting on a system to build confidence in its ability to withstand turbulent conditions. It starts with a hypothesis, such as the site will keep working if one database replica is lost, and then tests it.

Why it matters

Systems fail in ways nobody predicted, and the first time you learn how is usually during a real outage. Doing it deliberately, in a controlled way and in working hours, turns a surprise into a lesson. It improves reliability, failover plans and the speed of response.

There is a security angle too. Experiments can reveal whether monitoring notices a failed control, whether a service fails open when its authorisation service is down, and whether recovery matches your disaster recovery plan. It does not replace penetration testing.

How it shows up

You meet it in tools that inject failures into servers, containers or networks, and in scheduled game days where a team rehearses an incident. Good practice defines a steady state, limits the blast radius, starts in a test environment, and has a clear way to stop the experiment.

Mistakes: running experiments on production with no safeguards, no hypothesis so nothing is learned, and no one watching. Begin small, tell people, and write down what you learned. Security-focused versions test whether controls fail safely, which is the same concern as secure error handling.