Left of Boom, Right of Boom: Prevention and Remediation Are One Loop

TL;DR

• The industry has split reliability into two disciplines with two budgets and two toolchains: left of boom (prevent it — policy, testing, shift-left) and right of boom (survive it — alerting, on-call, incident response).
• The split is organizational, not technical. Both sides are asking the same question — which change, applied to this system, produces which failure — and neither side can answer it alone.
• The asymmetry is the interesting part: right of boom is where you learn what is true, left of boom is where that knowledge is worth the most, and almost nothing carries it across. Postmortems produce action items, not policy.
• A closed loop means a verified production failure becomes a pre-production gate automatically. That is not a process improvement; it is the only mechanism by which an autonomous system can safely get more autonomous over time.
• AI-generated infrastructure makes the gap urgent rather than merely wasteful. A generated Kubernetes manifest can be syntactically valid, policy-compliant, CI-green, and still wrong in a way only production reveals.

Two budgets, one question

Borrowed from explosive-ordnance work, "left of boom" means everything you do before the detonation, and "right of boom" everything after. It has become a useful shorthand in security, and it maps cleanly onto reliability.

Left of boom, in practice: admission policies, OPA/Kyverno rules, Terraform plan review, unit and integration tests, staging environments, chaos experiments, load tests, canary analysis. Owned by platform engineering. Justified as prevention.

Right of boom: alerting, paging, on-call rotations, runbooks, incident command, war rooms, postmortems. Owned by SRE and on-call. Justified as resilience.

Two org charts. Two budgets. Two vendor categories. Often two sets of people who talk mostly during an incident review.

But look at what each side is actually trying to answer. Left of boom asks: if I apply this change to this system, will something break? Right of boom asks: something broke — which change did it?

That is the same causal relationship, read in two directions. Prevention is prediction over that relationship; remediation is inference over it. Neither side can build a good model of it alone, because:

• Left of boom has the change but not the system. CI knows the diff precisely. It does not know your production topology — which services share that database, which IAM role is actually attached at runtime, which security group the node group ended up in after last month's migration, which config drifted by hand and was never reconciled.
• Right of boom has the system but has lost the change. During an incident you have the live system in front of you and you are reconstructing, under time pressure, which of the forty things that happened in the last six hours mattered.

Each side holds half the evidence. The split guarantees that neither half is ever complete.

The asymmetry that makes this expensive

Here is the part I find genuinely under-discussed.

Right of boom is where knowledge is created. A production failure is the only unambiguous evidence you will ever get about how your system actually behaves. Not how it was designed to behave — how it behaves. Every incident is a paid experiment, and you paid full price.

Left of boom is where that knowledge has the most leverage. A check that runs on every diff costs almost nothing and prevents the whole class, not the instance.

And the transfer mechanism between them is… a postmortem document with action items.

Which is to say: a human wrote down what happened, a human decided what to change, a human filed a ticket, and the ticket competes for priority against feature work. Sometimes the loop closes. Often it closes for the specific instance — "we added an alert for this" — and not for the class. Frequently the same shape of failure returns in a different service eighteen months later, and someone says didn't we hit this before?

That is not a discipline problem. It is a missing mechanism. We have automated detection and we have automated deployment; we never automated the path from verified cause to permanent gate.

The version that works looks like this:

1. A fault occurs in production and is remediated.
2. The cause is verified, not guessed — a specific, timed, attributable change, confirmed by the fact that reversing or correcting it resolved the symptom.
3. That verified cause is expressed as a structural rule: this class of change, applied to this class of object, in this class of environment, produces this failure.
4. The rule is installed as a pre-production gate: an admission policy, a plan check, a pre-merge check.
5. The next occurrence of that change never reaches production.

Step 2 is the load-bearing one, and it is why this cannot be built on correlation. "These signals moved together" does not yield a rule. A rule requires knowing precisely what changed — which means the loop needs control-plane state and change records, not just telemetry.

AI-generated infrastructure closed the window

You could survive an open loop when humans wrote all the manifests. Human volume was the rate limiter, and a human writing a PodDisruptionBudget had usually seen one before.

That rate limiter is gone. Infrastructure code is now generated — manifests, Helm values, Terraform modules, policies — and generated at a rate no review process was sized for. GitClear's 2026 code-quality research measures the consequence: block duplication is at its highest level on record, 81% above 2023, while genuinely refactored code has fallen to 3.8% of changed lines.

The failure mode of generated infrastructure is specific and worth naming precisely, because it defeats both halves of the current toolchain.

It is not a syntax problem. Generated YAML parses. Schema validation passes. Linters are clean. terraform plan succeeds. This is exactly why "just add more validation" does not help — the output is well-formed. What is wrong is the relationship between the generated object and the rest of your system, and no schema encodes that.

Two distinct classes:

Hallucinated structure. A manifest that references something that does not exist or does not mean what the model assumed: a ConfigMap name that was never created, a service account without the IAM annotation that grants the role it needs, a storage class that exists in one cluster and not another, an API version deprecated two releases ago. Valid YAML. Real object names. Wrong system.

Logic drift. More dangerous, because the object is entirely valid and coherent in isolation. The values are simply inconsistent with the live system's actual behaviour:

• A memory limit set to a plausible-looking 512Mi for a service whose steady-state working set is 700Mi. Deploys fine, passes tests, OOMKills under production traffic with exit code 137.
• An HPA with minReplicas: 2 on a Deployment whose PodDisruptionBudget has minAvailable: 2. Both correct. Together, they deadlock every node drain — and you find out during a cluster upgrade, at the worst possible time.
• A readiness probe on /healthz that returns 200 without touching the database, so Kubernetes reports a pod ready while it cannot serve a single real request.
• A retry policy with three attempts and no jitter, on a dependency that is already the bottleneck — which converts a slow dependency into a retry storm.

Unit tests do not catch these because each object is correct on its own. Observability does not catch them because nothing is broken yet — they are latent, waiting for the specific load, drain or failure that activates them. They live precisely in the gap between the two toolchains.

And critically: every one of these is a rule you could have written, if you had seen it break once. That is the whole argument for closing the loop.

Worked example: the same incident, twice

The most convincing evidence for a closed loop is a failure that recurs. Here is one, with real mechanics.

First occurrence — right of boom

14:20. Routine EKS node-group upgrade. The upgrade drains nodes one at a time.

14:23. The drain stalls on node-ip-10-0-31-88. kubectl drain sits there, evicting nothing.

What the operator sees:

$ kubectl drain node-ip-10-0-31-88 --ignore-daemonsets --delete-emptydir-data
evicting pod payments/payments-api-7d9c4f8b6-x2klm
error when evicting pods/"payments-api-7d9c4f8b6-x2klm" -n "payments":
Cannot evict pod as it would violate the pod's disruption budget.

Repeating, indefinitely. The upgrade window closes with one node cordoned and undrained, and the cluster in a half-upgraded state — the genuinely unpleasant outcome, because now you are neither on the old version nor the new one.

The cause, once found:

$ kubectl get pdb -n payments payments-api
NAME MIN AVAILABLE ALLOWED DISRUPTIONS AGE
payments-api 2 0 41d

$ kubectl get hpa -n payments payments-api
NAME MINPODS MAXPODS REPLICAS
payments-api 2 12 2

minAvailable: 2 with exactly 2 running replicas means allowed disruptions is 0. Off-peak, the HPA scales to its floor of 2. The PDB then forbids evicting either one. The drain can never proceed.

Both objects are individually correct and were individually reviewed. The PDB was added by one person for availability during deploys; the HPA floor was lowered by another to save money off-peak. Neither change is wrong. The combination is a deadlock that only appears during a voluntary disruption, off-peak.

Time to diagnose: the better part of an afternoon, most of it spent looking at the node and the kubelet rather than at a budget object nobody remembered existed.

The postmortem action item: "Raise minReplicas to 3 for payments-api." Correct, specific, and it fixes exactly one service.

The class, not the instance

The generalizable rule is not about payments-api. It is:

For any Deployment with both an HPA and a PDB, if pdb.minAvailable >= hpa.minReplicas,
voluntary disruption is impossible at the HPA floor. Reject or flag.

Also expressible for the percentage form (minAvailable: 50% against an even replica count behaves the same way), and checkable from declared state alone — no telemetry, no load, no staging environment. It is a pure relationship between two objects in the cluster's own API.

Second occurrence — left of boom

Four months later, an engineer adds a PDB to notifications-worker — a service that also has an HPA with minReplicas: 2. Reasonable change, matches the pattern used elsewhere, would pass any review.

With the rule installed as an admission policy, it never merges:

Error from server: admission webhook "reliability-gate.data-agent.co" denied the request:
[DA-PDB-001] PodDisruptionBudget notifications-worker sets minAvailable=2 while
HPA notifications-worker sets minReplicas=2 → allowedDisruptions=0 at the autoscaler
floor. Node drains and cluster upgrades will deadlock.
Evidence: incident INC-2026-0418 (payments-api, 4h12m, half-upgraded node group).
Fix: set pdb.minAvailable=1, or raise hpa.minReplicas to 3.

That message is the loop closing. It contains the class-level rule, the concrete fix, and the receipt — the actual incident that paid for the knowledge. An engineer who has never heard of INC-2026-0418 gets its lesson at the moment it matters, in the place it matters.

The cluster upgrade that would have stalled four months from now simply does not stall, and nobody ever knows.

Why the loop is what makes autonomy safe

There is a second reason to build this that has nothing to do with saving afternoons.

Autonomy has to be earned. Nobody hands a general-purpose model the keys to production, and nobody should. The question is what "earned" can even mean mechanically — and the answer is: a fault class becomes safe to automate once the system has been right about that class, repeatedly, verifiably.

That is the same evidence the prevention side needs. A fault class with a verified cause and a verified fix, confirmed enough times to trust, is simultaneously:

• safe to remediate autonomously (right of boom), and
• well-understood enough to gate before merge (left of boom).

One body of evidence, two uses. Which means the loop is not a nice-to-have on top of an autonomous system — it is the mechanism by which the system's autonomy grows. Break the loop and autonomy has to be granted by human decree, one policy at a time, forever.

This is not a hypothetical risk. Gartner expects that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance gaps identified only after a production incident. An agent that cannot show which fault classes it has earned, and on what evidence, is an agent that gets switched off the first time someone asks.

It also gives you the honest counter-argument to "AI will do this." A model can hypothesize a rule. Only a verified outcome can justify installing one. The value is not in generating rules; it is in having earned them.

Where DataAgent fits

Everything above is architecture-agnostic. What we built:

• PULSE detects. TIDE enriches with topology and drift evidence — which is what makes the PDB↔HPA relationship in the worked example visible at all, since it exists only in declared state. SURGE diagnoses over topology, producing a causal chain rather than a correlation. STEER gates against the Trust Ladder. RAPS executes the verified playbook. WAKE applies learning from verified outcomes.
• WAKE is the loop. It is the layer that takes a verified outcome and improves what the system does next time — the difference between a tool that resolves incidents and a system that stops producing them.
• The Trust Ladder is why autonomy is granted per fault class rather than per product tier. A class earns action rights through verified successes; unproven classes route to a human, with the reason stated.
• The Fault Ladder is the discipline of action: Detect → Enrich → Diagnose → Gate → Act → Learn. A fault enters at the bottom and graduates up.
• Publicly: 80% of common failures are addressable autonomously, 86% bug classification precision, 68% less time on routine infrastructure firefighting.

We describe this as Autonomous System Health — one loop across prevention and remediation, rather than an observability tool on one side and a policy engine on the other.

FAQ

What do left of boom and right of boom mean in reliability?

Left of boom is everything before a failure: admission policies, tests, staging, chaos experiments, canary analysis. Right of boom is everything after: alerting, on-call, incident response, postmortems. The terms come from explosive-ordnance work and are useful here because they make the split visible — and the split is the problem.

Isn't this just shift-left SRE?

Shift-left is half of it, and the half that gets stuck. Shift-left says move reliability work earlier, which is right, but it does not say where the rules come from. In practice teams shift left using rules they already know, which means the rules that matter most — the ones your specific system taught you by breaking — never make it into the gate. The loop supplies shift-left with evidence it cannot generate on its own.

Why don't unit tests catch AI-generated infrastructure problems?

Because the generated object is usually correct in isolation. A memory limit of 512Mi is a valid memory limit; a PDB with minAvailable: 2 is a valid PDB. What is wrong is the relationship to the live system — the actual working set, the autoscaler floor, the real dependency graph. Tests assert properties of the object under test, so a defect that exists only in the relationship between two correct objects has nothing to fail against.

What is logic drift?

An object that is syntactically valid and internally coherent but inconsistent with how the live system actually behaves. A limit below the real working set, a probe that reports ready without exercising the real dependency, a retry policy that turns a slow dependency into a retry storm. It is distinct from a hallucinated manifest, where the object references something that does not exist — logic drift references only real things, in a wrong combination.

Can I close the loop without an autonomous remediation platform?

Partly, and it is worth starting. Take your last twenty incidents, and for each one ask whether the cause can be expressed as a rule over declared state — two objects and a relationship between them. A surprising share can, and those become admission policies or plan checks today. What you cannot do manually is keep it current at the rate incidents occur, which is where automation earns its place.

Does closing the loop mean fewer alerts, or fewer incidents?

Fewer incidents, which is the point. Alert reduction addresses the volume of notifications about a problem class; the loop removes the class. The two are often confused because both make on-call quieter, but only one of them means the failure stopped happening.

Key takeaways

• Prevention and remediation are the same causal question read in two directions. The industry split them by org chart, and each half now holds evidence the other needs.
• Right of boom is where knowledge is created; left of boom is where it pays. The usual transfer mechanism is a postmortem action item, which fixes the instance and misses the class.
• AI-generated infrastructure makes this urgent. Generated manifests fail as hallucinated structure or logic drift — both syntactically valid, both invisible to tests and to observability.
• A closed loop turns a verified cause into a permanent gate, with the incident attached as the receipt.
• The same evidence that lets a fault class be safely automated is the evidence that lets it be gated before merge. That is why the loop is the mechanism of autonomy, not an add-on to it.