Is AI Making Your SRE Team Better? Ask the Scale, Not the Mirror

Nearly every engineer will tell you AI made them more productive. Almost none can tell you what it did to their change failure rate. Those are two different instruments, and only one of them is attached to production.

The short answer is this. AI is making engineers faster at producing changes, and at the same time it is making production less stable. What decides which of those two effects dominates in your organization is the quality of the platform underneath, and that is something you own and can measure this quarter.

Is AI making your SRE team better? The mirror says yes. The scale has not agreed yet.

Key takeaways

• AI adoption improves software delivery throughput but has a negative relationship with delivery stability, per DORA's 2025 State of DevOps report.
• Internal platform quality decides the outcome. Strong platforms turn AI into a compounding advantage, weak ones turn it into faster failure.
• Frontier AI models score below 50% on real-world Kubernetes incident diagnosis (ITBench-AA), so unsupervised AI operations is not yet safe.
• Track change failure rate by AI authorship, not just adoption, to get an honest read on whether AI is helping your team.

94% of technology leaders say AI-generated code is higher quality than code a human wrote, judged at review time. 78% say they get more production incidents once that code ships. Both numbers come from the same survey of the same people: New Relic's 2026 State of AI Coding.

That is not a contradiction. It is two instruments disagreeing, and it matters enormously which one you have been reading.

The mirror is a real instrument, just pointed at the wrong thing

DORA's 2025 State of DevOps report found 90% of respondents using AI at work, and more than 80% saying it made them more productive. That is the mirror, and it is not worthless.

It measures friction, and friction is real: waiting on a queue, fighting a syntax you use twice a year, writing the fourth near-identical Terraform module this quarter.

What the mirror cannot see is what happened to the system afterwards. It reports how the work felt to the person doing it, not what the work did once it was live.

Every engineer has a mirror. Very few teams have bought a scale.

What the scale says: more throughput, less stability

The same DORA report puts the numbers side by side. AI adoption now has a positive relationship with software delivery throughput, which is genuine progress and new this year. It continues to have a negative relationship with delivery stability. More change is arriving, and more of it is breaking.

The operational detail comes from New Relic's survey. 82% of respondents had at least one production failure tied to AI-generated code in the previous six months. 74% say at least a quarter of AI-written code needs significant rework. And 86% say senior engineers now spend more time fixing code than they used to.

There is a second number that explains the first. 62% of technology leaders say their teams trust AI-generated code enough to ship it without line-by-line review.

That is not carelessness, or at least not only carelessness. Verification is the one part of the pipeline that did not get faster, so as generation speeds up, review is the step that gets skipped.

The result is a queue that moves quickly right up to production and then stops being a queue at all, because the checking that used to happen before the deploy now happens after it, as an incident.

That last number is the one to sit with. The work did not disappear. It moved, from writing to verifying, and from whoever wrote it to whoever is on call.

If your seniors feel busier while your dashboards say the team ships more, nothing is wrong with either observation. You are looking at a transfer, not a saving.

Producing a change and operating one are different skills

It helps to split the question in two, because "is AI making us better" hides two very different answers.

At producing changes, the answer is clearly yes. That is what the mirror is reporting, and it is not an illusion.

At operating what got produced, there is no comparable evidence, and some against. ITBench-AA, built by IBM and Artificial Analysis from real enterprise IT tasks, asks models to diagnose live Kubernetes incidents by reading logs, following dependencies, and naming what broke. Every frontier model scores below 50%. The best sits at 47%.

A coin flip on the diagnosis half of the job is not an argument against using AI in operations. It is an argument against leaving it unsupervised there.

A 3 a.m. incident: an AI-written change that passed every review and still failed

A service gets a new capability. It needs to call one more AWS API, and the change is small enough to review in a minute.

It works on the engineer's laptop. It works in the development environment, where outbound traffic is wide open. Tests pass, because the AWS client is mocked.

In the locked-down production account there is no network path to that service, because nobody added one. The call does not fail. It hangs, quietly, until the client deadline expires.

At 3 a.m. the on-call engineer sees requests timing out and a trace that simply stops. The code is correct. The review was correct. The tests were correct.

The model wrote a change that was right in every way it could check, and wrong in the one way the running system knew about. That information was never in the repository. It was in the network.

Platform quality is the variable that flips the sign

Here is the finding that reframes the whole question. DORA measured what platform quality does to AI's effect, and the result is not a small adjustment.

Where internal platform quality is high, AI adoption has a strong, positive effect on organizational performance. Where platform quality is low, the same AI adoption does close to nothing.

AI does not have an effect on your organization. It has an effect through your platform.

Which means "is AI making my team better" is not really a question about AI. It is a question about whether the thing between your engineers and production can absorb a step change in how much change arrives.

If it can, AI compounds. If it cannot, AI is a faster way to reach the same ceiling, and the ceiling is where your incidents live.

It is worth being concrete about what "high platform quality" means, because it is easy to hear it as a compliment rather than a specification. In practice it is four things.

• A paved path, so the safe way to make a change is also the quickest.
• A gate that answers one question before anything runs: is this allowed here, now, by this actor.
• Blast-radius limits, so the worst case was decided in advance rather than discovered during the incident.
• Verification with evidence, so you can show afterwards what happened and why.

None of those are AI features. They are the things that decide whether a step change in change volume lands as throughput or as incidents, and every one of them was worth building before any of this arrived.

Four numbers worth more than the survey

If you want the scale rather than the mirror, these are the readings that answer the question honestly.

1. Change failure rate, segmented by whether the change was AI-authored. Most teams already collect the first half and have never cut it by the second, which is the only cut that answers the question being asked.
2. Rework rate within two weeks. Not "did it deploy" but "did it stay." A change that ships and gets rewritten twice is not throughput.
3. Time to detect and time to recover, tracked separately. Detection is where AI-shaped failures hurt most, because the code looks correct and the failure arrives somewhere the diff never mentioned.
4. The share of incidents whose fix is a new guard rather than a new patch. This is the only leading indicator in the set. The other three tell you how much it hurt last month; this one tells you whether next month's version of the same failure has anywhere to land.

That fourth number is unusual and worth adding. An organization patching every incident individually is running to stay still, and the number stays near zero right up until someone notices that the same class of outage keeps arriving in new clothes.

One objection lands immediately: nobody can tag AI authorship cleanly, because the work is mixed. A prompt writes the first draft, a person edits it, and the commit belongs to both.

That is true, and it is not a reason to skip the cut. Approximate it. Tag at the pull request rather than the line, accept that the boundary is fuzzy, and compare the two buckets over a quarter rather than a week.

A rough split that moves is worth more than a precise number that answers a different question. Self-report answers a different question than production does, and only one of them can widen anyone's autonomy.

The harness turns "is AI helping" into something you can measure

This is where the answer stops being a survey result and starts being architecture.

AI engineering has a name for the layer that makes an agent trustworthy: the harness. It is the tools the agent may call, the context it gets, the state it keeps, the guardrails it cannot argue past, and the verification that proves the work landed.

Coding agents proved the pattern first. The model is the brain, the harness is the body, and the model is replaceable while the harness is the moat. (For the longer argument, see Autonomy Is a Body Problem, Not a Data Problem.)

When we build an agent that acts on production, almost all the work is harness. RAPS, our library of pre-approved fix procedures, gives the agent a paved path a machine can walk rather than a blank prompt and a cluster.

Autonomy is earned rather than granted, so the agent proposes first and widens its reach only where its record says it should. It reads telemetry where the telemetry already lives, inside the cluster, instead of copying production data somewhere else to be analysed. Every action leaves evidence.

That last part is what turns the question from an opinion into a measurement. An agent whose actions are all recorded, verified, and reversible produces the scale readings automatically. You do not have to believe it is making things better. You can check.

The same is true of your engineers, incidentally. The harness you build for an agent is a strictly better version of the one you already owed the humans.

The verdict: what to check this quarter

For producing changes, yes, and the mirror is right about that.

For operating them, the honest answer is that it depends on something you own, and it is measurable this quarter. Cut your change failure rate by AI authorship. Watch rework, not just deploys. Count how many incidents end in a new guard.

If those numbers are moving the right way, AI is making your team better and you can prove it. If they are not, no amount of feeling faster will fix it, and shipping faster into a platform that cannot hold it just brings the ceiling closer.

Ninety-four percent of leaders looked at AI-generated code and said it was better than what a human writes. They were looking in the mirror, and the mirror was not lying, exactly. Buy a scale.

Want to see what a real harness around an operations agent looks like, step by step? Our free open-source autonomous SRE training series builds one in the open: detection, diagnosis, gated action, and verification, with the failure cases left in.

FAQ

Does AI-generated code cause more production incidents?

On the current evidence, yes. 78% of technology leaders report more production incidents once AI-generated code ships, and 82% had at least one production failure tied to AI-generated code in the previous six months (New Relic, 2026 State of AI Coding). DORA's 2025 State of DevOps report finds AI adoption has a negative relationship with delivery stability, even though it now has a positive relationship with throughput.

What determines whether AI adoption helps or hurts an engineering org?

Internal platform quality. DORA's 2025 report finds that where internal platform quality is high, AI adoption has a strong positive effect on organizational performance, and where platform quality is low, the same adoption does close to nothing. AI does not act on an organization directly; it acts through the platform between engineers and production.

What is a harness in AI operations?

A harness is the layer around a model that makes an agent trustworthy in production: the tools it may call, the context it receives, the state it keeps, the guardrails it cannot argue past, and the verification that proves the work landed. The model is the brain and the harness is the body, which is why the harness, not the model, is the durable part of the system.

How well do AI models perform at diagnosing production incidents on their own?

Below 50%. On ITBench-AA, a benchmark built by IBM and Artificial Analysis from real enterprise IT tasks, every frontier model scores under 50% at diagnosing live Kubernetes incidents, and the best sits at 47%. That makes unsupervised AI operations unsafe today, while supervised, gated AI operations remains workable.

References

What AI is doing to delivery outcomes

• New Relic, 2026 State of AI Coding, report and press release (94% rate AI code higher at review, 78% report more incidents, 86% report senior staff spending more time fixing code, 74% report at least 25% of AI code needs significant rework, 82% had a production failure tied to AI code in six months, 62% ship without line-by-line verification).
• Google Cloud, Announcing the 2025 DORA Report (90% AI use, over 80% report productivity gains, 30% report little or no trust, positive relationship with throughput, negative with delivery stability).

Why the platform decides the sign

• DORA, Capabilities: Platform engineering (90% use an internal developer platform, 76% have a dedicated platform team, platform quality decides whether AI adoption helps organizational performance).

How models actually do at operations

• Artificial Analysis, ITBench-AA leaderboard.
• IBM and Artificial Analysis, ITBench-AA: Frontier Models Score Below 50%.

About DataAgent

DataAgent, an autonomous observability and remediation platform, runs inside your cluster and closes the loop from detection to gated action to verified evidence. It reads telemetry where the telemetry already lives, acts through a library of pre-approved fix procedures, and earns autonomy one fault class at a time. Learn more at data-agent.co.