Definition
Mitigation is an action that reduces or stops the user-facing impact of an incident without necessarily addressing its cause. Remediation is an action that removes the cause and returns the system to a healthy state. Resolution is the point at which the incident is confirmed closed, and recovery is the restoration of everything the failure affected, including data and backlogs.
Explanation
The four words describe four moments on one timeline. Mitigation comes first and buys time: fail over, scale out, rate-limit, roll back. Remediation follows and fixes: correct the config, patch the bug, replace the node. Resolution is a judgment that the fix held and the incident can close. Recovery can outlast resolution: replaying a queue, backfilling a table, re-running a batch job. Teams that treat a mitigation as the fix close incidents that come back, which shows up as repeat pages for the same fault.
How DataAgent approaches it
DataAgent’s flow separates the two on purpose. It stabilizes first so the impact stops, then runs the deep root cause analysis to fix the cause, so the mitigation does not stay in place of a fix. The definitions follow the company’s post What Is Remediation in Production Systems?
Example
A memory leak in a new release pushes pods into OOMKilled restarts. Mitigation: raise the memory limit so the pods stop dying (impact drops, the leak remains). Remediation: roll back to the previous image, or ship the fix that closes the leak. Resolution: error rate and restart count are back to baseline for an hour, incident closed. Recovery: the 40 minutes of failed background jobs are re-queued.
Related terms
FAQ
- Which one does MTTR measure?
- It depends on the team’s definition. Mean time to resolve measures to resolution. Mean time to recover (the DORA usage) measures to restored service, which is closer to mitigation. Write down which one you track.
- Is a workaround the same as mitigation?
- Yes, in incident language. A workaround is a mitigation that stays in place until remediation lands.
- Can you remediate without mitigating first?
- Yes, when the fix is as fast as the workaround. For most incidents the workaround is faster, so mitigation comes first and remediation follows.