Mitigation Actions

mitigation actions allow resolve ai to propose remediation steps during incident investigations — such as silencing a noisy alert or reverting a suspect commit — for you to review and approve before anything is executed human in the loop resolve ai only proposes actions every action requires explicit human approval before execution the ai model never has direct access to your monitoring platform's write apis supported action types alert silencing temporarily mute alerts that match specific labels while an incident is being resolved this reduces noise and lets your team focus on the root cause platforms docid\ zmzph358 uu9bt5bgzgbh , docid\ cnsx1rvszhvqiqnbbmw7n , docid\ fjjburcvbrhmnojo22ehv , docid\ ebdmzvbxo7pnkorlxtrmv risk level low — silences can be revoked at any time, which immediately deletes them from the monitoring platform default duration configured by your resolve team (typically 60 minutes) revert commit coming soon alert threshold adjustment coming soon rollback deploy coming soon security and safety mitigation actions are designed with multiple safety layers to ensure you stay in full control the ai model never executes write operations directly during an investigation, resolve's ai agent evaluates whether a mitigation is warranted and, if so, creates a proposal the proposal is then presented to your team for review only after a human explicitly clicks "approve" does the execution engine carry out the action using your integration credentials this is the same approach used for docid 6zb yhvarfwowrh48sby5 , where resolve proposes prs that require explicit approval before creation key safeguards explicit approval required — every proposal must be approved by a human via the resolve ui or slack before any action is taken there is no auto execution no direct api access — the ai model that generates proposals is completely separated from the execution engine that interacts with your monitoring platforms the model cannot call write apis encrypted credentials — integration write credentials are stored encrypted and only used by the execution engine after human approval full audit trail — every action is recorded who proposed it, who approved or rejected it, when, and what was affected view the complete history in the docid\ yepr14h5q9af55y9amsxf revocable — approved alert silences can be revoked at any time, which deletes them from the monitoring platform immediately auto expiration — proposals that are not acted upon automatically expire based on the proposed silence duration scoped permissions — write permissions are scoped to specific operations (e g , silence management only) and do not grant broader access prerequisites integration permissions if your integrations currently use read only tokens (the default for investigation), you will need to upgrade them with write permissions to enable mitigation actions read only tokens will continue to work for investigation — only mitigation execution requires write access platform what to update required permissions for mitigation grafana service account token add alert provisioning\ write rbac permission (grafana v11+), or use admin role (older versions) docid\ zmzph358 uu9bt5bgzgbh datadog application key scopes add monitors downtime scope docid\ cnsx1rvszhvqiqnbbmw7n alertmanager api token token must have write access to the /api/v2/silences endpoint docid\ fjjburcvbrhmnojo22ehv sumo logic api credentials credentials must have write access to the muting schedules api docid\ ebdmzvbxo7pnkorlxtrmv you only need to update permissions for the platforms where you want to use mitigation actions other integrations can remain read only configuring mitigations once enabled for your organization, an admin can configure which action types are available organization defaults navigate to admin > mitigation activity in the resolve ui in the configuration section, toggle the action types you want to enable alert silencing — enable to allow silence proposals during investigations click save all action types default to disabled — you must explicitly enable each one team overrides you can customize mitigation settings for individual teams to override the organization defaults in the configuration section, expand team overrides select a team from the dropdown toggle the desired action types for that team click save to remove a team override and revert to organization defaults, click delete override on the team's configuration configuration hierarchy team override > organization default > disabled how mitigations work during an investigation alert evaluation — during an alert investigation, resolve evaluates whether a mitigation action is warranted given all of the production context proposal creation — if warranted, a proposal appears in the investigation canvas showing the proposed action (e g , "silence alert x for 60 minutes") the rationale for the proposal any other relevant metadata for the action (e g , risk level, duration, revertability, etc) notification — if the investigation is connected to a slack thread, the proposal is also sent there with interactive approve/reject buttons approving or rejecting in the resolve ui click approve or reject on the proposal card in the investigation canvas in slack click the approve or reject button on the proposal message on approval for alert silencing a silence, downtime, or muting schedule is created in your monitoring platform a "view in platform" link appears in the proposal on rejection the proposal is marked as rejected no action is taken revoking an approved action for alert silencing, you can revoke an approved silence at any time find the approved proposal in the investigation canvas click revoke the silence is immediately deleted from the monitoring platform proposal expiration proposals that are not acted upon automatically expire the expiration window is based on the proposed silence duration — if a silence would no longer be useful by the time it's approved, it expires instead activity dashboard the admin > mitigation activity page provides an overview of all mitigation proposals across your organization overview metrics total suggested — number of proposals created approved — proposals approved by a user rejected — proposals rejected by a user pending — proposals awaiting a decision revoked — approved actions that were subsequently revoked failed — proposals where execution failed after approval expired — proposals that expired before a decision was made approval rate — percentage of proposals that were approved activity log a filterable table showing all proposals with status, action type, user who acted, timestamps link to the original investigation expandable details showing rationale, audit trail, and metadata filters time range last 24 hours, 7 days, 30 days, or 90 days status all, approved, rejected, pending, revoked, failed, expired capability all, or filter by specific action type troubleshooting no mitigation proposals are appearing during investigations verify that the desired action types are enabled in admin > mitigation activity > configuration confirm your integration tokens have the required write permissions (see docid\ yepr14h5q9af55y9amsxf ) check that the integration health check is passing in the integrations page execution fails after approval the integration token likely lacks the required write permissions check the permissions table above and update your token verify the integration health check is still passing proposals are expiring before i can act on them proposals expire based on the proposed silence duration window contact the resolve team if you need to adjust the default duration