IaC‑Driven Rootly Workflows: Real‑Time Change Audits

Managing and auditing infrastructure changes in today's complex tech environments can be a major challenge. Manual processes are often slow, prone to human error, and rarely leave a clear audit trail. When an issue arises, figuring out what changed, who changed it, and why can feel like searching for a needle in a haystack.

The solution is to apply Infrastructure as Code (IaC) principles to your incident response platform. By integrating an incident management platform like Rootly with tools such as Terraform, you can unlock real-time change audits, create automated incidents, and enforce consistent response workflows across your entire organization. This approach involves managing your "playbooks-as-code," which brings the benefits of version control, peer review, and automation directly to your incident response strategies [6].

What are IaC-Driven Incident Response Workflows?

IaC-driven incident response workflows in Rootly involve managing your configurations—including services, severities, teams, and automation workflows—through code stored in a version control system like Git. Instead of clicking through a UI, you define your entire incident management setup in configuration files. This "Response-as-Code" method allows for powerful automation and integration directly within your codebase [7].

The key benefits of this approach include:

• Version Control: You get a complete, transparent history of every change made to your incident management process.
• Peer Review: Changes are proposed, reviewed, and approved through standard pull requests, which improves collaboration and ensures high-quality configurations.
• Consistency: Your Rootly configuration remains standardized across all environments, from development to production.
• Auditability: You have a clear, immutable log of who changed what and when, simplifying compliance and security reviews.

This transforms Rootly into a central orchestration hub for SRE automation, connecting alerts, communication, and remediation activities all in one place. By codifying these processes, teams can reduce manual toil and focus on building more resilient systems.

Getting Started: The Rootly Terraform Provider

The foundation for creating IaC-driven incident response workflows is the official Rootly Terraform provider. This tool allows your teams to declaratively manage Rootly resources just like any other piece of infrastructure.

Configuration and Setup

Getting started is straightforward. The provider is available in the official Terraform Registry, making it easy to integrate into your existing projects. For those who want to dive deeper into the code or contribute, the full source is available on GitHub [1].

The provider is also compatible with OpenTofu, the open-source fork of Terraform, and can be found in its registry [3]. For security, it's a best practice to configure your API key by setting it as a ROOTLY_API_TOKEN environment variable rather than hardcoding it into your files.

Rootly Terraform Module Example (2025)

Here is a practical example of a main.tf file that demonstrates how to define core Rootly resources. This is one of the most common rootly terraform module examples for 2025 that teams use to standardize their setup.

# main.tf

# Configure the Rootly Provider
terraform {
 required_providers {
   rootly = {
     source  = "rootlyhq/rootly"
     version = "~> 3.0"
   }
 }
}

provider "rootly" {}

# Define Incident Severities
resource "rootly_severity" "sev0" {
 name  = "SEV0"
 slug  = "sev0"
 color = "#d93f3d"
}

resource "rootly_severity" "sev1" {
 name  = "SEV1"
 slug  = "sev1"
 color = "#ea8a39"
}

# Define a Service
resource "rootly_service" "elasticsearch_prod" {
 name = "elasticsearch-prod"
 slug = "elasticsearch-prod"
}

# Define a Functionality
resource "rootly_functionality" "logging_in" {
 name = "logging-in"
 slug = "logging-in"
}

# Define a Custom Form Field
resource "rootly_form_field" "regions_affected" {
 kind       = "custom"
 name       = "Regions affected"
 input_type = "multi_select"
 enabled    = true
 required   = false
 trigger_params = {
   incident_created = {}
 }
}

# Define an Incident Workflow
resource "rootly_workflow_incident" "jira_task" {
 name        = "Create Jira Issue on Incident Start"
 description = "Automatically creates a Jira ticket for every new SEV0 or SEV1 incident."
 trigger_params = {
   incident_started = {
     severities = [rootly_severity.sev0.id, rootly_severity.sev1.id]
   }
 }
 task_params = {
   tasks = [
     {
       type = "create_jira_issue"
       name = "Create Jira Issue"
       params = {
         project_id = "PROJ"
         issue_type = "Bug"
         summary    = "Incident {{ incident.title }}"
       }
     }
   ]
 }
}

This code makes your incident response processes repeatable, predictable, and auditable. For more advanced examples and detailed documentation, you can explore the Terraform integration guide.

Importing Existing Configurations with Terraformer

What if your organization already has a Rootly configuration created through the user interface? Manually recreating everything in code would be tedious. This is where terraformer-rootly comes in. It’s a tool designed to import your existing Rootly resources into a Terraform state file, bringing them under IaC management.

The process is simple:

1. Install the Terraformer tool.
2. Set your Rootly API token.
3. Run the import command for the resources you want to manage (e.g., services, teams).
4. Upgrade the state file.

Now, your previously UI-managed resources are fully controlled by code.

Use Case 1: Real-Time Change Audit via IaC Pipelines

One of the most powerful applications of this approach is creating a rootly change audit via iac pipelines. By integrating Rootly into your continuous integration and continuous deployment (CI/CD) pipelines, you can build a complete, real-time audit trail of all infrastructure changes.

Here’s how the workflow looks:

1. A developer merges a pull request to change an infrastructure component (like a database setting) using Terraform.
2. The CI/CD pipeline automatically runs terraform apply to deploy the change.
3. As a final step, the pipeline uses the Rootly Terraform provider to create a low-severity "change event" incident in Rootly.
4. This event automatically logs what was changed, who approved it (via the pull request), and when it was deployed, creating an instant and immutable audit record.

From there, you can leverage Rootly's Workflows to trigger a notification to a specific Slack channel, giving the entire team visibility into production changes as they happen.

Use Case 2: From Infra Drift Detection to Rootly Incidents

Another critical use case is automating the process of turning infra drift detection → rootly incidents. Infrastructure drift occurs when your live environment deviates from the state defined in your code, often due to manual, untracked changes. This drift can introduce security vulnerabilities and reliability risks.

Here’s how to automate the response:

1. A scheduled job runs terraform plan or uses a dedicated drift detection tool to scan your infrastructure.
2. If drift is detected, a script calls the Rootly API to create a new incident.
3. The incident payload automatically includes the drift report, specifying which resources have changed and how they differ from the desired state.
4. A Rootly workflow, triggered by the new incident's creation, pages the on-call engineer and creates a dedicated Slack channel for investigation and remediation.

This level of automation dramatically reduces the Mean Time to Detect (MTTD) and remediate configuration drift. Adopting incident response automation is a key strategy for reducing alert fatigue and improving overall team health in 2025 and beyond [8].

Conclusion: Build a Resilient and Auditable System

Adopting an IaC-driven approach to incident management with Rootly and Terraform offers far more than just convenience. It provides unparalleled auditability, consistency across environments, and the ability to automate critical workflows like change auditing and drift detection.

By codifying your incident response processes, you empower your team to move from reactive firefighting to proactive, automated system management. This shift not only makes your systems more resilient but also frees up valuable engineering time for innovation.

Ready to transform your incident management? Explore how you can automate rollbacks and tagging to build a more robust and auditable system with Rootly.

‍