# OpenSRE — Complete Knowledge Base > OpenSRE is an open-source AI SRE platform that investigates production incidents autonomously. It deploys multiple AI agents orchestrated via LangGraph to gather context from your observability stack, reason about root causes, and produce structured incident reports — the way an experienced SRE would, but faster and around the clock. Unlike stateless AI tools, OpenSRE has episodic memory that learns from every past investigation and a Neo4j knowledge graph that maps your live service topology. GitHub: https://github.com/swapnildahiphale/OpenSRE License: Apache 2.0 Website: https://opensre.in --- ## Documentation ### Introduction OpenSRE is an open-source AI SRE platform that investigates production incidents autonomously. When an alert fires, OpenSRE's AI agents gather context from your observability stack, reason about root causes, and produce a detailed incident report — the way an experienced SRE would, but faster and around the clock. ## Who is OpenSRE for? OpenSRE is built for teams that are tired of manual, repetitive incident investigation: - **Platform engineers and SREs** who spend too much time on routine investigations - **On-call engineers** who need help at 3 AM when cognitive load is highest - **Engineering managers** who want to reduce MTTR and reliance on tribal knowledge - **DevOps teams** building their observability practice ## Key Capabilities ### Autonomous Incident Investigation When an alert fires, OpenSRE's planner agent breaks the investigation into parallel subtasks. Multiple investigation subagents execute simultaneously, each querying different data sources — Prometheus metrics, Kubernetes pod status, application logs, distributed traces. A synthesizer agent combines the findings and a writeup agent produces a structured report. ### Episodic Memory System OpenSRE remembers past investigations. After every incident, it extracts key metadata — root cause, affected services, alert type, severity — and stores it in its episodic memory. When a similar incident occurs, OpenSRE retrieves relevant past episodes and uses them to guide the new investigation. This is how a senior SRE builds intuition over years of on-call experience, replicated in software. ### Knowledge Graph OpenSRE maintains a live graph of your service topology in Neo4j. It knows which services depend on which, tracks recent deployments, and can perform blast radius analysis — given a failing component, which services are affected? This context is automatically provided to investigation agents. ### 50+ Investigation Skills OpenSRE comes with 50+ built-in investigation skills: checking Kubernetes pod health, querying Prometheus for anomalies, analyzing Grafana dashboards, reading Datadog traces, scanning Sentry errors, and more. Skills are loaded on-demand based on the incident context. ### Integrations Works with: Prometheus, Grafana, Datadog, Elastic/ELK, Splunk, Jaeger, New Relic, Sentry, PagerDuty, Slack, GitHub, Confluence, and Kubernetes. ## Architecture at a Glance OpenSRE uses a graph-based agent orchestration system built on LangGraph. Alerts enter via Slack (through the Slack bot) or directly through the web console. The sre-agent processes investigations and streams results via Server-Sent Events. ``` Slack → slack-bot → sre-agent (LangGraph) Web UI ────────────→ │ ┌───┴───┐ │ │ │ Memory Skills KG ``` ## Open Source OpenSRE is released under the Apache 2.0 license. Self-host it in your own infrastructure. Your data, your control. --- ### Quick Start OpenSRE runs as a set of Docker Compose services. You can have a fully functional AI SRE platform running locally in under 5 minutes. ## Prerequisites - **Docker** and **Docker Compose** installed - **An LLM API key** — OpenSRE uses OpenRouter by default, which gives access to Claude, GPT-4, and many other models. You only need `OPENROUTER_API_KEY`. - **Git** to clone the repository ## Installation ### 1. Clone the repository ```bash git clone https://github.com/swapnildahiphale/OpenSRE.git cd OpenSRE ``` ### 2. Configure environment Create a `.env` file in the project root: ```bash OPENROUTER_API_KEY=your-openrouter-api-key-here ``` That's the only required variable to get started. ### 3. Start all services ```bash make dev ``` This starts the following services: | Service | Port | Description | |---------|------|-------------| | PostgreSQL | 5433 | Primary database | | config-service | 8081 | Configuration API | | Neo4j | 7475 (HTTP), 7688 (Bolt) | Knowledge graph | | LiteLLM | 4001 | LLM proxy | | sre-agent | 8001 | Investigation agent | | web-ui | 3002 | Admin console | ### 4. Open the web console Navigate to http://localhost:3002 to access the OpenSRE admin console. ## Trigger Your First Investigation In the web console, you can trigger a test investigation: 1. Click **New Investigation** in the sidebar 2. Enter an alert description, for example: `High error rate on payments-service: 5xx errors spiked to 15% in the last 10 minutes` 3. Click **Investigate** 4. Watch as OpenSRE's agents gather context, investigate in parallel, and produce a report The first investigation may take 2-5 minutes depending on which skills are invoked and your LLM's response time. ## Add Slack Integration (Optional) To receive alerts and investigations via Slack: ```bash # Add to your .env SLACK_BOT_TOKEN=xoxb-your-bot-token SLACK_APP_TOKEN=xapp-your-app-token # Start with Slack bot make dev-slack ``` --- ### Architecture OpenSRE is built on three core systems working together: a LangGraph-orchestrated agent pipeline, an episodic memory system, and a Neo4j knowledge graph. Understanding how these interact explains how OpenSRE investigates incidents. ## System Overview ``` Slack → slack-bot (Bolt/Socket Mode) → sre-agent (LangGraph) Web UI ──────────────────────────────→ │ ┌────┴────┐ │ │ │ Memory Skills KG │ │ PostgreSQL Neo4j config-service ← used by web_ui, slack-bot, sre-agent ``` Two entry points: Slack (via slack-bot) and the web console (via web_ui). Both stream results via Server-Sent Events from sre-agent. ## LangGraph Orchestration The investigation pipeline is a directed graph with these nodes: | Node | Role | |------|------| | `init_context` | Parses the alert, loads episodic memory context | | `planner` | Breaks the investigation into parallel subtasks | | `subagent_executor` | Executes one investigation subtask | | `synthesizer` | Combines findings from all subagents | | `writeup` | Produces the final incident report | | `memory_store` | Stores the episode in episodic memory | The key architectural decision is the **Send() fan-out**: the planner emits multiple `Send("subagent_executor", task)` events that execute in parallel. Each subagent has access to 50+ investigation skills and runs its subtask independently. This parallel execution is what makes OpenSRE fast. **Data flow:** ``` Alert → init_context → planner → [Send() fan-out] ↓ subagent_executor × N (parallel) ↓ synthesizer → writeup → memory_store ``` ## Episodic Memory System After every investigation, OpenSRE stores the episode in its episodic memory. The episodic memory lifecycle: 1. **Investigation completes** — writeup node produces a structured report 2. **LLM extraction** — metadata is extracted: summary, root cause, alert_type, affected services, severity, resolution status 3. **Storage** — episode stored in PostgreSQL via config-service API 4. **Retrieval** — before the next investigation, `init_context` queries episodic memory for similar past episodes using weighted scoring: alert_type (0.5), service (0.3), resolved status (0.2) 5. **Context injection** — relevant past episodes are injected into the planner's context 6. **Strategy generation** — when 2+ episodes share an alert_type, OpenSRE auto-generates reusable investigation strategies This is what makes OpenSRE get better over time. The first time you see a payments-service outage, it takes longer. The tenth time, it has patterns, root causes, and strategies from past episodes. ## Knowledge Graph OpenSRE maintains a live service topology graph in Neo4j: - **Nodes**: services, deployments, infrastructure components, teams - **Edges**: depends-on, owned-by, calls, deployed-to During investigation, agents can query the graph: - **Blast radius analysis**: given a failing service, what services depend on it? - **Dependency traversal**: what does this service depend on that might have caused this? - **Ownership lookup**: which team owns this service? - **Recent change detection**: what was deployed recently near this service? ## Skills System The 50+ investigation skills are loaded on-demand. When an agent needs to check Kubernetes pod status, it calls `load_skill("k8s-debug")` which loads the skill's context and tools, then calls `run_script` to execute specific checks. This progressive loading keeps the agent's context window manageable. Skills are organized by domain: - Kubernetes: pod status, deployment health, resource usage - Metrics: Prometheus queries, Grafana dashboards, Datadog metrics - Logs: Elastic/ELK log analysis, Splunk search - Traces: Jaeger, Datadog APM, Sentry errors - Infrastructure: DNS, networking, cloud provider health ## Service Ports | Service | Host Port | Description | |---------|-----------|-------------| | PostgreSQL | 5433 | Primary database | | config-service | 8081 | Configuration API | | Neo4j HTTP | 7475 | Neo4j browser | | Neo4j Bolt | 7688 | Neo4j driver connection | | LiteLLM | 4001 | LLM proxy | | sre-agent | 8001 | Investigation agent API | | web-ui | 3002 | Admin console | --- ### Configuration OpenSRE uses a hierarchical configuration system managed by the config-service. Configuration flows from org-level defaults down to team-specific overrides — dicts merge, lists replace. ## Environment Variables The minimal required configuration is one env var: | Variable | Required | Description | |----------|----------|-------------| | `OPENROUTER_API_KEY` | Yes | LLM API key via OpenRouter | | `SLACK_BOT_TOKEN` | No | Slack bot token (xoxb-...) | | `SLACK_APP_TOKEN` | No | Slack app token (xapp-...) for Socket Mode | | `NEO4J_URI` | No | Neo4j connection URI (default: bolt://localhost:7688) | | `NEO4J_USERNAME` | No | Neo4j username (default: neo4j) | | `NEO4J_PASSWORD` | No | Neo4j password | | `ADMIN_TOKEN` | No | Admin token for web UI | ## Integration Configuration ### Prometheus ```yaml integrations: prometheus: url: "http://prometheus:9090" enabled: true ``` ### Grafana ```yaml integrations: grafana: url: "http://grafana:3000" api_key: "your-grafana-api-key" enabled: true ``` ### Datadog ```yaml integrations: datadog: api_key: "your-datadog-api-key" app_key: "your-datadog-app-key" enabled: true ``` ### Elastic / ELK ```yaml integrations: elasticsearch: url: "http://elasticsearch:9200" enabled: true ``` ### PagerDuty ```yaml integrations: pagerduty: api_key: "your-pagerduty-api-key" enabled: true ``` ## LLM Configuration OpenSRE routes LLM requests through LiteLLM, which supports any LLM provider: ```yaml # litellm_config.yaml model_list: - model_name: claude-3-5-sonnet litellm_params: model: openrouter/anthropic/claude-3-5-sonnet api_key: os.environ/OPENROUTER_API_KEY ``` ## Configuration Hierarchy Configuration is stored in the config-service and organized hierarchically: ``` org (base defaults) └── team (team-specific overrides) └── agent (per-agent overrides) ``` Dict values are **deep merged** at each level. List values are **replaced** — if a team specifies a list, it replaces the org-level list entirely. ## Skills Configuration Enable or disable specific investigation skills per agent: ```json { "agents": { "my-agent-id": { "skills": { "k8s-debug": true, "datadog-metrics": false } } } } ``` --- ### Investigation Skills OpenSRE's investigation skills are modular capabilities that let AI agents query specific parts of your infrastructure. Each skill encapsulates the tools and context needed to investigate one domain — from checking Kubernetes pod health to querying Prometheus metrics to reading Sentry error traces. ## How Skills Work When an investigation subagent needs to check Kubernetes pod status, it calls `load_skill("k8s-debug")`. This loads the skill's tools and domain context into the agent's working context. The agent then calls `run_script` to execute specific checks within that skill. This **progressive loading** approach keeps the agent's context window manageable — skills are loaded on-demand only when needed, not all at once. ## Skill Categories ### Kubernetes | Skill | What it investigates | |-------|---------------------| | `k8s-debug` | Pod status, restart counts, OOMKilled events | | `k8s-deployments` | Deployment health, rollout status, replica counts | | `k8s-nodes` | Node resource pressure, disk pressure, readiness | | `k8s-resources` | CPU/memory requests vs limits, resource quotas | | `k8s-events` | Cluster events filtered by namespace and severity | ### Metrics and Monitoring | Skill | What it investigates | |-------|---------------------| | `prometheus` | PromQL queries, alert history, metric anomalies | | `grafana` | Dashboard panels, alert states, annotations | | `datadog-metrics` | Metric queries, monitor states, service health | | `new-relic` | APM metrics, error rates, throughput | ### Logs | Skill | What it investigates | |-------|---------------------| | `elastic-logs` | Elasticsearch log queries, error pattern analysis | | `splunk` | Splunk search queries, alert history | | `cloudwatch-logs` | AWS CloudWatch log groups and insights | ### Distributed Tracing and APM | Skill | What it investigates | |-------|---------------------| | `jaeger` | Distributed traces, span analysis, service dependencies | | `datadog-apm` | APM traces, flamegraphs, service dependencies | | `sentry` | Error events, stack traces, release health | ### Alerting and Incident Management | Skill | What it investigates | |-------|---------------------| | `pagerduty` | Recent incidents, alert history, on-call schedule | | `opsgenie` | Alert timeline, team escalations | ### Communication and Documentation | Skill | What it investigates | |-------|---------------------| | `slack` | Recent messages in incident channels, runbook links | | `github` | Recent commits, pull requests, deployment markers | | `confluence` | Runbooks, service documentation, post-mortems | ### Infrastructure | Skill | What it investigates | |-------|---------------------| | `dns` | DNS resolution, TTLs, recent changes | | `networking` | Connectivity checks, latency, packet loss | ## Adding Custom Skills You can add custom investigation skills for your specific stack. Skills are defined as directories in `.claude/skills/` following the skill format: ``` .claude/skills/ my-custom-skill/ SKILL.md # Skill description and tools scripts/ # Executable scripts ``` The `SKILL.md` file describes what the skill does, what tools it provides, and how to use them. The `scripts/` directory contains the executable scripts that the skill invokes. ## Skills Filtering In multi-team deployments, you can enable or disable specific skills per agent via the configuration system: ```json { "agents": { "my-agent-id": { "skills": { "k8s-debug": true, "splunk": false } } } } ``` Disabled skill directories are removed from the agent's working context at session start. --- ### Episodic Memory System OpenSRE's episodic memory is a system that remembers past investigations and uses them to guide future ones. After every investigation, OpenSRE extracts structured metadata from the outcome and stores it. When a similar incident occurs, this stored knowledge is retrieved and injected into the new investigation's context — like a senior SRE recalling what worked last time. ## Why Episodic Memory Matters Without episodic memory, every incident investigation starts from scratch. An AI agent has no knowledge of past outages, no patterns to recognize, no proven approaches to try first. The first investigation of a `payments-service` timeout takes as long as the tenth. With episodic memory, OpenSRE builds institutional knowledge: - Recognizes patterns: "This alert type usually indicates a connection pool exhaustion" - Recalls root causes: "Last time this happened, it was a bad deployment at 14:32" - Applies strategies: "For this class of incident, start with Kubernetes pod restarts, then check Datadog APM" ## The Episodic Memory Lifecycle ### 1. Investigation Completes The writeup node produces a structured incident report. ### 2. LLM Metadata Extraction An LLM extracts structured metadata from the investigation: - **Summary**: 2-3 sentence description of what happened - **Root cause**: The identified root cause - **Alert type**: Category of alert (e.g., `high_error_rate`, `pod_crashloop`, `latency_spike`) - **Affected services**: List of services involved - **Severity**: critical / high / medium / low - **Resolution status**: resolved / unresolved / partial ### 3. Episode Storage The episode is stored in PostgreSQL via the config-service API. All investigations are stored, not just resolved ones — unresolved incidents are valuable for learning too. ### 4. Similarity Search Before a new investigation begins, `init_context` queries episodic memory for similar past episodes using weighted scoring: | Factor | Weight | |--------|--------| | Alert type match | 0.5 | | Service overlap | 0.3 | | Resolution status | 0.2 | ### 5. Context Injection The top matching episodes are formatted and injected into the planner's context. The planner can see: "Last time this alert fired on payments-service, it was a database connection pool issue resolved by restarting the connection pool manager." ### 6. Strategy Generation When 2 or more episodes share the same alert type, OpenSRE automatically generates a reusable investigation strategy. This strategy captures the common investigation path: which skills to run first, what patterns to look for, which services to prioritize. ## What Gets Better Over Time | After N investigations | What improves | |------------------------|--------------| | 1 | Baseline performance | | 2-3 | Similar incidents get context from past episodes | | 5+ | Strategies auto-generate for common alert types | | 10+ | High accuracy pattern recognition for recurring issues | ## Viewing Episodic Memory The web console at http://localhost:3002 includes an episodic memory browser: - Episode list with severity, services, and resolution status filters - Full investigation history per episode - Strategy viewer showing auto-generated strategies per alert type - Dashboard stats: total episodes, resolution rate, average investigation depth --- ### Knowledge Graph OpenSRE's knowledge graph is a Neo4j-powered map of your entire service topology. It stores services, their dependencies, ownership, recent changes, and infrastructure components. During incident investigation, AI agents query this graph to understand the blast radius of a failure, trace dependency chains, and identify what recently changed near the affected services. ## What the Knowledge Graph Stores ### Services Every service in your platform is a node in the graph: - Name, team, language, criticality - API endpoints and communication protocols - Upstream and downstream dependencies - SLOs and error budgets (if configured) ### Dependencies Edges in the graph represent relationships: - `DEPENDS_ON` — service A calls service B - `OWNS` — team X owns service Y - `DEPLOYED_ON` — service runs on this infrastructure component - `USES` — service uses this database or external API ### Infrastructure Infrastructure components as nodes: - Kubernetes clusters, namespaces, deployments - Databases (PostgreSQL, Redis, MySQL) - Message queues (Kafka, RabbitMQ, SQS) - Cloud services (RDS, S3, Lambda) ## How Investigations Use the Knowledge Graph ### Blast Radius Analysis Given a failing service, the knowledge graph answers: "what else is affected?" When `payments-service` starts returning errors, OpenSRE queries the graph for all services that `DEPEND_ON` payments-service, either directly or transitively. This blast radius list is provided to investigation subagents so they can check the health of downstream services proactively. ### Dependency Traversal When investigating a performance degradation, OpenSRE traverses the dependency graph upstream: what does this service depend on? Has anything in that dependency chain changed recently? ### Ownership Lookup The graph knows which team owns which service. This enables OpenSRE to include the right team context in its report: "This incident affects checkout-service (owned by Platform team) and depends on payments-service (owned by Payments team)." ### Recent Change Detection Changes to the graph — new deployments, config changes, infrastructure modifications — are timestamped. Investigation subagents query: "What changed in the vicinity of this service in the last 2 hours?" ## Querying the Graph The knowledge graph is accessible via Cypher queries through OpenSRE's Neo4j integration. Example queries: ```cypher // Find all services that depend on payments-service MATCH (s:Service)-[:DEPENDS_ON*]->(target:Service {name: "payments-service"}) RETURN s.name, s.team // Find services deployed in the last hour MATCH (d:Deployment)-[:DEPLOYED_TO]->(s:Service) WHERE d.deployed_at > datetime() - duration('PT1H') RETURN s.name, d.version, d.deployed_at ``` ## Building Your Service Topology ### Automatic Discovery OpenSRE can auto-discover service topology from: - Kubernetes service mesh (Istio, Linkerd) telemetry - Distributed tracing data (Jaeger, Datadog APM) - API gateway call graphs ### Manual Registration For services not auto-discoverable, register them via the config-service API or the web console's Knowledge Graph editor. ## Viewing the Knowledge Graph The web console includes a knowledge graph visualizer at http://localhost:3002/knowledge-graph. Explore service dependencies, run blast radius queries, and view recent topology changes. --- ### Integrations OpenSRE integrates with your existing observability stack. You don't need to change how you monitor your systems — OpenSRE connects to the tools you already use and queries them during investigations. ## Integration Overview | Category | Supported Tools | |----------|----------------| | Container Orchestration | Kubernetes | | Metrics & Monitoring | Prometheus, Grafana, Datadog, New Relic | | Logging | Elastic/ELK, Splunk, CloudWatch Logs | | Distributed Tracing | Jaeger, Datadog APM | | Error Tracking | Sentry | | Alerting & Incidents | PagerDuty, OpsGenie | | Communication | Slack | | Source Control | GitHub | | Documentation | Confluence | | LLM Providers | Any provider via LiteLLM | ## Monitoring Integrations ### Prometheus OpenSRE can run PromQL queries against your Prometheus instance to check metric anomalies, query alert history, and correlate metrics with incident timelines. ```yaml integrations: prometheus: url: "http://prometheus:9090" enabled: true ``` ### Grafana OpenSRE can read Grafana dashboard panels, check alert states, and pull annotations that mark deployments or other events. ```yaml integrations: grafana: url: "http://grafana:3000" api_key: "your-grafana-api-key" enabled: true ``` ### Datadog Full Datadog integration covers metrics, APM traces, monitors, and service health. ```yaml integrations: datadog: api_key: "your-datadog-api-key" app_key: "your-datadog-app-key" enabled: true ``` ### New Relic Query New Relic APM data, NRQL metrics, and synthetic monitor results. ```yaml integrations: new_relic: api_key: "your-new-relic-api-key" account_id: "your-account-id" enabled: true ``` ## Logging Integrations ### Elastic / ELK Stack OpenSRE queries Elasticsearch for log patterns, error spikes, and service-specific log analysis. ```yaml integrations: elasticsearch: url: "http://elasticsearch:9200" enabled: true ``` ### Splunk Full Splunk SPL query support for log investigation. ```yaml integrations: splunk: url: "https://your-splunk-instance:8089" token: "your-splunk-token" enabled: true ``` ## Tracing Integrations ### Jaeger OpenSRE can look up distributed traces in Jaeger by service, operation, or trace ID, and analyze span timing and error propagation. ### Sentry OpenSRE queries Sentry for recent error events, stack traces, and release health metrics correlated with the incident timeline. ```yaml integrations: sentry: dsn: "https://your-sentry-dsn" token: "your-sentry-token" enabled: true ``` ## Alerting Integrations ### PagerDuty OpenSRE can query PagerDuty for recent incident history, on-call schedules, and alert timelines — providing context about whether this is a recurring alert. ### Slack Slack integration serves two roles: 1. **Entry point**: Receive alerts and trigger investigations via Slack messages 2. **Investigation context**: Query recent messages in incident channels for context ## LLM Provider Integration OpenSRE routes all LLM requests through LiteLLM, which acts as a unified proxy. This means you can use any LLM provider: - **Anthropic**: Claude 3.5 Sonnet, Claude 3 Opus - **OpenAI**: GPT-4o, GPT-4 Turbo - **Open source models**: Via OpenRouter (Llama 3, Mistral, etc.) - **Self-hosted models**: Ollama, vLLM - **Any provider**: LiteLLM supports 100+ providers Change the model without changing any OpenSRE configuration — only the `litellm_config.yaml` needs updating. ## Adding Custom Integrations For tools not listed here, you can add custom integrations as investigation skills. See the Investigation Skills documentation for how to create a custom skill that queries your specific tool. --- ## Blog Posts ### What is OpenSRE? OpenSRE is an open-source AI platform that investigates production incidents the way an experienced SRE would — but faster and around the clock. ## The Problem When a production incident hits, engineers scramble to figure out what went wrong. They check dashboards, grep through logs, trace requests, and piece together a timeline. This process is slow, stressful, and depends heavily on tribal knowledge. ## How OpenSRE Helps OpenSRE automates this investigation process. When an alert fires, OpenSRE's AI agents: - **Gather context** from your monitoring tools — Prometheus, Grafana, Datadog, Elastic, and more - **Investigate systematically** using 50+ built-in investigation skills - **Learn from past incidents** through episodic memory, getting better over time - **Map service dependencies** via a knowledge graph powered by Neo4j - **Produce a detailed report** with root cause analysis, timeline, and remediation steps ## Key Features ### Episodic Memory Unlike stateless AI tools, OpenSRE remembers past investigations. When a similar incident occurs, it recalls what worked before — the same way a senior engineer builds intuition over years of on-call experience. ### Knowledge Graph OpenSRE maintains a live graph of your service topology. It knows which services depend on which, what changed recently, and how failures propagate through your system. ### 50+ Investigation Skills From checking Kubernetes pod status to analyzing Prometheus metrics to reading Sentry error traces — OpenSRE has a growing library of investigation skills that it selects based on the incident context. ## Open Source OpenSRE is fully open-source under the Apache 2.0 license. Self-host it in your own infrastructure. Your data stays with you. --- ### AI-Powered Incident Investigation: How It Works Traditional incident response is reactive. An alert fires, a human gets paged, and the investigation begins from scratch. OpenSRE changes this by deploying AI agents that investigate incidents the moment they're detected. ## The Investigation Pipeline When an alert reaches OpenSRE, here's what happens: ### 1. Context Gathering The system pulls in all available context: the alert payload, recent deployments, service health metrics, and any related incidents from its episodic memory. It already knows the topology of your system through its knowledge graph. ### 2. Planning A planner agent analyzes the context and decides which investigation paths to pursue. It selects from 50+ available investigation skills based on the alert type, affected services, and what's worked in similar situations before. ### 3. Parallel Investigation Multiple investigation subagents fan out to gather evidence simultaneously. One might check Kubernetes pod status while another queries Prometheus metrics and a third reads application logs. This parallel approach dramatically reduces investigation time. ### 4. Synthesis A synthesizer agent collects all findings, correlates the evidence, and identifies the most likely root cause. It considers multiple hypotheses and weighs the evidence for each. ### 5. Report Generation The final output is a structured incident report: a timeline of events, root cause analysis, blast radius assessment, and recommended remediation steps. ## Why Memory Matters The key differentiator is episodic memory. After each investigation, OpenSRE stores the full episode — what was investigated, what was found, and what the resolution was. When a similar incident occurs, the system retrieves relevant episodes and applies those learnings. This is how experienced SREs operate. They don't start from zero each time. They recognize patterns, remember past incidents, and know which diagnostic steps are most likely to be productive. OpenSRE codifies this process. ## Built for Your Stack OpenSRE integrates with the tools you already use: Prometheus, Grafana, Datadog, Elastic, Splunk, PagerDuty, Slack, Kubernetes, and more. It meets your infrastructure where it is. --- ### Getting Started with OpenSRE OpenSRE is designed to be straightforward to set up. This guide walks you through the basics of getting it running in your environment. ## Prerequisites - Docker and Docker Compose - An API key for an LLM provider (OpenRouter, Anthropic, or OpenAI) - Your monitoring stack credentials (Prometheus, Grafana, etc.) ## Quick Setup 1. **Clone the repository** ```bash git clone https://github.com/swapnildahiphale/OpenSRE.git cd OpenSRE ``` 2. **Configure environment** Copy the example environment file and add your API key: ```bash cp .env.example .env # Edit .env and set OPENROUTER_API_KEY ``` 3. **Start the platform** ```bash make dev ``` This starts all services: the SRE agent, web console, Neo4j knowledge graph, PostgreSQL, and LiteLLM proxy. 4. **Open the console** Visit `http://localhost:3002` to access the web console. From here you can trigger investigations, view reports, and configure integrations. ## Connecting Your Tools OpenSRE integrates with your existing monitoring stack. Configure connections to: - **Prometheus** — metrics queries - **Grafana** — dashboard context - **Kubernetes** — cluster state and pod health - **Slack** — receive alerts and investigation reports Configuration is done through the web console or by editing the team configuration file. ## What's Next Once OpenSRE is running, try triggering a test investigation. The platform will walk through its investigation pipeline and produce a report. Each investigation teaches the episodic memory system, making future investigations faster and more accurate. --- ### What is an AI SRE Agent? An AI SRE agent is an autonomous software system that investigates production incidents. It receives an alert, decides what to investigate, queries your monitoring tools, reasons about the evidence, and produces a structured incident report — the way an experienced site reliability engineer would, but without the 3 AM wake-up call. ## What Makes It an "Agent"? Not all AI tools are agents. A chatbot answers questions. An AI SRE agent acts autonomously toward a goal. The key properties of an AI SRE agent: **Goal-directed**: Given an alert, the agent pursues a goal — understand what's happening and why. It doesn't just answer one question and stop. **Tool use**: The agent has access to tools: run a Prometheus query, check Kubernetes pod status, read application logs. It decides which tools to use based on what it's learned so far. **Multi-step reasoning**: Incident investigation requires multiple steps — gather initial context, form hypotheses, gather more targeted evidence, revise hypotheses, reach conclusions. An agent executes this loop autonomously. **Adaptive**: If the first hypothesis doesn't pan out, the agent pivots. If a Kubernetes check returns nothing interesting, it moves to application metrics. Human-like reasoning, not brittle scripts. ## How AI SRE Agents Differ from Runbook Automation Runbook automation (like Shoreline or PagerDuty Automation) executes predefined scripts in response to specific alerts. It works well for known, well-defined incidents with known remediation steps. AI SRE agents handle the unknown: - New types of incidents you haven't seen before - Multi-service failures with complex blast radius - Incidents that don't match any existing runbook - Root causes that require reasoning across multiple data sources The tradeoff: runbook automation is predictable and fast for known incidents. AI agents are more flexible but less deterministic. ## The Investigation Workflow A typical AI SRE investigation in OpenSRE: ### 1. Alert Received An alert arrives via Slack or the web console: "High error rate on payments-service: 5xx errors at 12%, up from baseline 0.2%." ### 2. Context Gathering The `init_context` node: - Retrieves similar past incidents from episodic memory - Looks up payments-service in the knowledge graph (dependencies, team, recent changes) - Identifies affected downstream services via blast radius analysis ### 3. Investigation Planning The `planner` node receives this context and breaks the investigation into parallel subtasks: - "Check Kubernetes pod health for payments-service" - "Query Prometheus for error rate and latency metrics" - "Check Datadog APM traces for the error pattern" - "Look up recent deployments to payments-service" ### 4. Parallel Execution Multiple `subagent_executor` nodes run simultaneously, each handling one subtask. This parallel execution is what makes AI SRE agents fast — 4 investigation threads instead of 1. ### 5. Synthesis The `synthesizer` node receives all findings and produces a coherent narrative: "payments-service error rate spiked at 14:32, correlated with deployment of v2.4.1. APM traces show timeout errors in the database connection layer. Connection pool exhaustion confirmed in Kubernetes pod logs." ### 6. Report The `writeup` node produces a structured incident report with root cause, evidence, blast radius, and suggested remediation. The `memory_store` node saves the episode to episodic memory. ## The Role of LLMs LLMs are the reasoning engine inside an AI SRE agent. They handle: - **Planning**: deciding which tools to use and in what order - **Interpretation**: understanding what metrics and logs mean in context - **Synthesis**: combining evidence from multiple sources into a coherent conclusion - **Communication**: writing clear, actionable incident reports LLMs alone aren't enough — without the tool ecosystem, episodic memory, and knowledge graph, an LLM can only reason about what you tell it directly. OpenSRE combines all these components. ## OpenSRE's Implementation OpenSRE implements AI SRE agents using: - **LangGraph** for orchestrating the multi-agent pipeline - **50+ investigation skills** for querying your specific tools - **Episodic memory** for learning from past investigations - **Neo4j knowledge graph** for service topology context - **LiteLLM** for routing to any LLM provider --- ### What is Episodic Memory in SRE? Episodic memory is a system that allows AI SRE agents to remember and learn from past investigations. Instead of starting each incident investigation from scratch, an AI agent with episodic memory can recall: "Last time this alert fired on this service, the root cause was X, and we resolved it by doing Y." This accumulated institutional knowledge is what makes AI SRE agents get better over time. ## The Problem: Stateless AI Isn't Enough Most AI tools are stateless. They have knowledge from their training data, but they don't remember your specific systems, your specific incidents, or what's worked for you in the past. Imagine a senior SRE on their first day at a new company. They know SRE practices well, but they don't know your systems. Now imagine that same SRE after 2 years on-call. They've seen hundreds of incidents, they recognize patterns, they know the quirks of each service. Episodic memory is how we give that accumulated expertise to an AI agent. ## How Episodic Memory Works in OpenSRE ### Step 1: Investigation Completes After every incident investigation, the writeup node produces a structured report: what happened, what evidence was found, what the root cause was, what was done to resolve it. ### Step 2: Metadata Extraction An LLM extracts structured metadata from the investigation outcome: - **Summary**: 2-3 sentence description of the incident - **Root cause**: The identified root cause (e.g., "connection pool exhaustion due to sudden traffic spike") - **Alert type**: Category of alert (e.g., `high_error_rate`, `pod_crashloop`, `latency_spike`) - **Affected services**: Which services were involved - **Severity**: critical / high / medium / low - **Resolution status**: Was it resolved? How? ### Step 3: Episode Storage The episode is stored in PostgreSQL via OpenSRE's config-service. All investigations are stored — not just resolved ones. An unresolved incident with "we don't know yet" is still valuable future context. ### Step 4: Similarity Retrieval Before the next investigation begins, OpenSRE queries episodic memory for similar past episodes. Similarity is computed using weighted scoring: | Factor | Weight | Rationale | |--------|--------|-----------| | Alert type match | 0.5 | Same category of alert = most relevant | | Service overlap | 0.3 | Same services involved = highly relevant | | Resolution status | 0.2 | Resolved episodes have proven remediation | ### Step 5: Context Injection The top matching episodes are injected into the investigation planner's context. The planner sees concrete, grounded context from your actual environment: > "Similar past episode: On 2026-02-14, payments-service had a high_error_rate alert. Root cause: database connection pool exhaustion during traffic spike after a marketing campaign. Resolved by increasing pool size from 10 to 50 connections and adding circuit breaker. Duration: 23 minutes." ### Step 6: Strategy Generation When 2 or more similar episodes accumulate, OpenSRE automatically generates a reusable investigation strategy. This strategy captures the common investigation path: start with connection pool metrics, check for recent deployments or traffic spikes, query Datadog for database latency, etc. ## The Difference From Fine-Tuning Episodic memory is not fine-tuning. Fine-tuning bakes knowledge into the model weights — it's expensive, requires ML expertise, and goes stale as your systems evolve. Episodic memory is a retrieval system. Episodes are stored in a database, queried at investigation time, and injected as context. Updating it is as simple as new investigations happening. It stays current automatically. ## What Gets Better Over Time The compound effect of episodic memory: | Investigations | What changes | |----------------|-------------| | 1-5 | Baseline performance, building history | | 5-10 | Relevant past episodes surface for recurring alert types | | 10+ | Auto-generated strategies for common patterns | | 20+ | High-confidence root cause predictions for known patterns | ## Viewing Your Episodic Memory The OpenSRE web console includes an episodic memory browser. View all stored episodes, filter by service or severity, see which strategies have been generated, and review the context being provided to active investigations. --- ### Knowledge Graphs for Incident Response A knowledge graph gives AI SRE agents a map of your entire system during incident investigation. Instead of an AI agent that knows Kubernetes commands but doesn't know your specific services, you have an agent that knows: "orders-service depends on payments-service which depends on the payments database, and orders-service is owned by the Commerce team." This context transforms generic investigation into targeted, system-aware diagnosis. ## What is a Service Knowledge Graph? A service knowledge graph is a graph database (OpenSRE uses Neo4j) that stores your services as nodes and their relationships as edges: - **Services** → what you run - **DEPENDS_ON** → which services call which - **OWNS** → which team owns which service - **DEPLOYED_ON** → where services run (clusters, nodes) - **USES** → which databases, queues, or external APIs a service depends on Unlike a static architecture diagram that goes stale, a knowledge graph is updated continuously — new deployments, new services, new dependencies are reflected in near-real-time. ## How It Changes Incident Investigation ### Without a Knowledge Graph When `payments-service` has errors, an AI agent without service topology knowledge checks: "Is payments-service healthy? Yes/no." It has no way to know that three other services are currently failing because they all depend on payments-service. ### With a Knowledge Graph The AI agent queries: "What services depend on payments-service, directly or transitively?" The graph returns: `orders-service`, `checkout-service`, `subscription-service`. The agent checks all four simultaneously. The blast radius is understood within seconds of investigation start. ## Blast Radius Analysis Blast radius analysis answers: "If this service fails, what else breaks?" OpenSRE performs blast radius analysis by traversing the dependency graph from the failing node: ```cypher MATCH (s:Service)-[:DEPENDS_ON*1..3]->(failing:Service {name: "payments-service"}) RETURN s.name, s.team, length(path) as hops ORDER BY hops ``` This returns every service that depends on payments-service, up to 3 hops away. Investigation subagents check each of these services proactively, rather than waiting for more alerts to fire. ## Dependency Traversal for Root Cause When investigating a latency spike in `checkout-service`, the graph answers: "What does checkout-service depend on?" — `payments-service`, `inventory-service`, `shipping-service`, and `user-service`. Each of these is a candidate root cause. The knowledge graph focuses the investigation. ## Recent Change Detection One of the most valuable uses of the knowledge graph is change detection. The graph stores deployment events, configuration changes, and infrastructure modifications with timestamps. During investigation: > "What changed near checkout-service in the last 2 hours?" The graph returns: "payments-service was deployed at 14:32 (32 minutes before the incident)." This is often the fastest path to root cause. ## Building Your Knowledge Graph ### Automatic Discovery OpenSRE can populate the knowledge graph automatically from: - **Kubernetes service mesh data** (Istio, Linkerd): actual traffic flows become dependency edges - **Distributed tracing** (Jaeger, Datadog APM): trace data reveals service call patterns - **API gateway logs**: request routing reveals dependencies ### Manual Registration For services that aren't auto-discoverable, use OpenSRE's web console to register services and their dependencies manually. ## Neo4j as the Foundation OpenSRE uses Neo4j for the knowledge graph because: - **Graph queries are natural**: Cypher makes it easy to ask graph questions (blast radius, shortest path, connected components) - **Performance**: Graph traversal in Neo4j is O(depth) not O(nodes) — fast even at thousands of services - **Flexibility**: Schema-free means you can add new node types and relationships as your needs evolve --- ### OpenSRE vs Commercial SRE Tools: An Honest Comparison OpenSRE is an open-source AI SRE platform that investigates production incidents autonomously using episodic memory and a knowledge graph. If you're evaluating AI-powered incident response tools, here's how it compares to PagerDuty AI, Rootly AI, and Shoreline — honestly, including where the commercial tools have advantages. ## Feature Comparison | Capability | OpenSRE | PagerDuty AI | Rootly AI | Shoreline | |-----------|---------|-------------|----------|-----------| | Open Source | Apache 2.0 | No | No | No | | Self-Hosted | Yes | No | No | Partial | | Episodic Memory | Yes | No | No | No | | Knowledge Graph | Yes | No | No | No | | Investigation Skills | 50+ | Limited | Limited | Yes | | LLM Provider Choice | Any (via LiteLLM) | Fixed | Fixed | Fixed | | Slack Integration | Yes | Yes | Yes | Yes | | Web Console | Yes | Yes | Yes | Yes | | API Access | Yes | Yes | Yes | Yes | | Price | Free | $$$/month | $$$/month | $$$/month | ## The Open Source Advantage ### No Vendor Lock-in With PagerDuty AI, Rootly AI, or Shoreline, your incident history, runbooks, and automation are locked into their platform. If you switch tools, you start over. With OpenSRE, everything lives in your infrastructure — your database, your Neo4j instance, your Git repository. You own your data. ### Self-Hosted Data Sovereignty Enterprise teams with compliance requirements (SOC 2, HIPAA, FedRAMP) often can't send incident data to third-party SaaS platforms. OpenSRE runs entirely in your infrastructure. Your incident data, your episodic memory, and your knowledge graph never leave your environment. ### Any LLM Provider Commercial tools typically lock you into a specific AI provider. OpenSRE routes through LiteLLM, giving you full control: use Anthropic's Claude for quality, switch to open-source models for cost, run a local model for air-gapped environments. ### Customizable Investigation Skills OpenSRE's 50+ investigation skills are extensible. Add a custom skill for your internal monitoring tool, your proprietary deployment system, or your custom alerting pipeline. Commercial tools don't let you extend their investigation logic. ## Where Commercial Tools Have Advantages ### Managed Hosting If you don't want to run Kubernetes, PostgreSQL, and Neo4j yourself, a managed SaaS is simpler. OpenSRE requires infrastructure to run, which means maintenance overhead. ### Enterprise Support and SLAs PagerDuty and Rootly offer 24/7 enterprise support with SLAs. OpenSRE relies on community support and your own team's expertise. For teams without DevOps capacity, this matters. ### Compliance Certifications SOC 2 Type II, ISO 27001, and similar certifications take years to obtain. Commercial tools have them. OpenSRE doesn't — you're responsible for your own compliance posture. ## When to Choose OpenSRE Choose OpenSRE if: - You have infrastructure capacity to self-host - You need data sovereignty or air-gapped operation - You want to extend investigation logic with custom skills - You're cost-conscious (commercial tools can cost $50k-$200k/year for large teams) - You want to contribute to and influence the roadmap Choose commercial tools if: - You need managed hosting with zero ops overhead - You require enterprise SLAs and certified support - Your team lacks infrastructure experience - You need specific compliance certifications out of the box --- ### Reducing MTTR with AI-Powered SRE Mean Time to Resolution (MTTR) is the critical metric for incident response teams. AI SRE agents like OpenSRE can reduce MTTR by 60-80% by automating the investigation phase — gathering context, forming hypotheses, and identifying root causes without waiting for a human engineer to start the investigation. ## The MTTR Problem ### Manual Investigation Takes Too Long A typical production incident follows this timeline: 1. Alert fires → pager wakes someone up (1-5 minutes) 2. Engineer opens dashboards, starts investigating (5-15 minutes) 3. Gathers context from Kubernetes, metrics, logs (15-45 minutes) 4. Forms hypotheses, tests them (15-60 minutes) 5. Identifies root cause, implements fix (variable) Steps 2-4 — context gathering and hypothesis formation — account for 60-80% of MTTR. This is where AI agents make the biggest impact. ### Context Switching Kills Velocity During a P1 incident, engineers jump between 5-10 tools: Grafana dashboards, kubectl logs, Datadog traces, Sentry errors, PagerDuty history, Slack threads. Each context switch loses time. An AI agent can query all of these in parallel. ### Tribal Knowledge Doesn't Scale Senior engineers resolve incidents faster because they've seen similar issues before. They remember: "Last time payments-service had high error rates, it was the connection pool." New engineers don't have this knowledge. AI agents with episodic memory do. ## How AI Reduces Each Phase of MTTR ### Detection to Triage (0-5 minutes) AI agents start investigating the moment an alert fires — no waiting for an engineer to wake up, log in, and orient. By the time the on-call engineer sees the alert, OpenSRE has already: - Queried Kubernetes for pod health and recent events - Checked Prometheus for anomalous metrics - Looked up the service in the knowledge graph for dependencies - Retrieved similar past incidents from episodic memory ### Triage to Root Cause (5-20 minutes vs. 45-90 minutes) Instead of a single engineer checking tools one by one, OpenSRE dispatches multiple investigation subagents in parallel. One agent checks Kubernetes, another checks metrics, another checks logs, another checks distributed traces — simultaneously. The planner combines these findings into a coherent hypothesis. ### Root Cause to Resolution OpenSRE doesn't resolve incidents automatically (that would be dangerous without human oversight), but it gives the on-call engineer a structured report with: - Probable root cause with evidence - Timeline of events leading to the incident - Affected services and blast radius - Suggested remediation steps based on past incidents - Links to relevant dashboards and log queries The engineer validates the root cause and implements the fix, starting from a fully investigated state rather than a blank slate. ## The Learning Loop OpenSRE's episodic memory creates a compound effect on MTTR: | Investigation # | What happens | |-----------------|-------------| | First | Full investigation from scratch | | 2nd-5th | Episodic memory provides context from past episodes | | 6th-10th | Strategies auto-generate from patterns | | 10th+ | High-confidence pattern recognition, faster resolution paths | Each investigation makes the next one faster. After 10 investigations of similar incidents, OpenSRE has learned the common root causes, which skills to prioritize, and what remediation steps work. ## Measuring the Impact Track these metrics before and after deploying OpenSRE: - **MTTR per alert type**: Breakdown shows which incident categories benefit most - **Time to first context**: How long until the on-call engineer has a full picture - **Investigation depth**: Number of data sources queried per incident - **Repeat incident rate**: Episodic memory should reduce recurring incidents over time --- ## FAQ ### For SREs & Platform Engineers Q: What is OpenSRE? A: OpenSRE is an open-source AI SRE platform that investigates production incidents autonomously. It uses LangGraph to orchestrate multiple AI agents that gather context from your observability stack, reason about root causes, and produce structured incident reports. Unlike stateless AI tools, OpenSRE has episodic memory that learns from every past investigation, and a Neo4j knowledge graph that maps your service topology. Q: How does OpenSRE investigate incidents? A: When an alert fires, OpenSRE's planner agent breaks the investigation into parallel subtasks — checking Kubernetes health, querying Prometheus metrics, reading application logs, analyzing distributed traces. Multiple subagent executors run these subtasks simultaneously. A synthesizer combines the findings and produces a structured report with root cause, evidence, and remediation suggestions. Q: What investigation skills does OpenSRE have? A: OpenSRE has 50+ built-in investigation skills covering Kubernetes, Prometheus, Grafana, Datadog, Elastic/ELK, Splunk, Jaeger, Sentry, PagerDuty, Slack, GitHub, and more. Skills are loaded on-demand — agents request the skill they need, preventing context window bloat. You can also add custom skills for tools specific to your stack. Q: How does episodic memory work in OpenSRE? A: After every investigation, OpenSRE extracts metadata — root cause, alert type, affected services, severity, resolution status — and stores it as an episode. Before each new investigation, it retrieves similar past episodes using weighted similarity scoring (alert type, service overlap, resolution status). This context is injected into the investigation planner, so OpenSRE learns from every incident. Q: What integrations are supported? A: OpenSRE integrates with Kubernetes, Prometheus, Grafana, Datadog, New Relic, Elastic/ELK, Splunk, Jaeger, Sentry, PagerDuty, OpsGenie, Slack, GitHub, and Confluence. For LLM providers, it uses LiteLLM as a proxy, supporting Anthropic, OpenAI, OpenRouter, and any compatible provider including self-hosted models. Q: Can I add custom investigation skills? A: Yes. Skills are defined as directories in `.claude/skills/` with a `SKILL.md` describing the skill's tools and context, plus executable scripts. Any tool your engineers use can become an investigation skill — internal monitoring tools, proprietary deployment systems, custom alerting pipelines. Q: How does OpenSRE connect to my observability stack? A: OpenSRE connects to your stack via integration configuration in `litellm_config.yaml` and the config-service. Each integration has a URL and credentials. During investigations, agents query these integrations directly — Prometheus for metrics, Elasticsearch for logs, Jaeger for traces, Kubernetes API for pod health. ### For Engineering Managers Q: How does OpenSRE reduce MTTR? A: OpenSRE reduces MTTR by automating the investigation phase — the 60-80% of incident resolution time spent gathering context, forming hypotheses, and querying tools. It starts investigating the moment an alert fires, runs multiple investigation threads in parallel, and presents the on-call engineer with a fully investigated situation rather than a blank dashboard. Q: How is OpenSRE different from PagerDuty AI or Rootly AI? A: OpenSRE is open-source (Apache 2.0) and self-hosted, while PagerDuty and Rootly are commercial SaaS. OpenSRE's key differentiators are episodic memory (it learns from every incident) and a Neo4j knowledge graph (it maps your service topology for blast radius analysis) — neither commercial tool has these. OpenSRE also supports any LLM provider via LiteLLM, not just one fixed model. Q: Is OpenSRE production-ready? A: OpenSRE is actively used in production environments. It runs as a set of Docker Compose services (or Kubernetes via Helm chart) and has been tested with real production incident investigation scenarios. The core agent pipeline, episodic memory, and knowledge graph components are stable. As with any open-source platform, you're responsible for your own deployment and operations. Q: What's the cost of running OpenSRE? A: OpenSRE itself is free and open-source. The main cost is infrastructure (running PostgreSQL, Neo4j, and the agent services) and LLM API usage. Infrastructure costs are typically $50-200/month on a small cloud instance. LLM costs depend on investigation volume and which model you use — Claude Haiku or similar efficient models keep costs low. Q: Can OpenSRE replace our on-call rotation? A: OpenSRE augments on-call engineers, not replaces them. It handles the investigation phase autonomously — gathering context, forming hypotheses, identifying probable root causes. The on-call engineer validates findings and implements fixes. For low-severity, well-understood incidents with clear runbooks, you can configure automated remediation, but human oversight remains for production changes. ### For DevOps Generalists Q: How do I set up OpenSRE? A: Clone the repository, create a `.env` file with your `OPENROUTER_API_KEY`, and run `make dev`. This starts all services including PostgreSQL, Neo4j, the sre-agent, and the web console. The web console is available at http://localhost:3002. Full setup guide at /docs/quick-start. Q: What infrastructure does OpenSRE need? A: OpenSRE requires Docker and Docker Compose for local development. In production, it runs on Kubernetes via a Helm chart. Core dependencies are PostgreSQL (for episodic memory and config), Neo4j (for the knowledge graph), and a LiteLLM instance (for LLM routing). A small setup runs comfortably on 4 vCPUs and 8GB RAM. Q: Does OpenSRE work with Kubernetes? A: Yes — Kubernetes investigation is one of OpenSRE's primary use cases. The k8s-debug skill checks pod health, restart counts, and OOMKilled events. The k8s-deployments skill monitors rollout status and replica counts. OpenSRE also maps your Kubernetes services in the knowledge graph for dependency tracking and blast radius analysis. Q: What LLM providers does OpenSRE support? A: OpenSRE supports any LLM provider via LiteLLM. Out of the box, it's configured for OpenRouter, which gives access to Claude (Anthropic), GPT-4 (OpenAI), Llama 3 (Meta), and hundreds of other models. You can switch to direct Anthropic or OpenAI APIs, or run a local model via Ollama — just update `litellm_config.yaml`. Q: Can I run OpenSRE without cloud dependencies? A: Yes. OpenSRE can run fully air-gapped with local LLMs (via Ollama + LiteLLM), local PostgreSQL, and local Neo4j. All services are containerized. The only external dependencies are your existing monitoring tools (Prometheus, Grafana, etc.) — which you're already running. ### General Q: Is OpenSRE truly open source? A: Yes. OpenSRE is released under the Apache 2.0 license, one of the most permissive open-source licenses. You can use it for commercial purposes, modify it, and distribute it. The full source code is on GitHub at https://github.com/swapnildahiphale/OpenSRE. Q: What license does OpenSRE use? A: OpenSRE uses the Apache License 2.0. This is a permissive open-source license that allows you to use, modify, and distribute the software for any purpose, including commercial use, without restriction. You can self-host it in your production environment with no licensing fees. Q: How can I contribute to OpenSRE? A: Contributions are welcome on GitHub at https://github.com/swapnildahiphale/OpenSRE. You can contribute new investigation skills, improve existing integrations, fix bugs, improve documentation, or share feedback via GitHub Issues. The project welcomes skill contributions especially — if you've integrated OpenSRE with a tool we don't support yet, a pull request is the best way to contribute. --- ## Glossary **AI SRE**: An AI SRE (AI Site Reliability Engineer) is an autonomous software agent that performs incident investigation and response tasks that would otherwise require a human SRE. It gathers context from monitoring tools, reasons about root causes, and produces structured incident reports. In OpenSRE: OpenSRE is an open-source AI SRE platform. Its agents investigate production incidents using 50+ investigation skills across Kubernetes, Prometheus, Grafana, Datadog, and other tools. **Alert Fatigue**: Alert fatigue is the desensitization of on-call engineers to alerts due to high volume, frequent false positives, or low-signal notifications. It leads to missed critical alerts, slow response times, and burnout. In OpenSRE: OpenSRE addresses alert fatigue by automatically triaging and investigating alerts, reducing the cognitive load on on-call engineers and ensuring every alert gets proper investigation. **Blast Radius Analysis**: Blast radius analysis is the process of determining which services, systems, or users are affected by a failure or change. In microservices architectures, a single service failure can cascade to many dependent services. In OpenSRE: OpenSRE performs blast radius analysis using its Neo4j knowledge graph. When a service fails, it traverses the dependency graph to identify all affected downstream services, proactively checking their health during investigation. **Episodic Memory**: Episodic memory in AI systems is a mechanism for storing and retrieving memories of specific past events. Unlike semantic memory (general knowledge), episodic memory records what happened, when, and in what context — enabling learning from specific experiences. In OpenSRE: OpenSRE's episodic memory stores every past investigation as an episode, including root cause, affected services, and resolution details. Before each new investigation, it retrieves similar past episodes to guide the current investigation. **Incident Investigation**: Incident investigation is the process of determining the root cause, scope, and timeline of a production incident. It involves gathering evidence from monitoring tools, forming hypotheses, testing them, and documenting findings. In OpenSRE: Incident investigation is OpenSRE's primary function. Its AI agents conduct the full investigation cycle autonomously — from initial context gathering through root cause identification to report generation. **Investigation Skills**: In the context of AI SRE agents, investigation skills are modular capabilities that allow an agent to query specific tools or data sources. Each skill encapsulates the tools and context needed to investigate one domain. In OpenSRE: OpenSRE has 50+ built-in investigation skills covering Kubernetes, Prometheus, Grafana, Datadog, Elastic, Splunk, Jaeger, Sentry, and more. Skills are loaded on-demand during investigations. **Knowledge Graph**: A knowledge graph is a database that stores entities (nodes) and their relationships (edges) in graph form. In infrastructure contexts, it typically represents services, their dependencies, ownership, and infrastructure components. In OpenSRE: OpenSRE maintains a Neo4j-powered knowledge graph of your service topology. It enables blast radius analysis, dependency traversal, and ownership lookup during incident investigation. **LangGraph**: LangGraph is an open-source framework for building stateful, multi-agent AI applications. It represents agent workflows as directed graphs, where nodes are processing steps and edges define the flow of information between them. In OpenSRE: OpenSRE uses LangGraph to orchestrate its investigation pipeline: planner → parallel subagent executors → synthesizer → writeup → episodic memory storage. **MTTR (Mean Time to Resolution)**: Mean Time to Resolution (MTTR) is the average time from when an incident is detected to when it is fully resolved. It is a key metric for measuring incident response effectiveness and SRE team performance. In OpenSRE: Reducing MTTR is OpenSRE's primary value proposition. By automating the investigation phase (typically 60-80% of resolution time), OpenSRE significantly reduces MTTR for production incidents. **Observability**: Observability is the ability to understand the internal state of a system from its external outputs. In software engineering, observability is typically achieved through three pillars: logs, metrics, and traces. In OpenSRE: OpenSRE integrates with your entire observability stack — Prometheus for metrics, Elastic/Splunk for logs, Jaeger/Datadog for traces — querying all three pillars during incident investigation. **On-Call Automation**: On-call automation refers to software systems that automatically handle incident response tasks that would otherwise require a human engineer to be paged and manually intervene. It ranges from simple runbook execution to full AI-powered investigation. In OpenSRE: OpenSRE automates the investigation phase of on-call response. When an alert fires, OpenSRE investigates immediately without requiring an engineer to start the process, reducing time-to-investigation from minutes to seconds. **Root Cause Analysis**: Root cause analysis (RCA) is the process of identifying the fundamental reason a problem occurred, rather than just treating its symptoms. In incident management, RCA aims to understand why an incident happened to prevent recurrence. In OpenSRE: OpenSRE performs automated root cause analysis as part of every investigation, synthesizing evidence from multiple data sources to identify the most probable root cause and supporting evidence. **Runbook Automation**: Runbook automation is the process of codifying and automatically executing operational procedures (runbooks) in response to specific events or conditions. It transforms manual, step-by-step procedures into automated workflows. In OpenSRE: OpenSRE complements runbook automation. While runbook automation handles known, well-defined scenarios, OpenSRE's AI agents handle novel incidents that don't match existing runbooks. **Service Topology**: Service topology describes the structure and relationships of services in a distributed system — which services communicate with which, what dependencies exist, and how they're organized. Understanding topology is critical for blast radius analysis and root cause investigation. In OpenSRE: OpenSRE maps your service topology in a Neo4j knowledge graph, updated continuously with new deployments and dependency changes. This topology context is automatically provided to investigation agents. **SRE Agent**: An SRE agent is an autonomous AI system that performs site reliability engineering tasks — monitoring infrastructure health, investigating incidents, executing operational procedures, and maintaining system reliability — with minimal human intervention. In OpenSRE: The sre-agent is OpenSRE's core component: a LangGraph-orchestrated AI agent that receives alerts, coordinates investigation subagents, and produces incident reports. It's the brain of the OpenSRE platform. --- ## Comparison: OpenSRE vs Commercial Tools How OpenSRE compares to PagerDuty AI, Rootly AI, and Shoreline. | Capability | OpenSRE | PagerDuty AI | Rootly AI | Shoreline | |-----------|---------|-------------|----------|-----------| | Open Source | Apache 2.0 | No | No | No | | Self-Hosted | Yes | No | No | Partial | | Episodic Memory | Yes | No | No | No | | Knowledge Graph | Yes | No | No | No | | Investigation Skills | 50+ built-in + custom | Limited | Limited | Yes | | LLM Provider Choice | Any (via LiteLLM) | Fixed | Fixed | Fixed | | Slack Integration | Yes | Yes | Yes | Yes | | Web Console | Yes | Yes | Yes | Yes | | Custom Skills | Yes | No | No | Partial | | Price | Free (Apache 2.0) | $$$/month | $$$/month | $$$/month | ### Episodic Memory OpenSRE's episodic memory system learns from every investigation. After each incident, it extracts structured metadata — root cause, alert type, affected services — and stores it. Future investigations of similar incidents benefit from this accumulated knowledge. Neither PagerDuty AI, Rootly AI, nor Shoreline have an equivalent system. ### Knowledge Graph The Neo4j knowledge graph gives OpenSRE's agents a live map of your service topology. During investigation, they can perform blast radius analysis (what services are affected by this failure?), dependency traversal (what does this service depend on?), and recent change detection. This structural context is what transforms generic AI investigation into system-aware diagnosis. ### Open Source + Self-Hosted OpenSRE is Apache 2.0 licensed and fully self-hosted. Your incident data, episodic memory, and knowledge graph never leave your infrastructure. For teams with compliance requirements, air-gapped environments, or data sovereignty concerns, this is the decisive advantage. Commercial tools require sending incident data to external SaaS platforms. ### LLM Provider Choice Commercial tools lock you into a specific AI provider. OpenSRE routes through LiteLLM, which supports Anthropic, OpenAI, OpenRouter, Ollama, and any compatible provider. Use Claude for quality, switch to efficient open-source models for cost, run local models for air-gapped environments.