Skip to content
truffleroot

Root cause analysis built for your environment.

truffleroot learns how your systems fail, builds an RCA pipeline around them, and traces incidents from symptom to cause, with the evidence behind every link.

See how it worksTalk to us
  1. OnboardingLearnFrom incidents, runbooks, topology and telemetry
  2. Your RCA pipelineModelEvent types, causal rules, detectors, topology
  3. During an incidentInvestigateFrom alert to ranked root-cause candidates
  4. The outputExplainThe causal chain, with evidence on every link

Every incident feeds evidence back into the model, so the pipeline fits your environment better over time. Not a generic pipeline pointed at your systems. One built from them.

The problem

Your systems don't fail generically.

The same symptom means different things in different environments. Checkout latency might be a shrunken connection pool, a cache eviction, or a rate limiter tightened by a deploy two services downstream. Which one depends on your topology, your deployments and your dependencies.

Generic RCA systems begin with generic assumptions. truffleroot begins by learning the environment.

Same alertcheckout p99 latency above SLO
  1. payments-svc connection pool saturates
  2. A config change shrank the pool from 40 to 20
  1. Cart reads fall through to the database
  2. A cache node was evicted during a rollout
  1. Retries amplify the request volume
  2. A deploy tightened a downstream rate limiter

Bootstrapping

Teach truffleroot how your systems fail.

During onboarding, a bootstrapping agent works with your engineers and the knowledge you already have. It learns which events matter here, how components depend on each other, and how failures propagate through this environment in particular.

What it reads

  • Past incidents
  • Postmortems
  • Runbooks
  • ITSM tickets
  • War-room threads
  • Service topology
  • Traces, logs and metrics
  • Deployment and config state
  • Code and infrastructure config
  • Your engineers
bootstrappingwith your engineers

What it builds

  • 01Event types and propertiesThe events that matter in your environment, with the properties that tell one instance from another: a deploy, a config change, a pool saturating, a failover.
  • 02Causal rulesHow one kind of event leads to another here. Which changes precede which failures, on which components, within what window.
  • 03Detectors, thresholds and windowsHow each event type is recognised in telemetry, calibrated to your baselines, and how far back to look when something fires.
  • 04Dependency topologyWhich services depend on which, so an investigation scopes to the part of the graph that can actually be involved.

Together, these are the configuration for an RCA pipeline specific to your environment: a model of how it can fail. Your existing knowledge becomes an executable model for root-cause analysis.

Your engineers review and approve generated definitions and rules before they enter the active pipeline.

During an incident

From alert to causal chain.

An incident is not handed to a language model along with the whole telemetry firehose. It runs through a pipeline built for your environment.

  1. 01

    Scope

    Find the part of the dependency graph that can be involved.

  2. 02

    Events

    Collect the relevant events inside the right lookback windows.

  3. 03

    Causal graph

    Apply your causal rules and build the event graph.

  4. 04

    Candidates

    Rank the plausible root causes.

  5. 05

    Investigation

    Check each candidate against runtime state, telemetry, runbooks and past incidents.

  6. 06

    Explanation

    Return the causal chain with the evidence behind it.

INC-2231

checkout p99 latency above SLO

Root cause identified14:32:10 UTC
  1. Alertcheckout-api · slo_burn_rate · p99 2.6 s against a 1.2 s SLO
  2. Scope7 of 212 services · 2 hops upstream of checkout-api
    • checkout-api
    • payments-svc
    • payments-db
    • inventory-svc
    • config-service
    • web-gateway
    • redis-cart
  3. Events11 events in window · 14:00 to 14:32 UTC
    • latency_shift3
    • deploy2
    • error_rate2
    • config_change1
    • pool_saturation1
    • autovacuum1
    • slo_burn1
  4. Causal graph4 rules matched · 3 candidate chains
  5. CandidatesRanked, then investigated
    • 1
      config_changepayments-svcpool.max_connections 40 → 2014:11:04

      Pool saturation followed on the same service 15 minutes later. Active connections pinned at 20.

      confirmed
    • 2
      deployinventory-svcv3.8.114:02:37

      inventory-svc latency flat through the window.

      ruled out
    • 3
      autovacuumpayments-dborders14:20:15

      Query latency within baseline. No lock waits.

      ruled out
  6. ExplanationA config change on payments-svc, 21 minutes before the alert.
    Open the causal chain
Pipeline completeRules revision 14 · approved by payments on-call

Explainability

Not a score. An explanation.

See the sequence of events, the causal links between them, and the evidence supporting each conclusion.

Probable root cause

payments-svcconfidence 0.87

A ranked score tells you where to look. It cannot tell you whether it is right.

truffleroot returns what happened, what happened before it, how those events are connected, and why one of them is the root cause. Every link carries its evidence, so you can inspect it, challenge it, and act on it.

INC-2231 · Explanation

Root cause: a connection pool limit reduced by a config change

5 events · 4 causal links

At 14:11 the connection pool limit on payments-svc was reduced from 40 to 20. Under normal afternoon traffic the pool saturated fifteen minutes later, payments-svc latency rose, and checkout-api, which calls it synchronously, breached its SLO at 14:31.

  1. 14:11:04config_changepayments-svcpool.max_connections 40 → 20root cause
    Evidence for the next link
    • Revision cfg-4812 applied by the config service at 14:11:04. The diff sets max_connections from 40 to 20.
    • Rule: a pool config change on a service precedes pool saturation on that service, within 30 minutes.
    • Active connections pinned at 20 from 14:26:40. Before the change they peaked at 31.
  2. 14:26:40pool_saturationpayments-svcpool.wait_p95 3 ms → 1.9 s
    Evidence for the next link
    • 94% of slow payments-svc spans wait on pool.acquire. 2,113 traces sampled.
    • Rule: pool saturation precedes a latency shift on the same service.
  3. 14:29:12latency_shiftpayments-svcp99 180 ms → 2.4 s · timeouts 3.1%
    Evidence for the next link · 3 items
  4. 14:31:55latency_shiftcheckout-apip99 2.6 s against a 1.2 s SLO
    Evidence for the next link · 2 items
  5. 14:32:10alertcheckout-apislo_burn_rate fired
Confirm root causeChallenge a link2 candidates ruled out · reasons recorded

Between incidents

Every incident makes the model better.

Incidents expose what the model is missing: an event type it had no name for, a rule with the wrong window, two names for one service. That evidence becomes proposed refinements to your RCA configuration.

  • New event types
  • Improved event properties
  • New or modified causal rules
  • Namespace consolidation
  • Tuned thresholds and lookback windows

truffleroot progressively builds a better model of how your systems fail.

Proposed refinementAwaiting review
Type
Detector threshold
Detector
payments-svc · pool.wait_p95
Change
500 ms → 250 ms
Derived from
INC-2231
Evidencereplayed

Would have fired 4 m 10 s earlier in INC-2231. No additional fires across 90 days of history.

ApproveEditRejectInactive until approved

Refinements are proposals. Your engineers review them before they change the active pipeline.

Build RCA around the system you actually run.

Tell us what you run and what keeps breaking.

Talk to us