# How we prevent slop from hitting prod

> Nothing in your pipeline asks whether a change is safe for prod. This is the check we built that does, and how we got it to finish before your tests do.

Published: September 21, 2026
Authors: Vi Tran and Boris Tane
Canonical: https://polylane.com/blog/how-we-prevent-slop-from-hitting-prod/

You're probably either building a software factory or leasing one from a provider. You have a system that takes you from a prompt to a pull request. It handles tests, linting, formatting, and automated code reviews.

But you still don't have an answer to the most important question: **is this change okay to go to prod?**.

```mermaid
graph TB
    W["Agent writes the code"] --> T["Types and tests"]
    T --> L["Linter and formatter"]
    L --> R["Code review"]
    R --> Q(["Is this okay for prod?"])
    Q --> D["Deploy"]
    style Q fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```

All of your current checks look at the diff, but nothing in your factory knows anything about the production system the diff is about to land on. Nothing can really prevent slop from hitting prod.

We built this capability inside of [Polylane](https://polylane.com) and this post is about the technical details of how we implemented it.

## How it works

The one questions this system should answer is:

> Would this change, once merged and deployed, have a negative impact on production?

We do not really care about the typical things a code-review agent would check such as style, naming, test coverage, etc. And we decided to answer this question at the pull-request stage, along side all your existing tests.

Ultimately, Polylane comments on the pull request with a simple "go" / "no-go" message with the receipts from its investigation.

The flow is fairly simple:
- Is this Pull request touching files that may affect production?
- Which cloud resources are potentially afftected?
- Gather context about the current state of production for those resources
- Evaluate multiple potential failure modes this change may introduce
- Forecast how production might change with these changes deployed
- Alert developers of the likely potential failure modes

*A no-go verdict: the mechanism in the first line, the evidence under it, and what would make the change safe.*

This all relies on the [context graph](/product/context/) we continuously build connecting all the cloud resources in your various cloud accounts.

```mermaid
graph TB
    P["Open pull request"] --> Q(["Meaningful code change?"])
    Q -->|"docs, tests, comments"| C["Conclude the check"]
    Q -->|"yes"| R["Find affected resources<br/>in the context graph"]
    R -->|"none"| C
    R -->|"yes"| E["Gather context"]
    E --> X["Build failure trajectories"]
    X --> F["Forecast the affected series"]
    F --> V["Verdict"]
    V --> C
    style X fill:#fef3c7,stroke:#fcd34d,color:#78350f
    style F fill:#fef3c7,stroke:#fcd34d,color:#78350f
    style V fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```

## Assembling the context

Our context graph is key to making this work, it builds a registry of all your cloud resources, all your repositories, teams, etc. For instance, a compute node such as Lambda function is connected to the database it reads from and the queue that triggers it. We also add repositories to the graph. This is done by looking at the typical manifest files in the repository, for example terraform files, Cloudformation files or Wrangler files. This enables the connection between repositories and cloud resources.

```mermaid
graph LR
    Q["SQS queue<br/>orders-events"] -->|"triggers"| A["Lambda function<br/>checkout-api"]
    R["Repository<br/>checkout-edge"] -->|"deploys_to"| A
    A -->|"connects_to"| D["RDS instance<br/>orders-db"]
    style R fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```

When a pull request is submitted to the repository, we follow the paths on the context graph to collect all the cloud resources potentially affected by the change. We use a small models to filter through the resources as there could be a large number of resources deployed from the same repository. We pass this context alongside the diff, the PR description and the commits in the PR to the agent.

```mermaid
graph LR
    R["Repository"] -->|"deploys_to"| C["Candidate resources"]
    C --> F(["Small model filters"])
    F --> A["Agent"]
    D["Diff, description, commits"] --> A
    style A fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```

We also pass the diff through a set of deterministic heuristics to quickly guide the agent's attention towards things that are usually likely to have a negative impact on production:

- a migration that has to be applied by hand, or in a particular order relative to the deploy
- `CREATE INDEX` without `CONCURRENTLY`, or `ADD COLUMN ... NOT NULL` with no default, both of which take a lock on the table for the duration
- an endpoint being removed while the currently deployed version still reads it
- code that starts reading an environment variable, secret or binding that nothing in the diff provisions

These are recommendations, streering the agent to probably deployment risks.

## Failure trajectories

With the context provided above, the model comes up with various failure modes the new diff may introduce in production and goes and investigates each. 

Its output is a ledger of failure trajectories. A trajectory is one causal chain from a trigger, through the changed code, to an observable degradation on a specific metric, and every link in it carries a citation: a file and line, a log template with its count, a metric read, a config key, a graph edge.

The agent tries to both validate and invalidate each trajectory before coming to a verdict. Each ends in one of three states:

- `confirmed` when the trajectory has been confirmed against production.
- `plausible` when the chain is concrete but one or multiple links could only be "guessed" with no telemetry data to confirm it.
- `refuted` when production telemetry data provided enough evidence that this trajectory is unlikely to happen in production.

We take a conservative approach and any `confirmed` trajectory leads to a failed assessment.

```ts
export function deriveTrajectoryVerdict(trajectories: Trajectory[]): "pass" | "fail" {
  return trajectories.some((entry) => entry.status === "confirmed") ? "fail" : "pass";
}
```

```mermaid
graph LR
    D["The diff"] --> T1["Dropped index still<br/>used by checkout"]
    D --> T2["Retry change amplifies<br/>load on orders-db"]
    D --> T3["Removed binding<br/>breaks the worker"]
    T1 --> C1["confirmed"]
    T2 --> C2["plausible"]
    T3 --> C3["refuted"]
    C1 --> V["No-go"]
    C2 --> V
    C3 --> V
    style C1 fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
    style C2 fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
    style C3 fill:#d1fae5,stroke:#6ee7b7,color:#065f46
    style V fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```

## Forecasting the impact

We give the agent a tool to forecast timeseries based on historical data with injecting potential external factors.

Say a trajectory claims some change halves the timeout on a path that retries. Whether that matters depends on the traffic, and traffic isn't really a single number, it has a shape. A queue sitting at 60% depth and staying there is fine. The same queue at 60% but climbing every week is a different situation, and reading the last hour of metrics won't tell you which one you're looking at.

So before the agent drafts the verdict of a trajectory, it pulls the historical series for the affected resources and forecasts them together. We currently self-host the [Toto-2.0-22m model](https://huggingface.co/Datadog/Toto-2.0-22m).

```mermaid
graph LR
    A["Agent"] -->|"telemetry query"| P["Provider"]
    P -->|"aligned series"| A
    A -->|"up to 16 series"| M["Forecasting model"]
    M -->|"p10, p50, p90"| A
    style M fill:#fef3c7,stroke:#fcd34d,color:#78350f
```

The reason for a dedicated multivariate model rather than another prompt is that the series aren't independent. Request rate, error rate, latency and queue depth on one resource move together, and forecasting each on its own throws away the correlation that makes the forecast worth having. Every series in a call shares one attention group.

This is the most experimental capability we recently added, and still measuring its impact. But it already passes the vibes-eval.

**Figure 2. An example of how the forecast works**

64 hourly observations of one affected resource go in, and 24 hours of median and p10-p90 come back. The band widens with the horizon, and the agent reads the band rather than the median alone: a p10 that stays above the threshold a trajectory depends on is a different answer from one that crosses it.

*checkout-api requests per hour: 64 observed hourly buckets, then a 24 hour forecast. The median settles near 26,000 an hour; the p10-p90 band opens from about ±3,200 at the first step to ±6,300 at the last.*

## The latency constraint

Running the production impact assessment in the pull request flow means it must to be fast. Nobody wants a step that adds 15 minutes to their CI pipeline. For instance, on our own repositories this assesment is a required CI step, if it's slow, our entire SDLC gets bogged down.

We essentially have a budget of 2 to 3 minutes to produce an accurate impact assesment. Anything beyond that is not acceptable in a CI/CD pipeline.

Our first prototype was completely failing this constraint. Our median was nearly 7 minutes and it was common to see runs take up to 20 minutes.

**Figure 3. Median latency of each of the steps in the production impact assessment run**
Production medians, 15 September 2026, over 325 runs. Each phase is its own median, so the unaccounted block at the end is queueing between phases plus the arithmetic of adding medians.

| Span | Starts at | Duration |
| --- | --- | --- |
| Prelude (webhook, records, diff) | 0 s | 7.4 s |
| Sandbox (clone the head) | 8.7 s | 16.5 s |
| Review turn | 25.2 s | 242.4 s |
| Delivery (check run, comment) | 267.6 s | 3 s |
| Unaccounted | 270.6 s | 38.6 s |
| **Total** | | **309.2 s** |

The goal became figuring out how to reduce the number of model steps to get the entire flow under 3 minutes. We typically ran 30 to 40 sequential model steps, and in the worst case scenarios up to nearly 600 steps, each consuming 17 seconds of our budget, and the vast majority of their output was reasoning tokens. 

We made quite a few changes over the past 3 weeks, with various impact on performance. Here are the most significant ones.

### Setting a step budget and communicating it to the agent

We introduced a step budget, capped at 30 steps by agent turn and we clearly communicate to the model how many steps it has consumed on every step. Putting the count in the prompt lets the model plan around it, which turns out to matter more than the number itself.

```mermaid
graph TB
    A["Turn starts, 30 steps"] --> B["Model step"]
    B --> T["Tool call"]
    T --> C(["Steps left?"])
    C -->|"1"| E["Record the verdict now"]
    C -->|"more"| R["Next step, carrying<br/>the count in the prompt"]
    R --> B
    style R fill:#fef3c7,stroke:#fcd34d,color:#78350f
    style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```

### Starting new reviews instead of folding into the existing one

A typical pull request keeps getting new commits after it's opened. In the first prototype, we would fold the entire new diff in the same review thread as a steering user message. This steer would greatly confuse the model and lead to reading files it had already examined and re-running telemetry queries it had already completed.

Now, a new commit cancels the previous assesment run and starts a brand new thread. It seems counter-intuitive, but it ultimately brought down the latency for complete reviews.

### Re-using previous work

When a new commit lands in the pull request after a review was already completed, we used to naively start a new impact assessment from scratch.

We introduced the ability to re-use the previous assessment, and trigger a new assesment on the smaller diff between the two consecutive commits.

We deployed these changes throughout September and have gradually improved latency and the latency comfortably sits within budget.

**Figure 4. Median latency for an impact assessment turn**

| Day | Median impact assessment turn |
| --- | --- |
| 27 Aug | 544 s |
| 28 Aug | 473 s |
| 29 Aug | 453 s |
| 30 Aug | 443 s |
| 31 Aug | 449 s |
| 1 Sep | 477 s |
| 2 Sep | 507 s |
| 3 Sep | 387 s |
| 4 Sep | 275 s |
| 5 Sep | 199 s |
| 6 Sep | 206 s |
| 7 Sep | 219 s |
| 8 Sep | 237 s |
| 9 Sep | 224 s |
| 10 Sep | 210 s |
| 11 Sep | 245 s |
| 12 Sep | 328 s |
| 13 Sep | 251 s |
| 14 Sep | 346 s |
| 15 Sep | 213 s |
| 16 Sep | 166 s |
| 17 Sep (partial day) | 94 s |

## Does this even matter?

What's the point of burning all these tokens if we don't see any meaningful results? Our success metric is the number of incidents we prevent per week for each of our customers. A prevented incident is a pull request where:
- we flag a potential risk to production
- an engineer pushes one or multiple commits
- a new assessment concludes the potential risk is mitigated
- the pull request gets merged

**2.3 Production incidents prevented, per customer, every week**

This is already pretty significant and we expect this to keep growing. We are continously iterating and running evals to improve the performance and quality of our agent.