Braintrust LLM traces, evals and alerts in Polylane
Connect a Braintrust organization so Polylane picks up your time window alerts, watches each project's errors, latency and cost, and reads traces and experiments in investigations.
What changed
Connect a Braintrust organization with a service token, including organizations on the hosted EU data plane. The token must come from a permission group that can read and update all projects. Personal API keys are refused, because they follow a person into every organization they join.
Polylane adds its webhook to each of your time window alerts, on connect and every hour after, and leaves your other actions as they are. A firing alert opens an issue that Polylane investigates, and a recovery marks it recovered.
Every 30 minutes, Polylane checks your busiest projects for changes in root span error rate, p99 duration and volume, LLM span errors, estimated cost and tokens. During an investigation, agents search traces, read one as a span tree, query logs and evals with SQL, and compare an experiment with its baseline. See the Braintrust integration.
Why it matters
LLM apps break in ways a status code does not show: a slower model call, a jump in cost, an eval that regresses. Polylane reads the traces and experiments that explain the change and investigates it with the rest of your stack.
How to use it
- In
Braintrust, open Settings → Service tokens and create a token in the organization you want to connect.
- Add the token to a permission group that can read and update all projects, such as Owners or Engineers.
- Connect
Braintrust in Polylane and paste the token.
On the integration page you can change how often the checks run, or turn them off.