儀表板

凌晨 2:14,結帳失敗了。 到 2:15,調查已經在跑。

加入候補名單

警示響了。Polylane 接手。 真的還是雜訊:不到一分鐘就有答案。

Issues Elevated error rate on payments-api Timeline

Elevated error rate on payments-api

High Incident ·Detected from a Datadog alert
OverviewMetricsLogsTracesTimelinePropertiesInvestigation
  1. 02:14:07 Alert received

    Datadog · payments-api error rate · occurrence #1

  2. 02:14:09 Triage started

    Fetching the monitor query and metrics 30 min before, 5 min after the firing.

  3. 02:14:58 Triaged as an incident

    Severity high. Error rate 41× the 30-minute baseline, clustered on POST /v1/charges; deploy f3a9c12 landed 18 minutes before the firing.

  4. 02:15:02 Investigation started

    3 hypotheses · 3 passes each

  5. 02:31:44 Alert fired again (occurrence #2)

    Folded into this issue after 16 quiet minutes; the running investigation was nudged.

  6. 03:06:12 Alert recovered

    Provider recovery received. Issue resolved.

它一直挖,直到能證明為止。 假設從正反兩方論證,判定需要證據。

Threads Critical latency degradation in checkout-edge worker Share

What I did

I queued an autofix that restores the Hyperdrive pool hd-prod in the coreplane/checkout-edge repo with:

  • max_connections restored from 5 to 50 on the pool binding
  • a regression test that fails when checkout queues on the pool
  • wrangler.jsonc configured to deploy as checkout-edge

This gives Boris a PR with deployable code. Once merged, running wrangler deploy will replace the degraded worker with the fixed version.

The root cause report has been updated with the occurrence #2 evidence: Root Cause: checkout-edge Latency Degradation (Recurring).

Wall Time
checkout-edge worker · last hour
Export chart
Wall Time P99
28,381ms
Wall Time P95
2,696ms
Wall Time P50
118ms
010k20k30k
18:0018:2519:00
Worked for 4m 12s

AI can make mistakes. Please double check cited sources.

Dig deeper...

事件以修復收場,而不是一場後續會議。 找到根本原因,寫好修復,等你審查。

github.com/coreplane/payments-api/pull/491

Cap retries on the checkout webhook worker #491

polylane
Open polylane wants to merge 1 commit into main from polylane/autofix/chat/k3x9f2-4e7d21a
Conversation 1 Commits 1 Checks 1 Files changed 2
polylane bot commented 6 minutes ago ···

Retries on the checkout webhook worker were unbounded: a failing delivery re-queued itself forever and amplified load on payments-api. This caps delivery at 5 attempts with exponential backoff and dead-letters the payload after the last one.

What changed

worker/deliver.ts gains MAX_DELIVERY_ATTEMPTS = 5 and backoff between attempts; exhausted payloads land in checkout-webhooks-dlq instead of re-queueing.

Validation

npm test — 214 passed. A forced failing delivery stopped after 5 attempts and appeared in the dead-letter queue.

Root cause · Why it's safe · Out of scope
polylane added commit 4e7d21a Verified
Review required At least 1 approving review is required
ci / test Successful in 3m 12s Details
Review required Waiting on your review: Polylane never merges on its own
Merging is blocked

Polylane 如何運作。 它學習你的系統、監看它、調查,然後行動。

  • 它先學習你的系統

    情境圖描繪出你的雲端、儲存庫與可觀測性供應商之間的每一個資源與依賴關係。代理依據真實的拓樸推理,而不是猜測。

  • 不靠閾值的偵測

    每個供應商都有內建檢查,再加上從你自己儲存的查詢與儀表板產生的檢查。一次統計分析與一個代理共同決定,而改善永遠不會引發問題。

  • 拿得出證據的調查

    每一項主張都連回背後的查詢、日誌行或變更記錄。沒有證據的判定會退回無法定論。

  • 寫入是掙來的,絕非理所當然

    帳戶以唯讀方式連接。復原預設關閉、受頻率限制,且留有記錄。程式碼變更走你平常的審查流程。

  • 它每週都更敏銳

    記憶、每日筆記與對照真實資料重新確認的監控查詢:七月的調查會從六月的學到東西。

它接上你已經在跑的一切。 以唯讀方式連接,然後開始。

常見問題。

Polylane 會呼叫我嗎?

它不是呼叫器,也不打算成為呼叫器。Polylane 透過電子郵件、Slack 與主控台通知,而且只針對它自己偵測到的關鍵與高嚴重程度問題。目標是更少被叫醒:問題送到你面前時已經調查過了。

半夜偵測到問題會怎樣?

分流確認它是真的,調查自動開始,假設對照你實際的遙測資料平行檢驗。到早上,問題已經有判定、背後的證據與建議的下一步;啟用自動修復的地方,修復已經寫好,等著審查。

Polylane 接收哪些警示來源?

Datadog、Honeycomb、Axiom、Grafana Cloud、Better Stack、Sentry、CloudWatch、Vercel、Render 和 Cloudflare,再加上一個通用 webhook 涵蓋其他一切。每個來源以限定範圍的遙測權杖驗證身分,每一次觸發都變成已分流的問題。

如果 Polylane 把真正的事件判成雜訊呢?

「不是事件」是一個判定,不是刪除:問題留在主控台裡,附上推理過程,而你可以一鍵對任何東西啟動調查。判定需要證據,無法取得資料的調查永遠不會確認。

它會執行多少次調查?

自動調查每 24 小時滾動計算有上限:Free 方案為 10 次,付費方案由你自訂。手動啟動的永遠不計入。價格公開在 polylane.com/pricing。

更多使用情境

把夜晚交給代理。 把早晨留給自己。