Skip to content
Shipyard

Incidents

When something breaks, Shipyard heals it and writes it up here: what happened, how it responded, the fix, and what it learned so it can't happen the same way again. Every line comes from the records the system kept while handling it.

What High and Medium mean

High
Production was failing (errors). Traffic is rolled back to the previous good version right away, while the SRE agent finds the cause and reverts it, or fixes it forward if the fix is small.
Medium
Production was slow but still answering. Nothing is rolled back: the site is slow rather than failing, so the SRE agent fixes it forward and keeps the change. A break in the main pipeline that never reaches visitors is also Medium.

8 incidents · 8 resolved · typically in 13 min · 1 handed to a person

  1. Pipeline Resolved

    #34 Incident: main broken at 90324f1e

    · resolved in 2 min 49 s

    What happened
    main stopped building after a change: deploy-staging failed.
    Response
    Production was never affected: the deploy stages didn't run.
    Fix
    main built again after the fix, and the dispatcher closed it.
    Learned
    The agents were asked for the regression test that would have caught it, and it became a lesson for every agent that writes code. See request #35
    Full incident on GitLab
  2. Production Resolved

    #24 Incident probe-20261021-1128: Shipyard production: synthetic probe

    · resolved in 9 min 35 s

    What happened
    The synthetic probe saw GET /roadmap/list 500, in two checks 20 s apart.
    Response
    Traffic moved back on Cloud Run
    Fix
    The SRE agent found the cause and reverted it in !18. See the revert
    Cost to heal
    About 6.4 g CO2e: every pipeline and agent session the repair took, measured the same way as a shipped feature.
    Learned
    The agents were asked for the regression test that would have caught it, and it became a lesson for every agent that writes code. See request #25
    Full incident on GitLab
  3. Pipeline Resolved

    #19 Incident: main broken at 5336a464

    · resolved in 10 min

    What happened
    main stopped building after a change: security-gate failed.
    Response
    Production was never affected: the deploy stages didn't run.
    Fix
    main built again after the fix, and the dispatcher closed it.
    Learned
    The agents were asked for the regression test that would have caught it, and it became a lesson for every agent that writes code. See request #20
    Full incident on GitLab
  4. Production Resolved

    #18 Incident 0.oe2nu5eixkjp: Shipyard production: slow responses

    · resolved in 14 min

    What happened
    Responses slowed to 1.5 s (the alert fires above 1.5 s).
    Response
    Not done: production is slow but still answering, so the SRE agent fixes it forward and keeps the change.
    Full incident on GitLab
  5. Production Resolved

    #16 Incident 0.oe1eslu4i74k: Shipyard production: slow responses

    · resolved in 25 min

    What happened
    Responses slowed to 2.0 s (the alert fires above 1.5 s).
    Response
    Traffic moved back on Cloud Run
    Full incident on GitLab
  6. Production Resolved

    #15 Incident 0.oe1el9fzeugk: Shipyard production: 5xx responses

    · resolved in 13 min

    What happened
    32% of requests failed (the alert fires above 5%).
    Response
    Traffic moved back on Cloud Run
    Fix
    The SRE agent found the cause and reverted it in !12. See the revert
    Learned
    The agents were asked for the regression test that would have caught it, and it became a lesson for every agent that writes code. See request #17
    Full incident on GitLab
  7. Production Resolved A person was alerted

    #12 Incident 0.oe006had40gl: Shipyard production: 5xx responses

    · resolved in 12 min

    What happened
    33% of requests failed (the alert fires above 5%).
    Response
    Automatic rollback failed, so the incident was handed to a person while the agents worked on a fix.
    Fix
    The SRE agent found the cause and reverted it in !10. See the revert
    Learned
    The agents were asked for the regression test that would have caught it, and it became a lesson for every agent that writes code. See request #13
    Full incident on GitLab
  8. Pipeline Resolved

    #10 Incident: main broken at 9292cb4d

    · resolved in 16 min

    What happened
    main stopped building after a change: unit-tests failed.
    Response
    Production was never affected: the deploy stages didn't run.
    Fix
    main built again after the fix, and the dispatcher closed it.
    Full incident on GitLab