Skip to main content

Deploy Failed: verify Is Red

Symptoms​

  • The staging deploy job is red at "Wait until staging runs…".
  • The promote is red at a "Wait until prod runs…" step.
  • A verify run in qubital-infra is red.

Impact​

The commit may not be live, or something restarted that shouldn't have. Stop further deploys to that environment until you know which.

Steps​

  1. Open the verify run of the commit in qubital-infra (Actions → verify) and read the first error line of "Verify live state".

  2. Act on it:

    ErrorCauseWhat to do
    sync: not done after 10 minNorthflank didn't pick up the commit: its sync is down (check status.northflank.com), or the template holds a different version of the file (the message's last: part says which)Once sync works again, a push made during the outage is not replayed: merge a pull request that touches each affected file (a final newline is enough), then re-run this verify. After a break-glass edit, follow Northflank Break-Glass instead
    Northflank stored this file with defaults it addedThe file lacks fields Northflank fills in by defaultWrite the listed fields into the file through a pull request
    isn't linkedNo template is linked to that fileThe owner links it (see the qubital-infra README)
    the run failed or never startedThe template run failed in NorthflankNorthflank → Templates → the template → Runs. A wait-centrifugo timeout means the broker didn't come up: check its logs, then roll back
    unexpected rolloutA service restarted that the commit didn't changeSomeone restarted or edited it in the UI, or Northflank moved it. Find out which. A re-run of this verify can't pass
    wrong image or not COMPLETEDThe new container never became readyThe service's logs. Usually config or a migration: fix forward or roll back

  3. Re-run once the cause is fixed: the verify run itself, then the waiting producer job. For the promote, use Re-run all jobs.

  4. After an unexpected rollout, re-running that verify can't pass.

    • Prod promote: merge a pull request that changes a node's description in production/backend.json (it restarts nothing; a newline-only change doesn't count), wait for its verify to go green, then Re-run all jobs: the promote accepts any green verify since its digest was written.
    • Staging deploy: it recovers with the next merge to main, which deploys a new digest.

Verification​

The verify run is green, and the producer job (staging deploy or promote) is green.

Escalation​

The team owner. For a Northflank outage, follow their status page.