Skip to main content

Northflank Break-Glass

Symptoms​

A fix can't wait, and the normal path (a pull request to qubital-infra) can't deliver it: GitHub is down, CI is down, or Northflank's template sync is down.

Impact​

Prod or staging changes outside review. Only the team owner does this.

Steps​

GitHub or CI is down, Northflank's sync works:

  1. Reload the template's page in Northflank (a save from a stale page overwrites newer commits).
  2. Edit the template, save, and run it. Northflank pushes the whole file to qubital-infra's main as a commit.
  3. Once GitHub is back, review that commit and amend it with a pull request.

Northflank's sync is down:

  1. Edit the service in the UI (for example Environment → the key → Update & restart).
  2. In the same sitting, merge a pull request to qubital-infra that puts exactly the same value in the file. Without it, the next deploy silently reverts the edit. Its verify goes red and stays red by design: the change was live before any run applied it, so verify sees no rollout.
  3. Don't run that template by hand until step 4: Northflank still holds the old file, and a run would revert the edit.
  4. Once sync works again, merge a pull request that changes a node's description in the same file. A push made during the outage is not replayed, so this is what makes Northflank sync the file. Its run applies what already runs and restarts nothing.

Verification​

  • The service is healthy and /health answers.
  • The verify run of the step 4 pull request is green: the whole file, including the step 2 value, is what runs.

Escalation​

The team owner.