Skip to main content

Roll Back a Deploy

Symptoms​

A deploy to staging or prod broke something, and a fix can't ship quickly.

Impact​

The service restarts on the previous image. An api rollback across a DB migration fails at boot: the older binary refuses the newer schema.

Steps​

  1. Find the deploy commit in qubital-infra: a commit deploy(<env>/backend): api … by qubital-infra-deployer[bot]. A prod release's GitHub Release notes link it.
  2. Check for migrations between the two backend commits. Each deploy commit names its backend commit in a Source-Commit: trailer: take it from the commit you revert and from the previous deploy commit of the same service (git log -- <env>/backend.json in qubital-infra). Then, in qubital-backend:
    git diff --stat <previous Source-Commit> <reverted Source-Commit> -- internal/database/migrations
    If any changed: don't roll back. Fix forward with a new commit and release.
  3. Revert it through a pull request in qubital-infra:
    git revert <deploy commit>
    Revert only the api's commit; revert the broker's only if the broker itself is broken (a broker that knows more than the api uses is harmless).
  4. Get the pull request reviewed and merge it. Northflank applies it, and verify checks it.

Verification​

  • The verify run of the revert commit in qubital-infra is green.
  • The service runs the previous image (Northflank → service → Deployment) and /health answers.

After a prod rollback, the next release still needs a version above the one rolled back.

Escalation​

The team owner.