Roll Back a Deploy
Symptoms
A deploy to staging or prod broke something, and a fix can't ship quickly.
Impact
The service restarts on the previous image. An api rollback across a DB migration fails at boot: the older binary refuses the newer schema.
Steps
- Find the deploy commit in
qubital-infra: a commitdeploy(<env>/backend): api …byqubital-infra-deployer[bot]. A prod release's GitHub Release notes link it. - Check for migrations between the two backend commits. Each deploy commit names its backend commit in a
Source-Commit:trailer: take it from the commit you revert and from the previous deploy commit of the same service (git log -- <env>/backend.jsoninqubital-infra). Then, inqubital-backend:If any changed: don't roll back. Fix forward with a new commit and release.git diff --stat <previous Source-Commit> <reverted Source-Commit> -- internal/database/migrations - Revert it through a pull request in
qubital-infra:Revert only the api's commit; revert the broker's only if the broker itself is broken (a broker that knows more than the api uses is harmless).git revert <deploy commit> - Get the pull request reviewed and merge it. Northflank applies it, and verify checks it.
Verification
- The verify run of the revert commit in
qubital-infrais green. - The service runs the previous image (Northflank → service → Deployment) and
/healthanswers.
After a prod rollback, the next release still needs a version above the one rolled back.
Escalation
The team owner.