Rotate a Secret

Symptoms

A secret may be compromised, someone with access to it left, or the provider requires a new one.

Impact

The services that read the group restart (rolling). Some secrets have wider effects, listed in step 4.

Steps

1. Create the new value at the provider (or openssl rand -hex 32 for our own keys). Keep the old one valid until step 5.
2. Northflank → the project → Secrets → the group (api-secrets, realtime-shared, centrifugo-secrets, alloy-secrets) → edit the key → save.
3. Restart dependents from the group page, in the same sitting.
4. Special cases:
   • DB password: it is in two groups, api-secrets (POSTGRES_URL) and centrifugo-secrets (the outbox DSN). Change both before restarting.
   • realtime-shared: each value is there under two names, one for the api and one for the broker; change both names of a pair. The third pair is not a plain copy: CENTRIFUGO_PROXY_SECRET holds the secret, and CENTRIFUGO_CHANNEL_PROXY_SUBSCRIBE_HTTP_STATIC_HEADERS holds {"X-Centrifugo-Proxy-Secret":"<the same secret>"}.
   • TOKEN_ENCRYPTION_KEY: never paste a new value. Follow the key rotation procedure in the backend README.
   • COOKIE_* keys: rotating them signs every user out.
5. Revoke the old value at the provider once the services run the new one.

Never put a value in a template file, a pull request, a chat or a ticket.

Verification

• The restarted services are healthy in Northflank and /health answers.
• The feature that uses the secret works (a sign-in for WorkOS, a call for LiveKit, an upload for R2).

Escalation

The team owner.