# Runbook — Key rotation

**Audience:** Operator with deploy-host access.
**Trigger:** Periodic rotation (recommended every 6 months) OR a credential is suspected leaked.
**SLA:** 30 min hands-on for `NEXTAUTH_SECRET`; 2 hours for `CHANNEL_ENCRYPTION_KEY` because it requires re-encrypting existing rows.

## Keys in scope

| Key | Where | Impact of rotation |
|---|---|---|
| `NEXTAUTH_SECRET` | `.env` | All active sessions invalidated — every user re-logs in |
| `CHANNEL_ENCRYPTION_KEY` | `.env` | Existing `NotificationChannel.config` rows become unreadable unless re-encrypted first |
| `CRON_SECRET` | `.env` | External scheduler must be updated with the new value |
| `DATABASE_URL` | `.env` | Atomic — change DB user password, then update DATABASE_URL, then restart |
| `OPENROUTER_API_KEY` | `.env` | Chat AI returns errors until updated; non-fatal |
| Google OAuth client secret | `.env` | Google sign-in fails until updated |

## NEXTAUTH_SECRET

```bash
# Generate
NEW=$(openssl rand -base64 32)
echo $NEW
```

```bash
# Update .env
sed -i.bak "s|^NEXTAUTH_SECRET=.*$|NEXTAUTH_SECRET=\"$NEW\"|" .env

# Restart
sudo systemctl restart uptime-sentinel-web uptime-sentinel-worker
```

All sessions invalidate immediately. Communicate to users out-of-band ("you'll need to log in again").

## CHANNEL_ENCRYPTION_KEY — KEYRING-based (recommended, no downtime)

**audit3-followup (2026-05-29):** the preferred rotation flow now uses the `CHANNEL_ENCRYPTION_KEYRING` env var. Multiple keys coexist, each identified by a `kid` carried inside the envelope. Rotation is additive — no re-encryption pass required, no downtime.

### When does this apply?

- New deployments should start in KEYRING mode (set `CHANNEL_ENCRYPTION_KEYRING` and `CHANNEL_ENCRYPTION_ACTIVE_KID` from day one).
- Existing deployments can migrate at the next planned rotation; envelopes encrypted under the legacy `CHANNEL_ENCRYPTION_KEY` continue to decrypt forever as long as that var stays set.

### 1. Generate a new key

```bash
NEW_KEY=$(openssl rand -base64 32)
NEW_KID=$(date +%Y-%m)    # e.g. "2026-06"
echo "Add kid=$NEW_KID with key=$NEW_KEY"
```

### 2. Append to KEYRING (don't replace the old one yet)

```bash
# If KEYRING was previously unset, seed it with the legacy key first
# under a stable kid like 'legacy-2024':
#   CHANNEL_ENCRYPTION_KEYRING="legacy-2024:<old base64 key>,2026-06:<new base64 key>"
#
# Otherwise append the new kid:
sed -i.bak "s|^CHANNEL_ENCRYPTION_KEYRING=\"\\(.*\\)\"$|CHANNEL_ENCRYPTION_KEYRING=\"\\1,$NEW_KID:$NEW_KEY\"|" .env
```

### 3. Flip the active kid

```bash
sed -i.bak "s|^CHANNEL_ENCRYPTION_ACTIVE_KID=.*$|CHANNEL_ENCRYPTION_ACTIVE_KID=\"$NEW_KID\"|" .env
```

### 4. Restart (zero downtime — graceful rolling restart works)

```bash
sudo systemctl restart uptime-sentinel-web uptime-sentinel-worker
```

From this moment, **all NEW writes use the new key**. **All OLD reads continue working** via kid lookup against the keyring.

### 5. (Optional) Re-encrypt historical envelopes to the new kid

```bash
# Future-tense — the legacy reencrypt-existing-channels.ts script will
# learn to target kid-based envelopes. Until then, historical envelopes
# stay readable as long as their kid is in CHANNEL_ENCRYPTION_KEYRING.
```

### 6. After all envelopes are re-encrypted under the new kid

Remove the old kid from `CHANNEL_ENCRYPTION_KEYRING` (and from `CHANNEL_ENCRYPTION_KEY` if it was stored there) and restart. Confirm the smoke runner stays green: `npx tsx scripts/smoke/run-all.ts`.

## CHANNEL_ENCRYPTION_KEY — legacy single-key rotation

Use this flow only when not running in KEYRING mode (existing deployments that haven't migrated yet). Naive rotation **silently breaks every existing channel** — always re-encrypt before swapping.

### 1. Decrypt existing rows with the OLD key, write back plaintext

The repo has `scripts/encrypt-existing-channels.ts` for the forward direction. The reverse (decrypt to plaintext) is a one-shot operation — write a small ad-hoc script using `decryptIfNeeded` from `src/lib/crypto/secret-vault.ts`, run inside a transaction:

```bash
npx tsx -e "
import { prisma } from './src/lib/prisma';
import { decryptIfNeeded } from './src/lib/crypto/secret-vault';
const rows = await prisma.notificationChannel.findMany({ where: { deletedAt: null } });
for (const r of rows) {
  const plain = decryptIfNeeded(r.config);
  await prisma.notificationChannel.update({
    where: { id: r.id },
    data: { config: plain }
  });
}
console.log('Decrypted', rows.length, 'channels — stored as plaintext temporarily');
"
```

### 2. Stop the worker + web tier

```bash
sudo systemctl stop uptime-sentinel-web uptime-sentinel-worker
```

### 3. Swap the key in `.env`

```bash
NEW=$(openssl rand -base64 32)
sed -i.bak "s|^CHANNEL_ENCRYPTION_KEY=.*$|CHANNEL_ENCRYPTION_KEY=\"$NEW\"|" .env
```

### 4. Re-encrypt with the new key

```bash
npx tsx scripts/encrypt-existing-channels.ts
```

### 5. Bring services back + verify

```bash
sudo systemctl start uptime-sentinel-web uptime-sentinel-worker
curl -i https://uptime-sentinel.evidenceaction.org/api/health/deep
```

In the web UI, open one notification channel and confirm the saved value renders (decrypts) correctly.

## CRON_SECRET

```bash
NEW=$(openssl rand -hex 32)
sed -i.bak "s|^CRON_SECRET=.*$|CRON_SECRET=\"$NEW\"|" .env
sudo systemctl restart uptime-sentinel-web

# Then update wherever the external scheduler holds the secret (cron-job.org dashboard,
# Cloudwatch Events param, etc.). Until both sides match, the cron endpoint will 401.
```

## DATABASE_URL

```bash
# 1. Rotate the DB user password at the DB level
mysql -h <host> -u root -p \
  -e "ALTER USER 'brandnek_netapp'@'%' IDENTIFIED BY 'NEW-PASSWORD';"

# 2. Update DATABASE_URL — note the password URL-encodes (`:`, `@`, `/` etc. need %XX)
node -e "console.log(encodeURIComponent('NEW-PASSWORD'))"
# Splice that into the URL in .env

# 3. Restart
sudo systemctl restart uptime-sentinel-web uptime-sentinel-worker
```

## After ANY rotation

- Update the rotation log at the bottom of `docs/security/key-rotation-log.md` (create if missing) with: which key, when, who, why.
- If rotation was triggered by a suspected leak, also file the incident in `docs/runbooks/postmortems/` so the root cause is investigated.
- Confirm `npx tsx scripts/smoke/run-all.ts` is green.