# Pre-Launch Hardening — Remediation Plan

> Built from a 16-finding ground-truth verification (each finding investigated + adversarially re-checked against current code). Disposition below reflects what the code actually does *today*, not the audit's original text.

## Disposition matrix

| # | Finding | Verified status | Disposition |
|---|---------|-----------------|-------------|
| 1 | Executive/NOC whole-site hang (pool saturation) | **Already defused** — 60s cache + in-flight single-flight (`route.ts:85-121`) caps builds at ~1/min, ~4 conns; 9 usable can't saturate | **No action** (document) |
| 2 | Settings secret leak to VIEWER | **Half fixed** — secrets masked+encrypted since AUDIT-4/authz; only non-secret SMTP metadata still leaks via un-gated GET | **FIX (Task 1)**: role-gate GET |
| 3 | Executive per-country N+1 | **Real** — serial `statusIncident.count()` loop `route.ts:281-292` | **FIX (Task 2)**: reuse incident query |
| 4 | Executive blank / no error state | **Real** — all 5 screens `if(!data) return null`, no fetch timeout | **FIX (Task 3)**: timeout + error/loading UI |
| 5 | Debug/seed endpoints shipped | **Real** — 3 caller-less routes remain; deletion never landed | **FIX (Task 4)**: delete + smoke |
| 6 | Log PII (emails) in cleartext | **Real** — email sinks + shallow pino redact | **FIX (Task 5)**: mask sinks + redact paths (email only) |
| 7 | AI chat no cost ceiling | **Real** — no `max_tokens`, ≥2 calls/turn | **PARTIAL (Task 6)**: cap `max_tokens` on streaming call; per-user budget → flag |
| 8 | Monitoring can go dark (process model) | **Real, ops** — no in-process loop; worker `exit(1)` expects Docker | **FLAG (ops)** — §A |
| 9 | cron mode drops outbox/escalation/reports/partition jobs | **Real** — worker-only ticks; no cron equiv | **FLAG (design+ops)** — §B |
| 10 | Partition migration auto-applied by deploy.sh | **Real, likely already applied**; guard fix is brittle/could halt deploys | **FLAG (ops verify)** — §C |
| 11 | deploy.sh migrates before build, no backup | **Real, ops** | **FLAG (ops)** — §D |
| 12 | 2FA login gate flag-dependent on primary route | **Real** — bypass iff `AUTH_2FA_ENFORCED` false at runtime | **FLAG (auth sign-off)** — §E; diff ready |
| 13 | Email bypasses outbox (no durability) | **Real, medium risk** — naive routing → duplicate-email storm | **FLAG (design)** — §F |
| 14 | Rate limiter in-process | **Not a security issue** — brute-force is DB-backed; residual is DoS padding | **FLAG (config)** — §G |
| 15 | Retention row-deletes not DROP PARTITION | **Correct as-is** — rollup needs raw; real gap is partition-maint not running | folds into §B |
| 16 | SSE cross-process emit | **Real (mode-dependent)** — worker-mode emits in a different process than the SSE bus | **FLAG** — resolved by mode decision in §A |

---

## Safe batch — implement now (TDD, one PR on `claude/work-prelaunch-hardening`)

### Task 1: Role-gate GET /api/settings
**Files:** `src/app/api/settings/route.ts`; smoke `scripts/smoke/authz-mutation-roles.ts` (add GET assertion, same commit).
- Test first: a route-level assertion that a non-ADMIN role is rejected. (Route handler is hard to unit-test with next-auth; use the structural smoke as the guard + a focused unit test if feasible.)
- Fix: after the session check, `const role = session.user?.role; if (!['ADMIN','ADMIN_READ_ONLY'].includes(role)) return 403`. Matches the settings page guard (`page.tsx:37`) — no legitimate consumer breaks (verifier confirmed the only callers are the ADMIN-gated settings tabs; mobile app never calls it; ApiToken clients already 401).
- Commit: `security(settings): role-gate GET /api/settings read access`.

### Task 2: Kill executive per-country N+1
**Files:** `src/app/api/reports/executive/route.ts`; new pure helper for TDD.
- Extract `tallyIncidentsByCountry(incidents)` (pure) → unit test it (multi-country incident counts once per linked country; incident-level tally, NOT per-monitor — verifier's regression note).
- Reuse the existing 30-day `incidentsByMonitor` query with `include: { countries: {select:{code:true}} }`; build the Map, assign into `regionStats` after it's populated (`route.ts:269-278`); delete the loop at `route.ts:281-292`. Semantically identical, zero added queries.
- Commit: `perf(executive): replace per-country incident N+1 with a single grouped tally`.

### Task 3: Executive error/loading state + fetch timeout
**Files:** `src/lib/executive/useSentinelData.ts`; `src/app/executive/(rail)/layout.tsx`; `src/app/executive/wall/page.tsx`.
- Test first: hook sets `error` when the fetch rejects (mock fetch reject) and does not hang.
- Fix: use `AbortController` + `setTimeout(15000)` (NOT `AbortSignal.timeout` — verifier: NOC kiosk browser may predate it and it would throw synchronously → permanent error). Render an error panel + retry (`refresh()`) and a loading placeholder in the two render sites (rail `<main>`, standalone wall). Gate the green "LIVE / All systems operational" chrome so it doesn't show during `error` (verifier: misleading false-positive is the more dangerous half).
- Commit: `fix(executive): show error/loading state instead of a blank hang`.

### Task 4: Delete debug/seed endpoints
**Files:** delete `src/app/api/debug/env/route.ts`, `src/app/api/debug/notifications/route.ts`, `src/app/api/seed/route.ts` (+ empty dirs); drop stale `api/debug` ref in `src/proxy.ts:242` comment; new smoke `scripts/smoke/debug-endpoints-gone.ts` + register in `run-all.ts` (same commit).
- Grep-confirm zero runtime callers first.
- Commit: `chore(security): remove caller-less debug/seed diagnostic endpoints`.

### Task 5: Mask email PII in logs
**Files:** `src/lib/services/email.service.ts` (sinks ~:202/:203/:224), `src/app/api/user/preferences/route.ts:45` (log id not email), `src/lib/observability/logger.ts` (+ `logger.test.ts` in same commit — export the paths literal to avoid drift).
- Test first: `logger.test.ts` asserts an `email` field is redacted.
- Fix: mask/aggregate recipient emails at the confirmed sinks (log count or masked local-part); add `'email','*.email'` to pino `redact.paths`. **Do NOT** add `url`/`*.url` (verifier: over-redacts operational context) and **do NOT** touch `LOG_LEVEL`/logSmtp gating (design change, out of scope).
- Commit: `security(logging): mask recipient email PII at log sinks`.
- Flag (not code): the ~254 other `console.*` calls + live-box `smtp-debug.log`/Passenger log cleanup on `brandnek@s144`.

### Task 6: Cap AI chat output tokens
**Files:** `src/app/api/chat/route.ts`; smoke assertion.
- Fix: `const MAX_TOKENS = Number.parseInt(process.env.AI_CHAT_MAX_TOKENS || '2048', 10)`; pass `max_tokens: MAX_TOKENS` **only on the second (streaming/prose) call** (`route.ts:354-359`). **Do NOT** cap the first tool-selection call — `JSON.parse(toolCall.function.arguments)` sits outside the try/catch, so a truncated tool payload → 500 (verifier). Generous default 2048 avoids truncating legitimate reports.
- Commit: `security(chat): cap AI chat completion tokens (env AI_CHAT_MAX_TOKENS)`.
- Flag (not code): per-user daily budget needs a DB-count throttle + client-side 429 handling → §H.

### Verify + PR
- `npm test` (touched suites) green; `npx tsc --noEmit` clean; `npx tsx scripts/smoke/run-all.ts` all pass.
- Push `claude/work-prelaunch-hardening`; open PR → main. Do NOT delete `claude/work`.

---

## Flagged items — need your decision or the cPanel box (ready to act on your word)

**§A Monitoring supervision (Blocker).** Decide the mode: (a) run the worker under PM2/systemd with restart, or (b) commit to cron mode. Either way wire an **external dead-man's-switch** onto `/api/health/deep` (it already reports worker liveness/outbox backlog). Verify on the box: current `monitoring.mode`, whether any worker/PM2 runs, whether a probe watches health. This is the #1 go-live blocker.

**§B `/api/cron/tick` (enables §A option b, safely).** I can build an additive, secret-guarded endpoint that runs the idempotent jobs (outbox drain, dedup purge, heartbeat sweep, escalation, reports, rollup) **behind a MySQL `GET_LOCK` advisory lock** so overlapping minute-ticks can't double-send. It must **exclude** partition maintenance (destructive `DROP PARTITION`/`REORGANIZE` — keep operator-triggered in a window) and must **not** run alongside a live worker. Say go and I'll implement it as its own PR. Then you add one cPanel cron line.

**§C Partition migration baseline (Blocker).** Run `npx prisma migrate status` on the live DB. If `20260531000001_heartbeat_partition` shows applied → non-issue. If pending → apply by hand in a window on a restored-backup test first. I did **not** add an auto-guard (verifier: brittle, and it could fail-close every future deploy).

**§D deploy.sh order + backup (ops).** Recommended diff: `mysqldump` before migrate; run `npm run build` **before** `prisma migrate deploy` (keep migrate-before-restart). Ready to write; needs your review since it's the prod deploy script.

**§E 2FA primary-route gate (auth — needs your call).** One-line fix: drop `env.AUTH_2FA_ENFORCED &&` at `login/route.ts:72` so enrolled users are always challenged (matches the legacy path). **But**: if `AUTH_2FA_ENFORCED` is currently false/unloaded in prod, this immediately forces all enrolled users through TOTP — and given the TOTP-secret encryption churn history, any user whose secret can't decrypt + has no backup codes gets locked out. Confirm the live flag value first. Diff is ready.

**§F Email durability via outbox (medium risk).** Not a quick swap: `EmailService.send` loops per-recipient with no idempotency, so naive outbox routing either drops failed recipients or re-sends to all on retry (duplicate-alert storm). Correct fix needs per-recipient outbox rows + payload/decoder changes + rewriting `email-dedup.test.ts`. Worth doing as a dedicated, carefully-tested PR.

**§G Rate limiter (config only).** No code change needed for security (brute-force is DB-backed). Set `TRUSTED_PROXY_CIDRS` on the box for per-client granularity, or move to ingress rate limiting.

**§H AI chat per-user budget (follow-up).** DB-count of today's `ChatMessage` per user → 429 over cap, plus client-side 429 handling in `useChatStream.ts` (currently swallows it). Separate PR.