# Reliability Core + Quick Wins — Design

**Date:** 2026-06-03
**Status:** Draft for review
**Scope:** P0 (background-job reliability) + P1 (hygiene quick wins) from the external-audit triage.
**Explicitly OUT of scope:** horizontal scaling, HA/multi-node, Redis pub-sub, leader election, multi-tenancy, multi-region probing, status-page white-labeling. Uptime Sentinel is a single-org internal NOC tool for Evidence Action; the single-instance design is correct and stays.

---

## Problem

Periodic background work is implemented as ticks inside the long-lived worker `scripts/worker.ts`:
outbox drain (5s), dedup purge (60s), **escalations** (60s), heartbeat sweep (60s), **scheduled reports** (hourly), **audit-log export** (hourly), rollup (hourly), and **partition maintenance** (daily).

The production deploy (cPanel + Passenger) boots only `server.js`. It does **not** run `npm run worker`. The HTTP cron surface (`/api/cron`, `/api/cron/run-checks`, `/api/cron/cleanup`) drives **only**: monitor checks, heartbeat retention delete, missed-heartbeat detection, and raw→hourly rollup.

Therefore, unless a long-lived worker is actually running on the box, the following silently never run:

| Job | Service today | Consequence if it never runs |
|-----|---------------|------------------------------|
| Outbox drain | `OutboxService` | Enqueued notifications never send — silent alert failure |
| Escalations | `EscalationService` | `PendingEscalation` rows never fire — sev-1 never escalates |
| Scheduled reports | `report-dispatcher` | Reports never dispatched |
| Dedup purge | `DedupService` | `NotificationDedupe` grows unbounded |
| Audit export | `AuditExportService` | Audit archive never written |
| **Partition maintenance** | `PartitionMaintenanceService` | New `Heartbeat` partitions stop being created; inserts fail past the pre-created boundary (~Aug 2026) |

**Step 0 (operational, pending):** confirm whether prod runs the worker or is cron-mode. If cron-mode, this is a live P0. Either way the fix below makes correctness independent of a fragile long-lived process.

A secondary missing-index issue and several hygiene items (P1) ride along.

---

## P0 — Make every periodic job drivable by cron

### Design decision (recommended)
**Add one consolidated, secret-guarded HTTP endpoint — `/api/cron/tick` — that runs the same due-gated tick logic the worker uses, by reusing the existing service functions.** A single cPanel cron entry (`* * * * *`) calls it each minute; internal "is it due" gating (already persisted in `SystemSetting` for rollup/report/etc.) ensures heavy jobs only run on their cadence.

Why this over the alternatives:
- **vs. "supervise the worker" (pm2/systemd):** shared cPanel can't reliably host a long-lived Node process; a crash silently stops everything. Cron is what cPanel does reliably.
- **vs. "one endpoint per job":** more cron entries to misconfigure; the consolidated tick is one cron line and one place to reason about ordering.
- The long-lived worker remains valid and unchanged for anyone who prefers it — the two paths call the **same** service layer.

### Components
- **`src/lib/cron/tick.ts` (new)** — a `runDueTicks()` orchestrator that calls, in order, the already-existing services/functions: `OutboxService` drain → `DedupService` purge → `EscalationService` → `sweepHeartbeatMonitors` → `runRollupTickIfDue` → `dispatchDueReports` → `AuditExportService` → `PartitionMaintenanceService`. Each call is independently try/caught so one failing job never blocks the rest; per-job outcome is collected.
- **`src/app/api/cron/tick/route.ts` (new)** — `GET`, `requireCronSecret`, `dynamic = 'force-dynamic'`, returns a per-job summary `{ job, ran, ok, error? }[]`. Refactor `scripts/worker.ts` to call the **same** `runDueTicks()` so the two paths can never diverge.
- **No change to the monitor-check path** — `/api/cron/run-checks` already exists; `tick` is additive and covers the *other* jobs. (Optional: fold checks into `tick` too, behind the existing `monitoring.mode` gate, so a single cron line covers everything.)

### Data flow
cPanel cron → `GET /api/cron/tick?secret` → `runDueTicks()` → existing services → DB/notification channels. Notifications still flow through the in-process EventBus → `notification-worker` → `OutboxService`; `tick` guarantees the outbox is drained.

### Error handling
Per-job isolation (one try/catch each); failures captured to Sentry + returned in the JSON summary + counted in the existing Prometheus metrics. The endpoint returns 200 with a summary even if some jobs failed (so cron doesn't hammer on transient errors); a top-level 500 only on auth/parse failure.

### Testing
- Unit: `runDueTicks()` invokes every service once and isolates a thrown job (the others still run).
- Smoke: `scripts/smoke/cron-tick-coverage.ts` asserts `tick/route.ts` references every periodic service (structural guard so a future job can't be silently omitted) — registered in `run-all.ts` in the same commit.
- Reuse parity: a test that `scripts/worker.ts` and the route both call `runDueTicks` (no divergent inline logic).

### Deploy doc
Add the required cron line + `monitoring.mode` guidance to the deploy runbook. (Migration-deploy gap remains a separate operational note.)

---

## P1 — Quick wins (high certainty, low risk)

1. **Missing index** — add `@@index([active, nextCheckAt])` to `Monitor` (additive migration `*_monitor_claim_index`). Serves the cron claim scan (`active=1 AND isChecking=0 AND nextCheckAt<=now`). **Apply manually in prod** (deploy does not run `prisma migrate deploy`). Structural smoke asserts the index exists in schema + a migration file.
2. **Delete dead debug probes** — remove `src/app/api/debug/env/route.ts` and `src/app/api/debug/notifications/route.ts` (temporary 2026-05-31 2FA-outage diagnostics). Grep to confirm no references; smoke asserts the routes are gone.
3. **Zod-standardize hand-rolled validation** — `user/profile` (PUT), `user/tokens`, and a sweep of siblings (`user/preferences`, `user/channels`, `user/system-channels`). Add schemas under `src/lib/validations/`; parse at the route boundary; return 400 with the Zod issue. No behavior change for valid input; tighter rejection for invalid.
4. **Orphan-row cleanup** — `NotificationDedupe` (expired) + `PendingEscalation` (resolved/stale) purge. **Subsumed by P0**: these become two of the jobs in `runDueTicks()`, so no separate cron wiring is needed.

---

## Build sequence (PR-sized, cascade-aware)
1. **P1.1 index** (schema + migration + smoke) — isolated; merge first (schema file cascades).
2. **P1.2 debug-route deletion** — isolated.
3. **P0 cron tick** (`tick.ts` + route + worker refactor + smoke) — the core; touches `run-all.ts` + `worker.ts`.
4. **P1.3 Zod sweep** — independent of the above.
5. **P1.4** lands inside P0 (no separate PR).

Each PR: `npx tsx scripts/smoke/run-all.ts` + `npx eslint .` (whole-repo, per the lesson from PR #154) + `tsc` + `jest` before opening.

---

## Open question for review
- **P0 endpoint shape:** consolidated `/api/cron/tick` (recommended) vs. one route per job. Also: should monitor checks fold into `tick` (one cron line total) or stay on the separate `run-checks` route?