# Distributed SSE Bus — Design Plan (Auditor T2F, 2026-06-01) > **Status:** Plan only. No code change in this commit. > > **Why this doc lives here:** The auditor flagged the in-process SSE bus + `single-instance-guard.ts` as the architectural ceiling that hard-caps the system at a single replica (worker SPOF) and ~500 monitors. Removing the cap requires a real design pass, not a one-PR refactor. This document captures the proposed substrate, migration path, and explicit non-goals so the work can be picked up later without re-discovering the constraints. --- ## Current state (verified 2026-06-01) | Component | File | Role | |---|---|---| | In-process bus | [`src/lib/server-events.ts`](../../src/lib/server-events.ts) | `EventEmitter` singleton; routes call `emit(event)` and SSE handlers `on(event, ...)`. | | Single-instance lock | [`src/lib/startup/single-instance-guard.ts`](../../src/lib/startup/single-instance-guard.ts) | MySQL `GET_LOCK('uptime_sentinel_worker', 0)` at boot; `process.exit(1)` if held by another instance. | | SSE handler | `src/app/api/sse/dashboard/route.ts` (+ siblings) | Subscribes via `eventBus.on(...)`, streams events to one HTTP client. | **Failure mode if you remove the lock without replacing the bus:** N replicas, N independent EventEmitter instances. Dashboard A only sees events from monitor checks that happened on worker A. Dashboard B sees only B's slice. State diverges. **Performance ceiling note:** The 500-monitor target is downstream of the per-probe DB roundtrips (T2G), not of the SSE bus directly. T2F unlocks **horizontal scale**; T2G raises the **single-replica ceiling**. Both are needed for thousands of monitors. --- ## Goals & non-goals ### Goals 1. Multiple `node server.js` and `npm run worker` instances can run simultaneously. 2. Every SSE-subscribed client sees events emitted on any replica within ≤2s. 3. The notification outbox + monitor-claim logic (already SKIP-LOCKED) keeps working unchanged. 4. Single-source-of-truth event shape — no per-replica drift. ### Non-goals (out of scope for the first cut) - **Multi-region monitoring** — that needs a per-probe-agent dispatch layer on top of SKIP-LOCKED; separate workstream. - **Replacing MySQL** with Postgres. We assume MySQL stays; the substrate has to work with the existing infra. - **Event ordering across replicas** — at-least-once with monotonic-per-replica is fine. The SSE consumers already tolerate dedup. - **Replay on reconnect** beyond a small window — the existing client-side polling fallback already handles long disconnects. --- ## Substrate options (ranked) ### Option A — Redis Pub/Sub (recommended) A standalone Redis instance (or a managed Elasticache / Upstash endpoint). Worker + web replicas all `SUBSCRIBE` to topic channels; emitters `PUBLISH`. The existing `EventEmitter` in `server-events.ts` becomes a thin wrapper that wires `PUBLISH` on emit and `SUBSCRIBE` on first listener. **Pros** - Industry-standard for SSE fan-out at this scale. - Sub-millisecond latency; the `psubscribe` pattern is trivial to map onto the existing event names. - Existing notification outbox can use the same Redis for distributed locks (replacing `single-instance-guard.ts` with a distributed lock primitive — but that's an optional T2F.1). - cPanel hosts often have Redis available as a sibling service. **Cons** - New infrastructure dependency. Operator must provision Redis (and TLS + auth if exposed beyond localhost). - One more thing to monitor. **Migration shape** 1. Add `REDIS_URL` to env validation (`src/lib/env.ts`). 2. Wrap `server-events.ts` so `emit` PUBLISHes to Redis when `REDIS_URL` is set; falls back to the in-process emitter when not (dev / no-redis prod stays single-replica). 3. SSE handlers SUBSCRIBE on first stream open. 4. Remove `single-instance-guard.ts` once REDIS_URL is set in prod (gate behind a `WORKER_DISTRIBUTED=true` flag during rollout). 5. Smoke: spin up two `npm run worker` against the same DB + Redis; assert event from worker A is seen by SSE on worker B. ### Option B — Postgres `LISTEN`/`NOTIFY` via a separate Postgres just for pub/sub Same shape as Option A but using Postgres' built-in pub/sub instead of Redis. Requires adding Postgres to the stack purely for pub/sub. **Verdict: dismiss.** Adds a second DB engine to operate. Redis is simpler. ### Option C — MySQL polling (no new infra) A small `events` table; emitters INSERT; workers poll. No new infra. **Pros** - Zero new infrastructure. - Easy to reason about. **Cons** - Latency = poll interval. At 1s polling, that's 1s p99 added latency on every SSE event. At 100ms it's a steady ~100 SELECTs/s/replica even at idle. - Doesn't replace the `single-instance-guard.ts` lock — only the pub/sub. - More load on the DB which is already the bottleneck per T2G. **Verdict: only as a degraded mode** if Redis is unavailable. Not the recommended primary. --- ## Recommended path **Phase 1 (this work):** Adopt Redis Pub/Sub behind a `REDIS_URL` env var. When unset, the system runs exactly as today (in-process emitter, single-instance lock active). When set, the emitter wraps PUBLISH/SUBSCRIBE and `single-instance-guard.ts` skips its check (gated by `WORKER_DISTRIBUTED=true`). Estimated effort: **~1 week** — one engineer, including local docker-compose for Redis + a smoke that runs two workers + integration test that one worker's monitor-check emits the heartbeat event to the other's SSE stream. **Phase 2 (follow-up, separate PR):** Move the rate limiter (auditor finding #3) onto the same Redis, so it stops collapsing to one global bucket without `TRUSTED_PROXY_CIDRS`. **Phase 3 (follow-up):** Distributed locks for partition maintenance (T2E) so the daily tick doesn't double-fire when multiple workers boot near the same hour. --- ## Open questions for the implementer 1. **Auth model for the in-bundle SSE consumer ↔ Redis** — does the Redis instance accept connections from each Passenger replica via socket or TCP? cPanel-specific configuration; check with Nelson. 2. **Channel naming convention** — flat `events:dashboard:` vs. hierarchical `events..`. Suggest flat with a single subscribe — easier to grep + audit. 3. **Backpressure** — if one SSE client is slow, today we drop events to it. With distributed pub/sub the same per-connection backpressure applies; verify no replica-wide degradation. 4. **Single-instance-guard.ts removal** — defer to Phase 1.5 (after Redis is live and verified). Keep it active during rollout as a safety net. --- ## What we are NOT changing in this plan - The notification outbox (already SKIP-LOCKED, already distributed-safe). - The monitor claim logic (already SELECT ... FOR UPDATE SKIP LOCKED). - The HTTP SSRF guard, AAD on TOTP, region scoping, etc. (separate tier 1 fixes already shipped). - Per-probe DB roundtrips (that's T2G).