# Single-instance deployment (EA-8 / PR-39)

Uptime Sentinel currently runs as a **single app replica** against one
MySQL database. The check loop, SSE bus, and rate limiter are all
in-process. Running two replicas against the same DB produces
silently-degraded behaviour, not an outright failure — which is why
this guard exists.

## What breaks with two replicas (and why we refuse to allow it)

| Subsystem | Two-replica behaviour |
|---|---|
| Monitor check claim loop | Both replicas do `SELECT ... FOR UPDATE SKIP LOCKED` against the same `Monitor` table. Each monitor gets checked once (good), but per-replica claim batches mean each instance handles ~half the workload — fine in principle. However, the cron-vs-worker mode flag is read every 10 ticks, not atomically, so racing replicas can briefly disagree about who runs the work. |
| SSE bus | Each replica has its own in-process EventEmitter. Browsers connect to ONE replica via the load balancer; they see updates from THAT replica's bus, but NOT from the other. Dashboards show stale data for monitors owned by the other replica. |
| Rate limiter | Per-replica in-memory counters. An attacker hitting the load-balanced endpoint at 100 req/s/replica gets 2× the quota. Brute-force resistance evaporates. |
| Notification dedup | In-memory dedup table is per-replica. A flap that triggers in replica A and in replica B fires two outbox rows; the outbox dedup helps but isn't infallible. |

None of these are catastrophic data corruption. They're correctness
degradations that show up as confusing user reports ("I see the
monitor as up on this tab but down on this tab"). That's exactly the
shape of bug operators struggle to diagnose, so we refuse to boot
into the broken state.

## How the guard works

On startup, `instrumentation.ts` calls
`acquireSingleInstanceLock()`. The function opens a Prisma
interactive transaction that:

1. Runs `SELECT GET_LOCK('uptime-sentinel-web', 0) AS got`
2. If the lock is held by another connection (got = 0), throws an
   error which propagates up to `instrumentation.ts`, which calls
   `process.exit(1)` — the container fails to come up.
3. If the lock is acquired (got = 1), the transaction sleeps forever
   inside `await new Promise(() => {})`. As long as the transaction
   is alive, the connection it holds keeps the lock. The lock is
   automatically released by MySQL when the connection drops —
   process death, OOM kill, network failure, anything.

Cost: ONE connection from Prisma's pool is permanently checked out
for lock-holding. Prisma's default pool is sized 1-9 connections; if
this is a concern, raise `connection_limit` in `DATABASE_URL`.

## Operating notes

### Restart on a different node

If the new container starts before the old one's connection has
dropped, the new one will see `GET_LOCK` return 0 and refuse to
start. Wait for the old container to fully terminate, then start the
new one. Typical TCP keepalive timeout is ~10 minutes; force-killing
the old container drops its connection within seconds.

### Disabling the guard temporarily

Set `SINGLE_INSTANCE_GUARD=false` in `.env`. Only valid use: dev
debugging where you genuinely need two processes against one DB and
accept the consequences (you'll see weird dedup behaviour). NEVER
set this in production.

### Things that WOULD bypass the guard (don't do these)

- `docker-compose up -d --scale app=2` — second container exits with code 1
- Kubernetes `replicas: 2` — second pod CrashLoopBackoff
- Two `npm run dev` from different terminals on the same DB — second one exits

### Things that DON'T break the guard

- HMR reloads in dev — the guard is idempotent within a single Node process
- Prisma client recreation — the lock-holding transaction stays alive
- Killing+restarting on the same node — the OS releases the TCP socket, MySQL releases the lock, the new instance can boot

## Roadmap to HA

When EA's monitoring volume justifies horizontal scaling, this guard
must be replaced. The pieces:

1. **SSE bus → Redis pub/sub.** Already designed for this (in-memory
   listener is just a development convenience). ~3 days.
2. **Rate limiter → Redis-backed.** Each replica reads/writes the
   same counter set. ~1 day.
3. **Single-instance guard → leader election.** Replace `GET_LOCK`
   with `SETNX` + TTL in Redis. The leader runs the check loop; the
   followers handle web traffic only. ~2 days.
4. **ADR-014** documenting the transition.

Until that's done, the guard stays.

## Verifying it works on your machine

In one terminal:

```bash
npm run dev
# wait for "Acquired MySQL named lock 'uptime-sentinel-web'"
```

In a second terminal:

```bash
npm run dev
# expect: "[instrumentation] Single-instance lock acquisition failed: ..."
# expect: process exit code 1
```

Stop the first terminal, then run again in the second. It will
acquire the lock cleanly.