# Operations runbook

For the person running the platform. Setup: [deploy-production.md](deploy-production.md).
All commands assume the alias from the deploy guide:

```bash
alias vp='docker compose -f /opt/voice/docker-compose.prod.yml --env-file /opt/voice/.env.production'
alias fscli='vp exec freeswitch sh -c '"'"'fs_cli -p "$ESL_PASSWORD" -x "$0"'"'"''
```

## Health at a glance
| Check | Healthy |
|---|---|
| `curl -fsS https://api.example.com/readyz` | `{"status":"ok","checks":{"database":"ok","redis":"ok"}}` |
| `curl -fsS 127.0.0.1:3001/readyz` | `ok` (controller connected to FreeSWITCH over ESL) |
| `fscli 'sofia status'` | profiles `agents` and `carriers` RUNNING, one gateway per carrier (REGED or NOREG for IP auth) |
| `fscli 'show calls count'` | matches *Live calls* in the dashboard |
| Dashboard → Overview | carriers green, no spend-guard banner |
| `vp ps` | every service `Up` / `healthy` |

Metrics (ops profile): Grafana on `127.0.0.1:3003` — reach it with `ssh -L 3003:127.0.0.1:3003 vm`.
Alert rules ([alerts.yml](../infra/prometheus/alerts.yml)): `FreeSwitchDisconnected`, `CarrierUnhealthy`,
`SlowCallSetup`, `WebhookFailures`, `HighCallFailureRate`. To get them out of Prometheus, add an
Alertmanager or use Grafana alerting with your notification channel.

Logs: `vp logs -f --since 10m api controller freeswitch`. Application logs are JSON with
`request_id`/`call_id`; secrets are redacted at the logger.

## Incidents

### Calls fail to start (`POST /v1/calls` accepted, call ends `failed`)
1. Timeline of the call (dashboard → Calls → the call, or `GET /v1/calls/{id}`): the hangup cause tells
   where it failed. `NO_ROUTE_DESTINATION`/`INCOMPATIBLE_DESTINATION` → carrier rejected (wrong caller ID,
   destination not enabled on the carrier account, codecs). `USER_NOT_REGISTERED` → the agent's softphone is offline.
2. Controller readiness (`127.0.0.1:3001/readyz`). If ESL is down, see below.
3. Carrier health in the dashboard; `fscli 'sofia status gateway <gateway name>'`.

### FreeSWITCH / ESL disconnected
The controller reconnects on its own with backoff and **reconciles** on every reconnect: sofia
profiles, ACLs and gateways are re-applied, and calls the database shows as live but FreeSWITCH no
longer has are closed with `hangup_cause: CHANNELS_LOST` (`call.ended` is sent). The same check runs
every minute. If FreeSWITCH stays down: `vp logs freeswitch --tail 200`, then `vp restart freeswitch`.

### FreeSWITCH restart
`vp restart freeswitch` **drops every live call**: they end with `SYSTEM_SHUTDOWN` (graceful stop) or
`CHANNELS_LOST` (crash, closed by the controller's reconcile). Registrations
come back as softphones re-register (≤ 60 s); gateways come back from xml_curl. Do it outside business hours.

### Carrier down / gateway failed
Dashboard shows the carrier as degraded/down; calls through it fail. Steps: check the carrier's
status page; `fscli 'sofia status gateway <name>'`; for registration trunks, a credential change at the
carrier needs **Rotate credentials** in the dashboard. If the carrier has a failover carrier set,
outbound calls already retry there. Inbound calls from an unknown carrier IP are rejected by the ACL
— check the carrier's signaling IPs against the carrier's ACL (dashboard → carrier → technical details).

### Spend guard tripped
Outbound calling stops when today's outbound minutes reach the hard stop; live outbound calls are hung
up and a `usage.alert` (`kind: outbound_minutes_hard_stop`) is sent. Before resuming:
1. Audit log + Calls (filter outbound, today): are the destinations expected? Unknown premium or foreign
   destinations mean a leaked API key or agent credential → rotate those first (below).
2. Raise the limit if the volume is legitimate (Settings → Limits), then **Resume outbound calling**
   (`POST /v1/workspace/spend-guard/reset`). Without raising the limit, the next check trips it again.

### Webhooks failing
Each event is delivered with retries (exponential, ~2 h over 8 attempts). Dashboard → Webhooks →
Deliveries shows status, response code and body per attempt. After fixing the receiving endpoint,
replay failed deliveries (`POST /v1/webhook-deliveries/{id}/replay`, or the Replay button). An endpoint
can be disabled while the CRM is down; events are not queued for disabled endpoints.

### Recordings missing
`recording.ready` comes after the upload. Pending uploads are retried by the controller; files wait in
the `recordings` volume until uploaded (`vp exec controller ls /recordings`). Persistent failures: check
R2 credentials (`S3_*`) and bucket name in the controller logs.

### Stuck channels
`fscli 'show channels'` lists channels; a channel with no call in the dashboard and no audio can be
killed with `fscli 'uuid_kill <uuid>'`. The other way round — a call shown as live without channels —
fixes itself within a minute (`CHANNELS_LOST`). A voicemail whose recording stop was missed that way
leaves its file in the spool: `vp exec controller ls /recordings`.

## Routine tasks

### Key and secret rotation
| Secret | How | Downtime |
|---|---|---|
| API key | Dashboard → API keys → Rotate (`POST /v1/api-keys/{id}/rotate`): new key returned once, old one revoked | Update the CRM first |
| Webhook signing secret | Webhooks → endpoint → Rotate secret | Update the CRM; deliveries signed with the new secret from then on |
| Carrier API key | Carrier → Rotate credentials (validated before it is stored) | None |
| Agent SIP password | `POST /v1/agents/{ref}/reset-password` | Agent re-registers |
| `ENCRYPTION_KEYS` (KEK) | See below | None |
| `INTERNAL_API_SECRET`, `FREESWITCH_ESL_PASSWORD`, `TURN_SECRET`, `REDIS_PASSWORD` | Edit `.env.production`, `vp up -d` | FreeSWITCH restart (live calls drop) |
| `AUTH_SECRET` | Edit `.env.production`, `vp up -d api` | Dashboard users sign in again |

**KEK rotation** (carrier credentials and webhook secrets are encrypted with a per-secret data key,
which is wrapped with the KEK):
1. Add a new key in front: `ENCRYPTION_KEYS=k2:<openssl rand -base64 32>,k1:<old>` and set
   `ENCRYPTION_ACTIVE_KEY_ID=k2`; `vp up -d api controller`. New secrets use k2, old ones still decrypt with k1.
2. Re-wrap everything stored: `vp exec -T api node dist/cli.js keys:rewrap` → `{"rewrapped":N,"alreadyCurrent":M}`.
3. Remove `k1` from `ENCRYPTION_KEYS`; `vp up -d api controller`.

### Backups and restore
- Supabase keeps its own daily backups on paid plans (point-in-time recovery as an add-on).
- The `backup` service additionally dumps the `voice` schema every night at 02:30 UTC to
  `s3://$S3_BUCKET/backups/db/voice-<timestamp>.dump` and keeps 30 days (`BACKUP_KEEP_DAYS`).
- Run one now: `vp exec backup /usr/local/bin/backup.sh`.
- Restore (tested: restored row counts match the source):
  ```bash
  vp exec backup sh -c 'aws s3 cp "s3://$S3_BUCKET/backups/db/<file>.dump" /tmp/r.dump --endpoint-url "$S3_ENDPOINT" \
    && pg_restore --clean --if-exists --no-owner -d "$DATABASE_URL" /tmp/r.dump'
  ```
  Stop api and controller first (`vp stop api controller`), start them afterwards. To inspect a backup
  without touching production, restore into a separate database instead.
- Recordings live in R2 only; use R2's bucket settings (versioning / lifecycle) if you need copies.
- `ENCRYPTION_KEYS` is not in any backup — it lives in your password manager.

### Certificates
`sip.`/`turn.`: certbot renews automatically; the deploy hook restarts FreeSWITCH and coturn (drops
calls, so the timer runs at 04:15). Check: `certbot certificates`. `api.`/`app.`: Cloudflare edge
certificates renew themselves; the Origin Certificate on the VM is valid for 15 years.

### Live carrier check (after connecting a real carrier)
1. Carrier health green; `fscli 'sofia status gateway <name>'` REGED (registration) or NOREG (IP auth).
2. Outbound: dashboard → Test call to your mobile; hear audio both ways; check the CDR and recording.
3. Inbound: call the number from a mobile; it reaches the inbound rule's target.
4. Ordering (Telnyx/Twilio): search, order a cheap number in a country without requirements, release it.

### Capacity
See [load-test-report.md](load-test-report.md). Watch `voice_freeswitch_sessions` and host CPU;
FreeSWITCH needs ~1 core per ~100 transcoded calls. Scale up the VM before going above ~150 concurrent calls.