Scaling out

1. What one machine holds today

The design's principle is that the agent failing never means dead air: a turn that fails is asked again and then handed to a person; an instance that dies mid-call is replaced by another that resumes the call; a whole service that is down is answered by the carrier's fallback URL, static markup from somewhere independent that apologizes and dials the human queue. The one-machine version of the middle one exists today: when a call's socket drops, the carrier calls /cr-action, and the server reconnects the call with a fresh one-time token, up to RECONNECT_LIMIT times, then transfers it politely. With SESSION_STORE=file:<dir> the call is saved after every turn, so a restarted server on the same machine resumes it; a planned restart hands each call over to the server that starts next, and the carrier's fallback (pnpm fallback) catches a callback that finds no server. With SESSION_STORE=postgres://... several servers share their calls: when the one running a call dies or drains, the carrier's reconnect reaches another, which takes the call over and resumes it (the guide's section 14.14).

Everything a call needs lives in the server's memory or on its disk, by design: laptop mode is first-class, and every production piece arrives as an interface whose default is the laptop implementation. The session stores already have theirs (CallStateStore, ChatStateStore and TokenStore, with memory and file implementations), and one contract suite, runStoreContract, that a shared store must pass unchanged. The table in section 3 names each piece and where it is in the code.

2. The reference shape

• Containers, one image (built: the repository's Dockerfile, and one in every project npm create dialogwright writes; Node 22 under tini as a non-root user, /ready as its health check, a docker stop draining it), in separate deploys by role once roles exist (later). Voice and chat are public; the console is internal only.
• A load balancer that carries WebSockets, with an idle timeout longer than the longest call, and no sticky sessions. Carrier webhooks are stateless HTTP.
• Shared stores: sessions and the audit log in Postgres; traces in object storage; console events over Postgres's LISTEN/NOTIFY. Each is chosen by configuration, one at a time: SESSION_STORE takes memory, file:<dir> or postgres://... today (a Redis store may follow behind the same contract); AUDIT_STORE takes postgres://..., TRACE_STORE takes s3://<bucket>/<prefix> (AWS S3, Cloudflare R2, MinIO), and the console's events go over the session database with SESSION_STORE=postgres://...; unset, each is one machine's, as it was.
• A starter setup on a container platform with WebSockets: two instances and a managed Postgres. Production on containers across availability zones. A second region later, as the carrier's fallback target.
• For regulated data, services covered by the agreements your sector requires, encryption at rest, retention policies and in-region data.
• Locally, a docker compose stack (deploy/compose) with Postgres, an S3-compatible store, two instances and HAProxy, where the kill-an-instance test runs: pnpm multi:test kills (SIGKILL) and stops (SIGTERM, the drain) the server holding a call on Twilio and on Telnyx, and checks the other takes it over with nothing lost (passed on arm64; a weekly and by-hand CI workflow runs it on amd64 once pushed, not yet run). Its in-process version runs in every test run: server/manyServers.e2e.test.ts puts two servers on one Postgres (PGlite, in the process) and one in-process S3 store behind a balancer, kills or drains the one with the call, and the other resumes it, with one trace, one verifying audit chain and one console across both; and server/restart.e2e.test.ts kills one server mid-call and resumes the call on a new process.

3. What has to be shared

Each piece of state one machine keeps today, and what it needs when there are many:

4. A call's WebSocket

A phone call touches the server in three steps, and with several instances each can land on a different one:

1. The carrier posts the voice webhook (/voice/<carrier>). The instance that answers mints a one-time token and answers with a start document that names the socket's URL, with the token in its query.
2. The carrier opens the WebSocket (/conversation/<carrier>?token=...). The token is checked at the upgrade; the call id arrives only afterwards, in the setup frame. The whole call then runs on this one socket, on whichever instance accepted it.
3. If the socket drops (an instance dies, a deploy closes it, a network blip), the carrier posts the action callback (/cr-action/<carrier>) with the call id, and the answer is a new start document with a new token. The reconnected socket may land anywhere.

There are two ways to make that work:

The session in a shared store (the design's choice)

Any instance can take any step: it reads the token, the counter and the session from the store, and the call resumes ("Sorry, I lost you for a moment"). The load balancer needs no affinity. On one machine this is built: a restarted server loads the call from the file store; across instances, from the Postgres store (Phase 8b, built).

Route by call id

Keep state in memory and have the load balancer send every step of a call to the instance that holds it. Today the socket's URL carries only the token, so a balancer cannot see the call id at the upgrade; it would need the call id, or a key derived from it, in the URL. And it saves nothing when the instance holding the call is the one that died, which is the case that matters.

A call is a long-lived socket. The load balancer's idle timeout must be longer than any call (the server ends a session it has held for SESSION_MAX_AGE_MS, two hours by default), and the carrier's own reconnect limit still applies.

5. Web chat across instances

A chat outlives its socket: the client reconnects with the resume token from the last ready and carries on, until CHAT_IDLE_MS of quiet ends it. Today the session lives on the machine that started it (and, with the file store, survives that server's restart), so a resume reaches only that machine. Across instances (built), the session and its resume token live in the shared store, a resume rotates the token there, and a resume that takes a chat over closes the old socket on whichever instance holds it, over the same notices the console uses. The widget needs no change: it already reconnects and resumes with backoff.

6. Idempotent writes

When an instance dies after a tool started a write (booking an appointment, filing a report) but before the session recorded it, the call resumes elsewhere and the turn may run again. So every write tool gets a key derived from the session (the call id, the form, a hash of the confirmed values): a retry with the same key never acts twice. In-flight effects are recorded in the session and re-issued with the same key. Built: a tool that declares idempotent: true gets ctx.idempotencyKey, the same for the same confirmed write; a downstream service request always carries a key, recorded in the call before it is sent and sent again once, with the same key, by a server that resumes a call waiting on it. Still to come: timeouts, retries with those keys, a circuit breaker and caching, declared per tool. Your own systems must honour the key; the engine can only send it.

7. Readiness and drain

Built for one machine: /ready answers 503 once the server stops taking new calls, while /health stays liveness. On SIGTERM or SIGINT it drains for DRAIN_MS (30 seconds by default): a new call goes to the handoff number, a new chat is told it is busy, and live calls go on. With the file store, a call still live at the end is handed over: its carrier is told to wait RESTART_PAUSE_S seconds and connect again, and the stopping server refuses sockets until it exits, so the connection reaches the server that starts next.

The design for a rolling deploy: new instances report ready before they take traffic; old ones stop taking new calls and keep serving the ones they have; after a drain limit (about 15 minutes) they close the remaining sockets, and those calls reconnect to the new version from the shared session (built: a draining server gives each call up as its socket closes, and another takes it at once). What is shown so far is in one process, two servers on one database: a call survives its server crashing or draining. The kill-an-instance test, separate processes on a real Postgres behind a real load balancer, is what will show a deploy never drops a call.

8. The console behind single sign-on

Built for one machine: the console is kept off the internet by CONSOLE_LOCAL_ONLY, or opened to its one owner through the tunnel by CONSOLE_AUTH=token's one-time sign-in link (an HMAC-signed session cookie, rate-limited sign-ins, a strict Content-Security-Policy), with each sign-in and replay in the audit. Across instances it becomes its own role (--role console), internal only, behind single sign-on or a VPN, reading the shared stores. With AUDIT_STORE, the access log of who replayed which call already goes to the shared audit (built). The design adds roles (viewer, supervisor, admin) to the sign-in.

9. Secrets

The server takes every secret from its environment, and prints only each one's length. Give each role only the secrets it uses, from your platform's secrets manager, never baked into an image:

• voice: the carriers' (TWILIO_AUTH_TOKEN, TELNYX_PUBLIC_KEY) and the model's (TYPESAFE_API_KEY, or the gateway's);
• chat: the model's; its sign-in settings (CHAT_JWKS_URL, CHAT_ISSUER, CHAT_AUDIENCE) are configuration, not secrets;
• both, if you use the handoff note: ANTHROPIC_API_KEY;
• the stores' credentials, to the roles that reach them.

Rotate a key by rolling the instances, which the drain above makes safe.

10. Observability

The data to ship already exists, in two formats with stable fields:

Traces (one JSON record per turn, v: 2)

The session and turn, the event, the questions and the model's answers, the gate's rows and the decision, the actions, the slots, timing (retrieveMs, planMs, askMs, resolveMs, totalMs), usage (tokens and costUsd), the config hash and, on the first record, which model answered. They hold what callers said: ship them to storage you treat as personal data.

The audit (one JSON entry per decision)

seq, at, callId, channel, type (call_started, identity, gate, tool_result, handoff, call_ended, ...), a minimized detail, and prevHash and hash. Minimized by design: ids by their last four, never a date of birth or a code.

The design adds OpenTelemetry metrics and traces, logs that carry ids and never personal data, and alerts on fallback hits, handoffs for system failure, the reconnect rate and turn latency, plus a stable event stream derived from the audit for business reporting.

11. The decision model's limits

Every instance asks the same model account, so its rate limits are shared by all of them: size them against your busiest hour, with the provider. Each request has a budget, JEV_TIMEOUT_MS (1500 ms by default); the client retries once within it, so a caller waits up to twice that on a turn the model never answers, then hears the slow-turn line with the question still open. A turn that fails is asked again, then handed to a person. A per-instance cap on requests in flight, so that a surge queues rather than trips the provider's limit, belongs with the resilient tools (design §11), not built yet.

12. The roadmap

Phase 8, local-first on a machine you own, is done: the store interfaces and their contract suite, a file store, resume after a restart, idempotent writes, readiness and drain, retention, service files, carrier fallback documents and console sign-in, proved by the kill-the-server test. Phase 8b, many servers, is done for its trimmed scope: the session store in Postgres passing the same contract suite, with one owner per call, the audit in Postgres, traces on S3-compatible storage, console events over the database, a container image, and a docker compose stack where the kill-an-instance test runs, proved by the in-process two-server test and by pnpm multi:test under Docker. Left for later, as needs show: --role, console roles with single sign-on, OpenTelemetry, a Redis store and hosting guides. Laptop mode stays first-class throughout: the same code runs in every mode, and the contract suite keeps the implementations honest.

The roadmap Production architecture, in the design The foundations Run it on a machine you own