Reliable Delivery

By default an action is best-effort: it rides one transport attempt, and a dropped frame or a mid-flight disconnect surfaces as an error for you to handle. That’s the right default for a request/response call you’ll just retry.

Some actions aren’t like that. A stream of updates that must all arrive, in order, without duplicates — game moves, a chat feed, incremental sync — shouldn’t lose an update to a flaky network or deliver it twice after a reconnect. For those, opt the action into the reliable tier:

shared.ts

export const act_session = act_app.createChildDomain({
  domain: "act_session",
  actions: {
    // Ordered, at-least-once, deduped for the life of a (resumable) connection.
    host_move: actionSchema().input({ schema: v.object({ n: v.number() }) }).reliable(),

    // Same, but the dedup state is *persisted* server-side, so it also survives
    // a server eviction/restart (a Durable Object waking, a redeploy).
    host_event: actionSchema().input({ schema: v.object({ n: v.number() }) }).reliable({ persist: true }),
  },
});

That’s the entire configuration surface: .reliable() is the only delivery knob, and persist its only parameter. There is no per-message policy to tune. Both sides read the tier from the shared action definition, so nothing extra travels on the wire beyond a small per-frame sequence number (and a cumulative acknowledgement coming back). Actions that don’t opt in are completely unaffected.

Transport: reliable delivery works over duplex carriers (WebSocket / WebRTC / in-memory — a persistent connection). Over an HTTP exchange there’s no standing connection to resend on, so .reliable() there is simply best-effort. Point a reliable stream at a WebSocket carrier.

Direction: reliable delivery engages when the dialing side sends to the listening side (connector → acceptor). A server push (pushToClient / broadcast) of a reliable-declared action is still best-effort — see the direction note below.

What you get — and the one thing you owe

• Ordered — your handler never sees frames out of order; an early arrival waits until the gap before it fills.
• At-least-once — an unacknowledged frame is resent when the transport reconnects, so a drop never loses an update.
• Deduped — a resent frame the server already received is acknowledged but not re-delivered to your handler (for a fireAndForget().reliable() stream), or is re-run so its reply regenerates (for an action with an .output() — the caller may still be waiting on that reply).

The one thing you owe in return is the flip side of at-least-once: reliable handlers must be idempotent. In rare windows (a reply lost right at a disconnect, a delivery deadline that fired just as the frame landed) the same update can reach your handler twice. Make handling it twice a no-op — upsert by an id, ignore an already-applied move — and every path is safe.

When the promise settles

Opting in changes when a reliable action’s promise settles, because the whole point is to keep retrying instead of failing fast:

• An action with an output resolves when its reply arrives — same as best-effort, except a transport drop no longer rejects it; the frame is resent and the promise resolves after the reconnect.
• A fire-and-forget action (.fireAndForget().reliable()) resolves on send — delivery continues in the background across reconnects.
• Either way, every reliable send has a delivery deadline (reliableActionTimeout on connectChannel, default 60s). If the peer hasn’t acknowledged the frame by then — say it’s simply unreachable — the frame is abandoned: a still-pending action rejects with a reliable_delivery_abandoned error, and an already-settled fire-and-forget send surfaces through a one-time console warning. So a reliable call always terminates, and background delivery never retries forever.

client.ts

connectChannel(clientRuntime, appChannel, {
  peer: serverCoord,
  storage,
  transports: [{ carrier: wsCarrier(() => ({ url: wsUrl })) }],
  reliableActionTimeout: 30_000, // optional — how long an unacknowledged frame retries (default 60s)
});

Abandoned frames don’t wedge the stream

An abandoned frame (deadline expired, or you called running.abort() while it was unacknowledged) can’t block everything behind it. The sender drops it — along with any older frames still unacknowledged, each failing loudly the same way — and tells the receiver to skip past those sequence numbers. The receiver advances, delivers anything it was holding behind the gap, and the stream continues: later sends deliver normally. Ordering among delivered frames is always preserved; an abandoned frame is a surfaced failure, never a silent hole.

Observing delivery — waitForAck and stream events

A fire-and-forget reliable send resolves on send, so its promise can’t tell you what became of the frame. Two surfaces do:

Per send — hold the RunningAction and await its delivery settlement:

const running = act_session.action.host_move.request(move).run({ streamKey: runId });
await running.waitForAck(); // resolves when the peer's cumulative ack covers this frame;
                            // rejects with the abandon reason (deadline / abort / sweep / stream close)

For a best-effort action waitForAck() simply mirrors the action itself (resolves on success, rejects on failure), so generic code composes without branching on the tier. Observers can also watch runningAction.addUpdateListeners for the reliability update, which fires when reliability.acked flips — the devtools chip’s ✓ rides this.

Per connection — no handles, for app-level monitoring:

connectChannel(runtime, channel, {
  ...,
  onReliableEvent: (event) => {
    if (event.type === "abandoned") {
      // frames event.fromSeq..event.toSeq of (event.domain/event.actionId [#event.streamKey])
      // were dropped undelivered — re-push from your own records, or show "receiver may be behind".
    }
    if (event.type === "overflow") { /* a send hit the unacked-window cap — coalesce/shed */ }
  },
});

Which failures surface where

For a fire-and-forget reliable send, the failure classes are deliberately split:

What happened	When it surfaces	How
Stream at its unacked cap	at the call	the send’s promise rejects with reliable_outbox_overflow (+ the overflow event)
Peer unreachable right now	not an error	background retry with backoff, until the deadline
Delivery deadline expired	after the promise resolved	waitForAck() rejects, the abandoned event fires, one console warning per route
Explicit abort() / stream close	after the promise resolved	same as deadline, with the abort/close reason

A reply-carrying reliable send is simpler: everything above that would be “silent” instead rejects its still-pending promise loudly.

Note the at-least-once caveat both ways: a rejected settlement means the sender stopped trying — the ack itself may have been lost after delivery, so an “abandoned” frame may still have arrived (idempotent handlers make that harmless).

Backpressure

Unacknowledged frames are held until the peer confirms them, and that can’t grow without bound: each stream caps at 1024 unacknowledged sends. Past the cap, new sends on that stream fail with a reliable_outbox_overflow error — surface it as “connection lost, please retry” rather than silently dropping updates.

You don’t have to wait for the cliff: the connector exposes read-only pressure stats, so a high-frequency sender can coalesce batches or shed cosmetic traffic before overflowing:

const connector = connectChannel(runtime, channel, { ... });
connector.reliablePending();                                    // total unacked sends, all streams
const p = connector.reliablePending(act_session.action.host_events, runId);
// p.unackedCount / p.oldestUnackedAgeMs / p.maxUnackedPerStream — headroom = max - count

Surviving a server restart

A socket drop + reconnect to a still-running server is the everyday case and needs nothing from you: the client resends what wasn’t acknowledged, the server’s dedup state is intact, and each update is delivered exactly once.

A server restart mid-stream (a redeploy, a Durable Object evicted with its memory gone) is the harder case, and the two tiers answer it differently:

• .reliable() (the session tier) self-heals the in-flight part: the fresh server notices it has no state for a mid-stream frame and asks the client to re-sync; the client renumbers its unacknowledged frames and resends, and the stream continues in order instead of stalling. What this can’t recover is frames the old server had already confirmed — those were delivered before the restart, and the fresh server doesn’t remember them, so a handler may see them again on replay (idempotency covers it).
• .reliable({ persist: true }) (the persisted tier) persists the dedup state itself, so the stream survives the restart outright — a replayed frame is recognised and skipped, even across an eviction.

The persisted tier needs somewhere to keep that state. On Cloudflare it’s one line — back it with the Durable Object’s SQLite:

server.ts (Durable Object)

import { serveDurableObject, cloudflareReliableLog } from "@nice-code/action/platform/cloudflare";

const server = serveDurableObject(this.ctx, appChannel, {
  runtime,
  reliableStore: cloudflareReliableLog(this.ctx), // persisted-tier streams survive eviction
  channelCases: { /* … */ },
});

cloudflareReliableLog needs a SQLite-backed DO class (enable it in wrangler). If you omit reliableStore, persist: true actions gracefully behave like the session tier. On other backends, implement the small IReliableLogStore port and pass new ReliableLog(store) — both exported from @nice-code/action/advanced. Storage stays bounded: out-of-order frames are only persisted within a window (maxGapWindow, default 1024), and you can opt into compaction of delivered history with cloudflareReliableLog(ctx, name, { keepDelivered: 256 }).

Independent streams of one action — streamKey

By default a reliable action is one ordered stream per peer: every frame shares one sequence. That’s right when the action is the stream. But when one connection multiplexes several independent logical streams through the same action — one per game room, per entity, per chat channel — you don’t want a gap in one room holding back the others. Pass a streamKey and each key gets its own ordered, deduped, independently-resending stream:

client.ts

// One connection, many rooms, one action — each room id is its own stream.
function sendRoomEvent(roomId: string, event: TRoomEvent) {
  return act_room.action.host_event.request(event).runToOutput({ streamKey: roomId });
}

sendRoomEvent("alpha", a); // stream #alpha — its own sequence
sendRoomEvent("beta", b);  // stream #beta — fully isolated from alpha

streamKey is a per-call option on .run() / .runToOutput() — the key is usually only known at runtime (a room id), which is why it isn’t part of the action’s schema. Omitting it gives you exactly the single default stream. A gap, an abandonment, or backpressure in one key never affects another.

Because keys are chosen by the client, the server caps how many distinct keyed streams one client can open (maxKeyedStreamsPerClient on serveChannel, default 256). What counts against the cap: distinct keys per bound client per action route, on the accepting side. A key stops counting when its stream is closed (below); an idle-but-unclosed key keeps counting. Client-side, each used key holds a small seq-counter entry for the life of the tab — closeReliableStream reclaims it.

Keyed frames use a slightly extended wire format, so both ends need a library version that supports streamKey. Against an older peer you get a one-time dev warning and the keyed frames are not delivered.

Closing a stream — closeReliableStream

Keyed streams are created implicitly on first use; when a stream’s real-world subject is over (the game run ended, the room was left), close it explicitly:

const connector = connectChannel(runtime, channel, { ... });
// at run teardown:
connector.closeReliableStream(act_session.action.host_events, runId);

One call: every still-unacknowledged send on the stream is abandoned (pending actions reject with reliable_stream_closed; settled fire-and-forget sends stop resending; the abandoned event fires with the swept range), the receiver is told to skip past them, and both sides release the stream’s state — including the key’s slot in the maxKeyedStreamsPerClient quota.

It is synchronous on the sender’s state: after it returns, nothing from that stream can resend. That ordering is the point — see multiplexed peers below. Closing a stream you never sent on is a no-op, and a closed key can be reused safely (the library self-heals the seq bookkeeping in every crash/race interleaving).

Multiplexed peers — many server instances behind one coordinate

The one sharp edge to know about. A reliable stream is identified by (peer coordinate + action route [+ streamKey]), but frames are delivered over whatever transport currently dials that coordinate. On Cloudflare the natural shape is many Durable Object instances behind one peer coordinate, selected by a mutable dial URL (/session/ws?runId=X → idFromName(X)). If your dial state changes while unacknowledged frames are still in the outbox, the retries follow the connection — into a different physical instance: run A’s unacked tail can be redelivered into run B’s DO after you switch runs.

Two lines close the window entirely:

1. Close the stream at teardown, before changing the dial state — connector.closeReliableStream(action, runId) is synchronous on the sender’s state, so once it returns nothing from the old run can resend.
2. Defense-in-depth: stamp the payload and guard server-side — carry the entity id (runId) in every reliable payload and have the instance drop payloads that aren’t its own (input.runId !== ctx.id.name for an idFromName DO). Dropping is safe and terminal, because of the contract below.

Acknowledgement is a receipt contract, not an outcome contract. The receiver acknowledges a frame when its inbox accepts it — ordering/dedup bookkeeping — regardless of what your handler then does with it. A handler that inspects a frame and discards it still settles that frame: it will not retry. This is guaranteed behavior (the server-side-guard pattern above depends on it), and it’s also why “acked” never means “the handler liked it” — only “it arrived, in order, once”.

What reliability spans — and what it doesn’t

Reliable delivery spans drops and reconnects, not reloads. The outbox holding unacknowledged sends is memory: a page reload or tab close discards it (nothing is ever redelivered cross-run by a reload — the outbox dies with the tab — but nothing is delivered later either). For a send that must survive a reload (an end-of-run submission racing a tab close), persist the input yourself and replay it through a normal reliable send on next launch — replayed sends get fresh seqs, and your idempotent handler (the contract you already signed up for) makes the replay safe end-to-end.

Relatedly, reliable streams are scoped per runtime boot: the ActionRuntime generates a fresh per-instance id (insId) at construction when you don’t set one, so a reloaded app is a new stream identity rather than a collision with the server’s retained bookkeeping for the old one. Only set insId yourself if it is genuinely unique per process boot.

One direction — pushes are best-effort

Reliable delivery engages on the dialing side’s sends (connector → acceptor): the dialing side keeps the resend state, and its process lifetime is the natural bound for it. A server push (pushToClient / broadcast) of a reliable-declared action does not get an outbox or a sequence — it’s delivered best-effort, exactly like any other push (with a one-time dev warning per route, so the direction downgrade is never silent).

That’s deliberate. A push’s receiver is a browser tab or a phone that routinely disappears for longer than any resend window could cover — so a push stream that truly must all arrive needs durable catch-up on the server, not a live-socket resend. Build it from the persisted log: append each push to a per-client ReliableLog, replay contiguousPrefix() when the client reconnects, and prune once the client confirms. (The building blocks — ReliableLog, ReliableInbox, and the stores — are exported from @nice-code/action/advanced.) For a single push that needs confirmation, awaiting pushToClient(...).waitForResultPayload() and retrying already gives you at-least-once.

Two backends that each dial the other (worker ↔ worker) get reliable delivery in both directions today — reliability follows the dialer.

Seeing it work

The devtools panel tags every reliable action’s row with a chip — reliable · 3 (the frame’s sequence number), gaining a ✓ once the peer confirms delivery, or persisted · 3 for the persisted tier. On the serving side, the request logger tags each reliable request with its sequence, the cumulative acknowledgement, and whether it was a deduplicated redelivery:

▶ act_session/host_move  via ws  from envId[frontend]…  reliable seq=3 ack=3
▶ act_session/host_move  via ws  from envId[frontend]…  reliable seq=3 ack=3 (redelivered)

What reliable delivery deliberately does not do

To keep the guarantee knob-free and the library maintainable, these are out of scope by design. That list reads as scoped, not as dead-ends — every item has a worked recipe in the next section:

• No message priorities → priority lanes.
• No per-message TTLs or deadlines → TTL classes.
• No exactly-once effects → the free idempotency key.
• No ordering across different actions → the envelope action.
• No reliable broadcast / reliable push → the persisted-log recipe.

Building on top — the recipes

1. Priority lanes

Ordering is per stream, so make a stream per priority class: streamKey: "critical" and streamKey: "bulk" are independently ordered, independently backpressured lanes of one action — a slow bulk lane never head-of-line-blocks the critical one (cross-lane ordering is intentionally absent; that’s what a priority scheme wants). A scheduler reads pressure per lane with connector.reliablePending(action, "bulk") and sheds or coalesces the cheap lane first.

2. Per-message TTL classes

There is no per-call deadline knob — compose one from the two levers that exist. Set the connection deadline to your longest class, then abort the short-lived class down with a timer:

// host_end is precious (let it retry ~10 minutes); host_events batches are stale after ~15s.
connectChannel(runtime, channel, { ..., reliableActionTimeout: 10 * 60_000 });

let lastBatch: RunningAction<any, any> | undefined;
function sendBatch(events: TEvent[]) {
  lastBatch = act_session.action.host_events.request({ runId, events }).run({ streamKey: runId });
  const running = lastBatch;
  const ttl = setTimeout(() => running.abort(), 15_000);       // the per-class deadline
  running.waitForAck().then(() => clearTimeout(ttl), () => clearTimeout(ttl));
}

Two facts make this correct: an abort sweeps older unacknowledged frames on the same stream with it (they could no longer deliver in order anyway) — so give differing TTL classes their own streamKey lane — and different actions are different streams, so host_events aborts never touch host_end. Note the lastBatch variable is also your teardown handle: aborting only the most recent send abandons the whole unacked tail (or just call closeReliableStream).

For the precious class, remember the lifetime boundary: a 10-minute retry still dies with the tab. A must-arrive payload wants the persist-and-replay pattern too.

3. Exactly-once effects

The wire dedups delivery; your handler owns the effect. The library hands the handler a free idempotency key for it — the receiver-side facts of the frame being handled:

// In a Durable Object case: effect + dedup in ONE transaction, keyed by the library's own (stream, seq).
host_move: (action, conn) => {
  const rel = action.context.reliability; // { seq, streamKey?, redelivered } — undefined for best-effort
  this.ctx.storage.sql.exec(
    `INSERT OR IGNORE INTO effects (stream_key, seq, payload) VALUES (?, ?, ?)`,
    rel?.streamKey ?? "", rel?.seq, JSON.stringify(action.input),
  );
},

Tier nuance: for a fire-and-forget reliable stream, a duplicate never reaches your handler at all (the receiver suppresses it before dispatch) — redelivered: true appears only on reply-carrying re-runs, where the re-run regenerates the reply; use it to return a memoized reply instead of re-running an expensive body. The one duplicate window no flag can cover is a session-tier server reset replay — that’s exactly what the (stream, seq)-keyed transaction above absorbs.

4. Ordering across event types — the envelope

If several kinds of update must stay in one order — start, then events×N, then end, all for one entity — do not send them as three actions and re-derive tolerance server-side. Send one action whose input is a discriminated union, keyed per entity:

host_record: actionSchema()
  .input({ schema: v.variant("kind", [vStart, vEvents, vEnd]) })
  .fireAndForget()
  .reliable(),
// every send: .run({ streamKey: runId }) — one run = one totally-ordered stream

One action, one stream per run: start → events → end arrive in exactly that order with zero app-level reordering tolerance, and the handler is a switch (input.kind). (This is the shape a migration from a legacy multi-action protocol should collapse to — if you’re carrying “arrival-order tolerance” code, you’re holding the shape wrong.)

5. Reliable server→client push

Reliability engages connector→acceptor (direction); a push stream that must all arrive needs durable catch-up, which subsumes what a live-socket resend could give:

// Server: append each push to a per-client log — `append` assigns the seq, carry it in the payload.
const log = cloudflareReliableLog(this.ctx, "push");
const { seq } = log.append(`push::${clientId}`, update);
server.pushToClient(client, act_feed.action.entry.request({ seq, update })); // live path, best-effort

// Client: dedup/order with its own inbox (from `@nice-code/action/advanced`), ack via a normal action.
const inbox = new ReliableInbox<TUpdate>();
onPush: { entry: ({ seq, update }) => {
  for (const u of inbox.receive("feed", seq, update).deliver) apply(u);
  act_feed.action.ack.request({ upTo: inbox.contiguousSeq("feed") }).runToOutput();
}}

// Server, on reconnect: replay the gap-free prefix; prune what the client confirmed.
for (const update of log.contiguousPrefix(`push::${clientId}`)) resend(update);
// on ack: store.deleteThrough(`push::${clientId}`, upTo) — or set `keepDelivered` retention instead.

For a single push that needs confirmation, skip all of this: await pushToClient(...).waitForResultPayload() and retry.

6. Stream teardown with multiplexed peers

The worked shape of the multiplexed-peers warning:

async function endRun(runId: string) {
  // 1. Close the run's streams FIRST — synchronous; nothing from run A can resend after this.
  connector.closeReliableStream(act_session.action.host_events, runId);
  // 2. Only then flip the dial state the WS carrier derives its URL from.
  activeRun.set(nextRunId);
}

Keep the payload stamp + server-side guard as defense-in-depth — the guard is terminal because acks are receipt, not outcome.