Reliable Delivery

By default an action is best-effort. It gets one attempt over the network. If a message (a frame) is dropped, or the connection dies mid-call, you get an error to handle. That’s the right default for a request/response call you would retry anyway.

Some actions are different. Take game moves, a chat feed, or incremental sync: a stream of updates that must all arrive, in order, with no duplicates. A flaky network shouldn’t lose one, and a reconnect shouldn’t deliver one twice. For those, opt the action into the reliable tier:

shared.ts

export const act_session = act_app.createChildDomain({
  domain: "act_session",
  actions: {
    // Ordered, at-least-once, deduped for the life of a (resumable) connection.
    host_move: actionSchema().input({ schema: v.object({ n: v.number() }) }).reliable(),

    // Same, but the dedup state is *persisted* server-side, so it also survives
    // a server eviction/restart (a Durable Object waking, a redeploy).
    host_events: actionSchema()
      .input({ schema: v.object({ runId: v.string(), events: v.array(v.string()) }) })
      .reliable({ persist: true }),
  },
});

.reliable() is the only delivery setting, and persist is its only parameter. There is no per-message policy to tune. Both sides read the tier from the shared action definition. The only extra data on the wire is a small sequence number on each frame and a cumulative acknowledgement coming back. Actions that don’t opt in are unaffected.

Transport: reliable delivery needs a persistent two-way connection, a duplex carrier such as WebSocket, WebRTC or in-memory. An HTTP exchange has no standing connection to resend on, so .reliable() there is best-effort. Point a reliable stream at a WebSocket carrier.

The same rule applies when a channel falls back mid-stream. Transports are tried in declared preference order for each dispatch. So when a connection with both a WebSocket and an HTTP-exchange carrier loses its socket, dispatches go over the exchange as plain best-effort sends: no seq stamping, no resend window, no dedup. They can also overtake earlier unacknowledged socket sends still held in the outbox.

Those held sends resend when a duplex connection is next established, with their guarantees intact. Exchange frames carry no reliable seq, so the interlude cannot corrupt the stream’s ordering or dedup state. It sits outside the stream, covered only by the idempotent-handler contract. Reliable delivery over the HTTP exchange leg is not built. It is a planned design, waiting on demand from exchange-only deployments.

Direction: reliable delivery applies when the dialing side sends to the listening side (connector to acceptor). A server push (pushToClient / broadcast) of a reliable-declared action is still best-effort. See the direction note below.

What you get — and the one thing you owe

• Ordered — your handler never sees frames out of order. A frame that arrives early waits until the gap before it fills.
• At-least-once — an unacknowledged frame is resent when the transport reconnects, so a drop never loses an update.
• Deduped — for a fireAndForget().reliable() stream, a resent frame the server already received is acknowledged but not delivered to your handler again. For an action with an .output(), it is re-run so its reply regenerates, because the caller may still be waiting on that reply.

The one thing you owe in return comes from at-least-once: reliable handlers must be idempotent. In rare windows the same update can reach your handler twice: a reply lost right at a disconnect, or a delivery deadline that fired as the frame landed. Make handling it twice a no-op, for example upsert by an id or ignore an already-applied move, and every path is safe.

When the promise settles

Opting in changes when the promise settles, because a reliable action keeps retrying instead of failing fast:

• An action with an output resolves when its reply arrives, the same as best-effort. But a transport drop no longer rejects it. The frame is resent and the promise resolves after the reconnect.
• A fire-and-forget action (.fireAndForget().reliable()) resolves on send. Delivery continues in the background across reconnects.
• Either way, every reliable send has a delivery deadline (reliableActionTimeout on connectChannel, default 60s). If the peer hasn’t acknowledged the frame by then, say it’s unreachable, the frame is abandoned. A still-pending action rejects with a reliable_delivery_abandoned error. An already-settled fire-and-forget send reports it through a one-time console warning. So a reliable call always ends, and background delivery never retries forever.

client.ts

connectChannel(clientRuntime, appChannel, {
  peer: serverCoord,
  storage,
  carriers: [wsCarrier(wsUrl)],
  reliableActionTimeout: 30_000, // optional — how long an unacknowledged frame retries (default 60s)
});

Abandoned frames don’t wedge the stream

An abandoned frame can’t block everything behind it. A frame is abandoned when its deadline expires, or when you call running.abort() while it is unacknowledged. The sender drops it, along with any older frames still unacknowledged, each failing loudly the same way. It then tells the receiver to skip past those sequence numbers. The receiver advances, delivers anything it was holding behind the gap, and the stream continues: later sends deliver normally. Ordering among delivered frames is always preserved. An abandoned frame is a reported failure, never a silent hole.

Observing delivery — waitForAck and stream events

A fire-and-forget reliable send resolves on send, so its promise can’t tell you what became of the frame. Two things can.

Per send: hold the RunningAction and await its delivery settlement:

const running = act_session.action.host_move.request(move).run({ streamKey: runId });
await running.waitForAck(); // resolves when the peer's cumulative ack covers this frame;
                            // rejects with the abandon reason (deadline / abort / sweep / stream close)

For a best-effort action waitForAck() mirrors the action itself: it resolves on success and rejects on failure. So generic code needs no branching on the tier. You can also watch runningAction.addUpdateListeners for the reliability update, which fires when reliability.acked flips. The devtools chip’s ✓ is driven by this.

Per connection: no handles needed, for app-level monitoring:

connectChannel(runtime, channel, {
  ...,
  onReliableEvent: (event) => {
    if (event.type === "abandoned") {
      // frames event.fromSeq..event.toSeq of (event.domain/event.actionId [#event.streamKey])
      // were dropped undelivered — re-push from your own records, or show "receiver may be behind".
    }
    if (event.type === "overflow") { /* a send hit the unacked-window cap — coalesce/shed */ }
  },
});

Which failures surface where

For a fire-and-forget reliable send, failures show up in different places on purpose:

What happened	When it surfaces	How
Stream at its unacked cap	at the call	the send’s promise rejects with reliable_outbox_overflow (+ the overflow event)
Peer unreachable right now	not an error	background retry with backoff, until the deadline
Delivery deadline expired	after the promise resolved	waitForAck() rejects, the abandoned event fires, one console warning per route
Explicit abort() / stream close	after the promise resolved	same as deadline, with the abort/close reason

A reply-carrying reliable send is simpler: anything above that would be “silent” instead rejects its still-pending promise loudly.

The at-least-once caveat cuts both ways. A rejected settlement means the sender stopped trying. The ack itself may have been lost after delivery, so an “abandoned” frame may still have arrived. Idempotent handlers make that harmless.

Backpressure

Unacknowledged frames are held until the peer confirms them, and that can’t grow without bound. Each stream caps at 1024 unacknowledged sends. Past the cap, new sends on that stream fail with a reliable_outbox_overflow error. Show it as “connection lost, please retry” rather than silently dropping updates.

You needn’t wait for the cap. The connector exposes read-only pressure stats, so a high-frequency sender can coalesce batches or shed cosmetic traffic before overflowing:

const connector = connectChannel(runtime, channel, { ... });
connector.reliablePending();                                    // total unacked sends, all streams
const p = connector.reliablePending(act_session.action.host_events, runId);
// p.unackedCount / p.oldestUnackedAgeMs / p.maxUnackedPerStream — headroom = max - count

Surviving a server restart

A socket drop and reconnect to a still-running server is the everyday case and needs nothing from you. The client resends what wasn’t acknowledged, the server’s dedup state is intact, and each update is delivered exactly once.

A server restart mid-stream is the harder case: a redeploy, or a Durable Object evicted with its memory gone. The two tiers handle it differently:

• .reliable() (the session tier) repairs the in-flight part by itself. The fresh server notices it has no state for a mid-stream frame and asks the client to re-sync. The client renumbers its unacknowledged frames and resends, and the stream continues in order instead of stalling. What this can’t recover is frames the old server had already confirmed. Those were delivered before the restart and the fresh server doesn’t remember them, so a handler may see them again on replay. Idempotency covers it.
• .reliable({ persist: true }) (the persisted tier) saves the dedup state itself, so the stream survives the restart outright. A replayed frame is recognised and skipped, even across an eviction.

The persisted tier needs somewhere to keep that state. On Cloudflare it’s one line, backed by the Durable Object’s SQLite:

server.ts (Durable Object)

import { serveDurableObject, cloudflareReliableLog } from "@nice-code/action/platform/cloudflare";

const server = serveDurableObject(this.ctx, appChannel, {
  runtime,
  reliableStore: cloudflareReliableLog(this.ctx), // persisted-tier streams survive eviction
  channelCases: { /* … */ },
});

cloudflareReliableLog needs a SQLite-backed DO class (enable it in wrangler). If you omit reliableStore, persist: true actions fall back to behaving like the session tier. On other backends, implement the small IReliableLogStore interface and pass new ReliableLog(store). Both are exported from @nice-code/action/advanced. Storage stays bounded: out-of-order frames are only persisted within a window (maxGapWindow, default 1024), and you can opt into compaction of delivered history with cloudflareReliableLog(ctx, name, { keepDelivered: 256 }).

Independent streams of one action — streamKey

By default a reliable action is one ordered stream per peer: every frame shares one sequence. That’s right when the action is the stream. Sometimes one connection carries several independent streams through the same action, such as one per game run, room or chat channel. A gap in one run shouldn’t hold back the others. Pass a streamKey and each key gets its own stream, with its own ordering, dedup and resending:

client.ts

// One connection, many runs, one action — each run id is its own stream.
function sendRunEvents(runId: string, events: string[]) {
  return act_session.action.host_events.request({ runId, events }).run({ streamKey: runId });
}

sendRunEvents("alpha", a); // stream #alpha — its own sequence
sendRunEvents("beta", b);  // stream #beta — fully isolated from alpha

streamKey is a per-call option on .run() / .runToOutput(). The key is usually only known at runtime (a run id), which is why it isn’t part of the action’s schema. Omit it and you get the single default stream. A gap, an abandonment, or backpressure in one key never affects another.

Because the client chooses the keys, the server caps how many distinct keyed streams one client can open (maxKeyedStreamsPerClient on serveChannel, default 256). The cap counts distinct keys per bound client per action route, on the accepting side. A key stops counting when its stream is closed (below). An idle but unclosed key keeps counting. On the client, each used key holds a small seq-counter entry for the life of the tab. closeReliableStream keeps that entry so the key stays safe to reuse.

Keyed frames use a slightly extended wire format, so both ends need a library version that supports streamKey. Against an older peer you get a one-time dev warning and the keyed frames are not delivered.

Closing a stream — closeReliableStream

Keyed streams are created implicitly on first use. When the thing a stream is about is over (the game run ended, the room was left), close it explicitly:

const connector = connectChannel(runtime, channel, { ... });
// at run teardown:
connector.closeReliableStream(act_session.action.host_events, runId);

One call abandons every still-unacknowledged send on the stream: pending actions reject with reliable_stream_closed, settled fire-and-forget sends stop resending, and the abandoned event fires with the swept range. The receiver is told to skip past them, and both sides release the stream’s state, including the key’s slot in the maxKeyedStreamsPerClient quota.

It is synchronous on the sender’s state: after it returns, nothing from that stream can resend. That ordering matters for multiplexed peers below. Closing a stream you never sent on is a no-op. A closed key can be reused safely. The client keeps the key’s sequence counter, so new sends carry on from where the old ones stopped and can’t be mistaken for duplicates, even if the server never saw the close. A server that did forget the stream re-syncs on the first new send.

Multiplexed peers — many server instances behind one coordinate

The one sharp edge to know about. A reliable stream is identified by (peer coordinate + action route [+ streamKey]), but frames are delivered over whatever transport currently dials that coordinate. On Cloudflare the natural shape is many Durable Object instances behind one peer coordinate, selected by a mutable dial URL (/session/ws?runId=X maps to idFromName(X)). If your dial state changes while unacknowledged frames are still in the outbox, the retries follow the connection into a different physical instance. Run A’s unacked tail can be redelivered into run B’s DO after you switch runs.

Two steps close the window entirely:

1. Close the stream at teardown, before changing the dial state. connector.closeReliableStream(action, runId) is synchronous on the sender’s state, so once it returns nothing from the old run can resend.
2. As a second layer of defence, stamp the payload and guard on the server. Carry the entity id (runId) in every reliable payload and have the instance drop payloads that aren’t its own (input.runId !== ctx.id.name for an idFromName DO). Dropping is safe and terminal, because of the contract below.

An acknowledgement is a receipt, not a verdict on the outcome. The receiver acknowledges a frame when its inbox accepts it, which is ordering and dedup bookkeeping, regardless of what your handler then does with it. A handler that inspects a frame and discards it still settles that frame, and it will not retry. This is guaranteed behavior, and the server-side guard above depends on it. It is also why “acked” never means “the handler liked it”. It only means “it arrived, in order, once”.

What reliability spans — and what it doesn’t

Reliable delivery covers drops and reconnects, not reloads. The outbox holding unacknowledged sends lives in memory, so a page reload or tab close discards it. A reload never redelivers anything cross-run, because the outbox dies with the tab, but nothing is delivered later either. For a send that must survive a reload, such as an end-of-run submission racing a tab close, persist the input yourself and replay it through a normal reliable send on next launch. Replayed sends get fresh seqs, and the idempotent handler you already owe makes the replay safe end-to-end.

Reliable streams are also scoped per runtime boot. The ActionRuntime generates a fresh per-instance id (insId) at construction when you don’t set one. So a reloaded app is a new stream identity, and doesn’t collide with the server’s retained bookkeeping for the old one. Only set insId yourself if it is genuinely unique per process boot.

Streams are scoped per connection too, but only for connections that have their own identity. If one runtime holds several duplex links to the same peer, name them with connectionId. That label is folded into the per-boot insId rather than replacing it, so the boot epoch above is untouched while each link gets its own streams. Without it, two connections sharing a route would feed one inbox stream from two independent sequence counters.

One direction — pushes are best-effort

Reliable delivery applies to the dialing side’s sends (connector to acceptor). The dialing side keeps the resend state, and its process lifetime is the natural bound for it. A server push (pushToClient / broadcast) of a reliable-declared action does not get an outbox or a sequence. It is delivered best-effort, like any other push. You get a one-time dev warning per route, so the direction downgrade is never silent.

That’s deliberate. A push’s receiver is a browser tab or a phone that routinely disappears for longer than any resend window could cover. So a push stream that truly must all arrive needs durable catch-up on the server, not a live-socket resend. Build it from the persisted log: append each push to a per-client ReliableLog, replay contiguousEntries() when the client reconnects, and pruneThrough() once the client confirms. The building blocks (ReliableLog, ReliableInbox, and the stores) are exported from @nice-code/action/advanced. For a single push that needs confirmation, awaiting pushToClient(...).waitForResultPayload() and retrying already gives you at-least-once.

Two backends that each dial the other (worker to worker) get reliable delivery in both directions today, because reliability follows the dialer.

Seeing it work

The devtools panel tags every reliable action’s row with a chip. reliable · 3 shows the frame’s sequence number, and gains a ✓ once the peer confirms delivery. The persisted tier shows persisted · 3. On the serving side, the request logger tags each reliable request with its sequence, the cumulative acknowledgement, and whether it was a deduplicated redelivery:

▶ act_session/host_move  via ws  from envId[frontend]…  reliable seq=3 ack=3
▶ act_session/host_move  via ws  from envId[frontend]…  reliable seq=3 ack=3 (redelivered)

What reliable delivery deliberately does not do

To keep the guarantee free of settings and the library maintainable, these are out of scope by design. None is a dead end: each has a worked recipe in the next section.

• No message priorities. Use priority lanes.
• No per-message TTLs or deadlines. Use TTL classes.
• No exactly-once effects. Use the free idempotency key.
• No ordering across different actions. Use the envelope action.
• No reliable broadcast / reliable push. Use the persisted-log recipe.

Building on top — the recipes

1. Priority lanes

Ordering is per stream, so make a stream per priority class. streamKey: "critical" and streamKey: "bulk" are two lanes of one action, each with its own ordering and backpressure. A slow bulk lane never holds up the critical one (no head-of-line blocking). There is deliberately no ordering across lanes, which is what a priority scheme wants. A scheduler reads pressure per lane with connector.reliablePending(action, "bulk") and sheds or coalesces the cheap lane first.

2. Per-message TTL classes

There is no per-call deadline setting, so compose one from the two levers that exist. Set the connection deadline to your longest class, then use a timer to abort the short-lived class sooner:

// host_end is precious (let it retry ~10 minutes); host_events batches are stale after ~15s.
connectChannel(runtime, channel, { ..., reliableActionTimeout: 10 * 60_000 });

let lastBatch: RunningAction<any, any> | undefined;
function sendBatch(events: string[]) {
  lastBatch = act_session.action.host_events.request({ runId, events }).run({ streamKey: runId });
  const running = lastBatch;
  const ttl = setTimeout(() => running.abort(), 15_000);       // the per-class deadline
  running.waitForAck().then(() => clearTimeout(ttl), () => clearTimeout(ttl));
}

Two facts make this correct. First, an abort sweeps older unacknowledged frames on the same stream with it, since they could no longer deliver in order anyway. So give differing TTL classes their own streamKey lane. Second, different actions are different streams, so host_events aborts never touch host_end. The lastBatch variable is also your teardown handle: aborting only the most recent send abandons the whole unacked tail. Or call closeReliableStream.

For the precious class, remember the lifetime boundary: a 10-minute retry still dies with the tab. A must-arrive payload needs the persist-and-replay pattern too.

3. Exactly-once effects

The wire dedups delivery, but your handler owns the effect. The library gives the handler a free idempotency key for this, made of the receiver-side facts of the frame being handled:

// In a Durable Object case: effect + dedup in ONE transaction, keyed by the library's own (stream, seq).
host_move: (action, conn) => {
  const rel = action.context.reliability; // { seq, streamKey?, redelivered } — undefined for best-effort
  this.ctx.storage.sql.exec(
    `INSERT OR IGNORE INTO effects (stream_key, seq, payload) VALUES (?, ?, ?)`,
    rel?.streamKey ?? "", rel?.seq, JSON.stringify(action.input),
  );
},

For a fire-and-forget reliable stream, a duplicate never reaches your handler at all, because the receiver suppresses it before dispatch. redelivered: true appears only on reply-carrying re-runs, where the re-run regenerates the reply. Use it to return a memoized reply instead of re-running an expensive body. The one duplicate window no flag can cover is a session-tier server reset replay, and the (stream, seq)-keyed transaction above absorbs exactly that.

4. Ordering across event types — the envelope

Sometimes several kinds of update must stay in one order: start, then events×N, then end, all for one entity. Don’t send them as three actions and re-derive tolerance server-side. Send one action whose input is a discriminated union, keyed per entity:

host_record: actionSchema()
  .input({ schema: v.variant("kind", [vStart, vEvents, vEnd]) })
  .fireAndForget()
  .reliable(),
// every send: .run({ streamKey: runId }) — one run = one totally-ordered stream

With one action and one stream per run, start, events and end arrive in exactly that order with zero app-level reordering tolerance, and the handler is a switch (input.kind). A migration from a legacy multi-action protocol should collapse to this shape. If you’re carrying “arrival-order tolerance” code, you’re holding the shape wrong.

5. Reliable server→client push

Reliability applies connector to acceptor (direction). A push stream that must all arrive needs durable catch-up, which also covers what a live-socket resend could give:

// Server: append each push to a per-client log. `append` assigns the seq. Carry it in the payload.
const log = cloudflareReliableLog(this.ctx, "push"); // "@nice-code/action/platform/cloudflare"
const stream = `push::${clientId}`;
const { seq } = log.append(stream, update);
server.pushToClient(client, act_feed.action.entry.request({ seq, update })); // live path, best-effort

// Client: order and dedup with its own inbox (from "@nice-code/action/advanced"), confirm with a normal action.
const inbox = new ReliableInbox<TUpdate>();
function receive(seq: number, update: TUpdate) {
  for (const u of inbox.receive("feed", seq, update).deliver) apply(u);
  act_feed.action.ack.request({ upTo: inbox.contiguousSeq("feed") }).runToOutput();
}
onPush: { entry: ({ seq, update }) => receive(seq, update) }

// Client, on every (re)connect: pull what it missed. `have` is -1 for a fresh inbox (a page reload).
const have = inbox.contiguousSeq("feed");
const { startsAt, entries } = await act_feed.action.resume.request({ have }).runToOutput();
inbox.skipTo("feed", startsAt - 1); // seqs before `startsAt` were pruned, so stop waiting for them
for (const { seq, payload } of entries) receive(seq, payload);

// Server, the `resume` handler: what is left after `have`, each entry with its real seq.
const entries = log.contiguousEntries(stream, input.have);
return { startsAt: entries[0]?.seq ?? log.contiguousSeq(stream) + 1, entries };

// Server, the `ack` handler: drop what the client confirmed.
log.pruneThrough(stream, input.upTo);

The skipTo line matters after a reload. A new ReliableInbox expects seq 0. If the log was pruned, the replay starts later than that, and without the skip the inbox would hold every entry forever, waiting for seqs that no longer exist. On a plain reconnect the inbox is already past startsAt - 1, so the call does nothing. The pruned entries were applied by the page that confirmed them. If that applied state doesn’t survive a reload either, load a snapshot of it before resuming.

For a single push that needs confirmation, skip all of this: await pushToClient(...).waitForResultPayload() and retry.

6. Stream teardown with multiplexed peers

The worked shape of the multiplexed-peers warning:

async function endRun(runId: string) {
  // 1. Close the run's streams FIRST — synchronous; nothing from run A can resend after this.
  connector.closeReliableStream(act_session.action.host_events, runId);
  // 2. Only then flip the dial state the WS carrier derives its URL from.
  activeRun.set(nextRunId);
}

Keep the payload stamp and server-side guard as a second layer of defence. The guard is terminal because acks are a receipt, not an outcome.