Tuning NATS Workloads

This page provides reference sizing information for components running against wasmcloud:nats@0.1.0. The numbers here come from a measured campaign against the driver with the following specifications:

• 512Mi host
• NATS 2.12.8 with JetStream
• 186 cells

For more information on NATS workloads and the grant model, see NATS Workloads and Grants.

Size capacity to the burst

subscription-capacity (denominated in messages) most often determines whether a workload loses traffic, offering the following protection:

capacity ÷ (arrival rate − drain rate)  seconds

A handler draining 566 msg/s against 2,000 msg/s arrival, at the stock capacity of 1024, shed 60% of a 10,000-message run. Raising capacity above the whole burst fixed it completely at every admission setting tested. The shed warning prints would_have_absorbed=N, which is the host computing the right value for you.

max-in-flight is inert in that scenario. Sweeping it from 1 to 8192 moved delivery by noise, because the buffer in front of the semaphore is what overflows, not the semaphore.

Above 1 MB, bytes bind before messages

subscription-capacity-bytes defaults to 32 MiB, and admits capacity ÷ payload messages:

payload	messages admitted
16 KiB	2,048
1 MB	32
2 MB	16
5 MB	6

Past roughly 1 MB the byte budget, not the message count, is the binding constraint. Tune subscription-capacity-bytes there and leave subscription-capacity alone.

The acknowledgement window derives from bytes

For JetStream, the host computes:

max_ack_pending = min(max-in-flight × 2, subscription-capacity, capacity_bytes ÷ per_message_bytes)
                  floored at 16

per_message_bytes is the stream's max_msg_size where it sets one, and otherwise the server's max_payload. That is the worst case.

Setting the stream's max_msg_size near your real message size is the cheapest fix available here. It makes the window derive correctly on its own, without pinning anything. Without it, a server max_payload of 64 MiB against 16 KiB messages drives the window to the floor.

The floor of 16 cannot be undercut by capacity alone. Set max-ack-pending explicitly, or set max-in-flight below 8.

The host-wide ceiling

Every subscription on a host shares one backlog ceiling:

host backlog ceiling = guest memory ÷ 4

128 MiB on a 512Mi host. Past it, deliveries shed with reason="host memory budget" rather than a per-subscription reason. Capacity and host memory can only be chosen together.

Instance reuse

poolSize keeps warm instances and reuses them across deliveries, on every delivery path: request/reply, core, JetStream and KV watch.

It is a latency win at small payloads and a memory cost at large ones, because each warm instance retains its heap, so resident memory scales with poolSize × payload. Leave it at 1 for payloads of 1 MB or more unless host memory is sized for that product.

It is also a state contract. Package-level state survives across deliveries on a warm instance, so write handlers that treat it as a cache. Leaving poolSize unset, or 0, declares state ephemeral: a fresh instance per delivery, guaranteed.

Sizing in one place

Given payload M, burst size B and fan-out factor F:

in-flight bytes         ≈ max_ack_pending × M          (JetStream)
per-subscription buffer ≈ subscription-capacity-bytes  (default 32 MiB)
host backlog ceiling    = guest memory ÷ 4             (shared by ALL subscriptions)
fan-out resident        ≈ F × M                        (size the pod from this)
server                  max_payload ≥ M + headers;  max_pending ≥ 10 × M
stream                  max_msg_size ≈ M;  max_bytes ≥ B × M

Five rules that held across every measured cell in the campaign:

1. Set max_payload just above M, never at the 64 MiB cap. The driver reasons from it, it does not merely enforce it.
2. Always set the stream's max_msg_size.
3. Size host memory from the fan-out factor, not the message size.
4. poolSize is a latency win at small payloads and a memory cost at large ones.
5. Never leave the default heap at GiB scale for a component that needs MiB.

Diagnosing

At large payloads the error text usually does not name the cause. Diagnose from the pairing of signatures rather than the loudest one.

Backpressure and capacity

You see	It means	Do
core subscription is shedding … reason="backlog full" … would_have_absorbed=N	This subscription's own buffer filled.	Set subscription-capacity to at least N. If queued_bytes is near capacity_bytes, raise subscription-capacity-bytes instead, or raise the drain rate with poolSize.
shed reason="host memory budget"	The host-wide ceiling, not this subscription.	Raise host memory, lower per-subscription capacity, or split workloads across hosts.
derives max-ack-pending 0 … using 16 instead	The per-message estimate is the server's max_payload, so the window floored. Your true exposure is 16 × real payload.	Set the stream's max_msg_size, or pin max-ack-pending.
slow=0 alongside a large shed_total	Both are correct. slow consumer is the NATS client; shed_total is the driver, and it is windowed.	Read shed_total for shedding. Never read one as the other.

The large-payload signature

You see	It means	Do
A storm of nats: IO error, nats: timed out, disconnected from NATS, consumers rebuilding, and no shed lines	A process is being OOM-killed and the connection is collateral damage. These name NATS, so they read as a network fault. The network is fine.	Confirm with kubectl get pod -o jsonpath='{..lastState.terminated.reason}'. Then reduce in-flight bytes or raise host memory.
More receipts than messages sent	The guest cannot settle within ack-wait, so JetStream redelivers. That is duplication, not loss.	Raise the drain rate with poolSize or max-in-flight, or lower the delivery rate. Raising max-ack-pending alone makes it worse.
Delivery stops mid-run with no error, and the consumer count drops to zero	An ephemeral push consumer idled past 120s and the server reclaimed it.	Use a queue group, which makes the consumer durable: jetstream-subscriptions: STREAM:filter:new:group.
Loss worsening run over run, with several random-named consumers in nats consumer ls	Orphaned ephemeral consumers from earlier crashes, each receiving every message.	Delete them and bounce the workload. A queue group avoids the whole class.

Replicas do not add buffer

Each replica gets its own subscription and its own buffer. Without a queue group, each also receives its own full copy of the traffic. Distribute with a queue group (core-subscriptions: subject:group) rather than by adding replicas.