Install Cosmonic Control

Benchmarks · Cosmonic Control

How many sandboxed agents one host runs

Every agent here is a WebAssembly component with its own conversation, its own grants to a model and to its tools, and no network of its own. We start them on one machine, a few thousand at a time, send each one a run, and stop at the first limit.

Harness
Tools per agent

Harnesses

Side by side

The same host, the same model and the same limits, with only the agent changed. Scaled to zero, an agent costs what its conversation and its place in the host cost, so harnesses land close together. Kept warm, an agent keeps its instance between runs, and the harness's own memory decides how many fit.

Agents served, scaled to zero
Agents served, kept warm

Footprint

What else changes it

The harness is one axis. Tools are another: each typed tool is its own component in the agent's workload, with its own instance and its own boundary, and a run that calls one pays for it. Here every tool is called once a run, so the numbers are the cost of tools in use.

Memory per agent, scaled to zero
Memory per agent, kept warm

The ramp

Memory climbs in a straight line

Each point is one step: start the next batch of agents, then send a run to each new one. Host memory grows by the same amount per agent all the way up, and run latency stays flat, so the machine runs out of memory before it runs out of anything else.

Mode
Host memory
Run latency, p99

Kubernetes benchmark

One million WorkloadDeployments on one Arm worker

Kubernetes counted one million Ready WorkloadDeployments before and after the chat test. Each deployment completed a chat with the mock model. The benchmark used SQLite for agent sessions and 32 concurrent chat probes.

Measured on one Arm worker
MetricResult
Ready WorkloadDeployments1,000,000
Successful chats1,000,000
Successful chats/s461.32
Failed chat attempts0
Host pod working set40.63 GiB
Host pod working set per hosted workload42.52 KiB

Chat throughput is the number of successful chats divided by the time from the initial readiness check to the last successful chat. The memory figure uses the host pod’s cAdvisor working set. The denominator includes the one million benchmark workloads and 2,000 calibration workloads.

GitOps rehearsal

From a Git commit to successful chats

One fleet-size value in Git, reconciled by Argo CD into one Control WorkloadDeployment per agent on the same Arm worker. Every agent must complete a chat run and return the mock model’s expected text. The benchmark then reverts its commit and records memory after the fleet is gone.

Measured through Kubernetes and Argo CD
RunAgentsCommit to verified fleetSuccessful chats/sPod KiB/agentAbove idle KiB/agentRevert to zeroPod working set: active → reverted

Loading the measured rehearsals…

Different memory scopes. The lab counts the host process’s anonymous RSS, swap and page tables. These deployed runs count the hostgroup pod’s cAdvisor working set, including file cache. Above-idle memory subtracts the first zero-fleet baseline for the same pod. Nexus sessions, the operator, Argo, the mock model, Cilium and the OS are outside that pod number. The reported rate comes from 16 concurrent chat probes.

Control plane: us-east-2. Worker: us-east-1. Direct Tailscale latency: about 15 ms. The mock model runs on the worker. Codex is pinned by its attested image digest. The stage and upper comparison runs used Control’s built-in trim after 50 stops and five quiet seconds. No manual trim or host restart was used. Memory may remain above the cold baseline.

Raw observations and commit IDs: deployed-runs.json. Fleet chart and Perses dashboard. Reproduce the deployed run.

Method

How we measure

  • One host, in one process. The bench starts Cosmonic Control's host with the agent plugins on, and every agent is its own workload, so it gets its own session, its own run subject and its own grants.
  • A real conversation store. Sessions live in JetStream, in a NATS server the bench starts beside the host. Its memory is recorded, and is not counted in the host's. The store's size cap is lifted: Control's default, 2 GiB, holds about 340,000 short conversations, and would stop the count before memory does.
  • No language model. An OpenAI-compatible mock on loopback returns a short final message. In runs that use tools, it returns a tool call first.
  • Memory is what the host holds. Anonymous resident memory plus swap and page tables of the host process, after each step. Per agent is the growth over the host's own baseline, divided by the agents running.
  • An agent counts after a successful chat run. The headline uses the last step at which every agent completed its run.
  • Stops at the first limit. The memory cap, a failed start or run, or the time budget. The cap leaves room for the NATS server and the operating system. Warm agents also stop when the host's pool of instance slots is full: each one keeps an instance, and the pool's size follows from how much address space each is given.

Data: runs.json, every run's steps and settings.