Benchmarks · Cosmonic Control
How many sandboxed agents one host runs
Every agent here is a WebAssembly component with its own conversation, its own grants to a model and to its tools, and no network of its own. We start them on one machine, a few thousand at a time, send each one a run, and stop at the first limit.
One host, one harness
Toward a million agents and more
Harnesses
Side by side
The same host, the same model and the same limits, with only the agent changed. Scaled to zero, an agent costs what its conversation and its place in the host cost, so harnesses land close together. Kept warm, an agent keeps its instance between runs, and the harness's own memory decides how many fit.
Footprint
What else changes it
The harness is one axis. Tools are another: each typed tool is its own component in the agent's workload, with its own instance and its own boundary, and a run that calls one pays for it. Here every tool is called once a run, so the numbers are the cost of tools in use.
The ramp
Memory climbs in a straight line
Each point is one step: start the next batch of agents, then send a run to each new one. Host memory grows by the same amount per agent all the way up, and run latency stays flat, so the machine runs out of memory before it runs out of anything else.
Stopping
What a stop gives back
At the end of each ramp the bench stops every agent at once. Stopping is quick, but the host gives almost nothing back by itself: its allocator keeps the freed memory for reuse. Asking the allocator to trim returns more than nine tenths of it to the operating system.
Kubernetes benchmark
One million WorkloadDeployments on one Arm worker
Kubernetes counted one million Ready WorkloadDeployments before and after the chat test. Each deployment completed a chat with the mock model. The benchmark used SQLite for agent sessions and 32 concurrent chat probes.
| Metric | Result |
|---|---|
| Ready WorkloadDeployments | 1,000,000 |
| Successful chats | 1,000,000 |
| Successful chats/s | 461.32 |
| Failed chat attempts | 0 |
| Host pod working set | 40.63 GiB |
| Host pod working set per hosted workload | 42.52 KiB |
Chat throughput is the number of successful chats divided by the time from the initial readiness check to the last successful chat. The memory figure uses the host pod’s cAdvisor working set. The denominator includes the one million benchmark workloads and 2,000 calibration workloads.
GitOps rehearsal
From a Git commit to successful chats
One fleet-size value in Git, reconciled by Argo CD into one Control WorkloadDeployment per agent on the same Arm worker. Every agent must complete a chat run and return the mock model’s expected text. The benchmark then reverts its commit and records memory after the fleet is gone.
| Run | Agents | Commit to verified fleet | Successful chats/s | Pod KiB/agent | Above idle KiB/agent | Revert to zero | Pod working set: active → reverted |
|---|
Loading the measured rehearsals…
Different memory scopes. The lab counts the host process’s anonymous RSS, swap and page tables. These deployed runs count the hostgroup pod’s cAdvisor working set, including file cache. Above-idle memory subtracts the first zero-fleet baseline for the same pod. Nexus sessions, the operator, Argo, the mock model, Cilium and the OS are outside that pod number. The reported rate comes from 16 concurrent chat probes.
Control plane: us-east-2. Worker: us-east-1. Direct Tailscale latency: about 15 ms. The mock model runs on the worker. Codex is pinned by its attested image digest. The stage and upper comparison runs used Control’s built-in trim after 50 stops and five quiet seconds. No manual trim or host restart was used. Memory may remain above the cold baseline.
Raw observations and commit IDs: deployed-runs.json. Fleet chart and Perses dashboard. Reproduce the deployed run.
Method
How we measure
- One host, in one process. The bench starts Cosmonic Control's host with the agent plugins on, and every agent is its own workload, so it gets its own session, its own run subject and its own grants.
- A real conversation store. Sessions live in JetStream, in a NATS server the bench starts beside the host. Its memory is recorded, and is not counted in the host's. The store's size cap is lifted: Control's default, 2 GiB, holds about 340,000 short conversations, and would stop the count before memory does.
- No language model. An OpenAI-compatible mock on loopback returns a short final message. In runs that use tools, it returns a tool call first.
- Memory is what the host holds. Anonymous resident memory plus swap and page tables of the host process, after each step. Per agent is the growth over the host's own baseline, divided by the agents running.
- An agent counts after a successful chat run. The headline uses the last step at which every agent completed its run.
- Stops at the first limit. The memory cap, a failed start or run, or the time budget. The cap leaves room for the NATS server and the operating system. Warm agents also stop when the host's pool of instance slots is full: each one keeps an instance, and the pool's size follows from how much address space each is given.
Data: runs.json, every run's steps and settings.