Upgrading

Standard upgrade

Check the Behavior changes index and the per-version section for any release between your current version and the target. These flag breaking changes, new values, and potential issues to address before running the recipe below.

Apply the updated CRDs before upgrading the Helm releases; Helm does not upgrade CRDs automatically:

helm show crds oci://ghcr.io/cosmonic/cosmonic-control \
  --version 0.14.0 | kubectl apply -f -

Then upgrade the control plane with the same values file used during installation. If you installed with no values file, as in Get Started, leave out -f:

# Check the currently installed release and chart version
helm list -n cosmonic-system

# Upgrade to a new chart version
helm upgrade cosmonic-control oci://ghcr.io/cosmonic/cosmonic-control \
  --version 0.14.0 \
  --namespace cosmonic-system \
  -f cosmonic-control-values.yaml

Upgrade the HostGroup release separately, reusing the same HostGroup values file you installed with — a bare helm upgrade with no -f resets the release to chart defaults and drops your hostPlugins (including the wasmcloud:nats grants), resources sizing, and any egressPolicy/caBundle/TLS settings:

helm upgrade hostgroup oci://ghcr.io/cosmonic/cosmonic-control-hostgroup \
  --version 0.14.0 \
  --namespace cosmonic-system \
  -f hostgroup-values.yaml

Wait for the rollout to complete before considering the upgrade done:

kubectl rollout status deploy -l app.kubernetes.io/instance=cosmonic-control -n cosmonic-system
kubectl rollout status deploy -l app.kubernetes.io/instance=hostgroup -n cosmonic-system

On a single-node cluster (kind, k3d or k3s), a Traefik rollout stalls with the new pod Pending and the event didn't have free ports for the requested pod ports. The Traefik pod binds host ports 80 and 443, and the rollout starts the new pod before stopping the old one, so the new pod has nowhere to run. Delete the old pod to let it through; requests through Traefik fail until the new pod is ready:

kubectl get pods -n cosmonic-system -l app.kubernetes.io/name=traefik
kubectl delete pod -n cosmonic-system <old-traefik-pod>

Chart version vs. appVersion

Chart versions do not track appVersion one-to-one. A single chart version (e.g. 0.4.1) may ship updated appVersion values as underlying Cosmonic Control images are patched. Check the current appVersion before upgrading:

helm show chart oci://ghcr.io/cosmonic/cosmonic-control --version 0.14.0

Rolling back

Rolling back the Helm release

To roll back to the previous release after a failed upgrade:

helm rollback cosmonic-control -n cosmonic-system
helm rollback hostgroup -n cosmonic-system

helm rollback restores the Helm release to its previous revision. Check the rollback status:

kubectl rollout status deploy -l app.kubernetes.io/instance=cosmonic-control -n cosmonic-system
kubectl rollout status deploy -l app.kubernetes.io/instance=hostgroup -n cosmonic-system

CRD rollback caveat

warning

Helm does not downgrade CRDs on rollback — by design, to prevent accidental data loss. If the failed upgrade included CRD changes, rolling back the Helm release will leave the newer CRD versions in place.

If you need to roll CRDs back to a previous version, apply the old CRD manifests manually:

# Pull CRDs from the target (older) chart version
helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version <previous-version> | kubectl apply -f -

Verify the CRDs were restored before proceeding:

kubectl get crd | grep -E 'cosmonic|wasmcloud'

Behavior changes since v0.4.0

A scan of operator-visible changes by release. Each entry links to the upgrade section below where the change is explained in full. Releases not listed here ship only dependency bumps and internal fixes; see cosmonic/control releases for those.

Version	Change	Action
v0.14.0	Raw-socket egress is enforced by default	Check before upgrading if any workload or host plugin opens raw wasi:sockets connections. Cover each destination in allowedHosts, or keep egressPolicy.socketEgress: count until you have
v0.14.0	Guest memory is enforced by default, and a new process memory limit applies	Check before upgrading if guests run close to the memory budget. Raise resources.limits.memory, or keep hostMemory.guestMemoryMode: count
v0.14.0	hostPath volumes need hostVolumes.allowedHostPaths	Action required for every workload with a hostPath volume, or it fails to start
v0.14.0	Named keyvalue, blobstore and messaging bindings take their backend address from the HostGroup	Check before upgrading if a named hostInterfaces entry sets url, root or prefix. Move those keys to a hostPlugins entry
v0.14.0	Admission policies for tenant namespaces (policies.enabled)	None; opt-in, needs Kubernetes 1.30 or later
v0.14.0	Traefik host ports are optional (ingress.traefik.hostPorts)	Set it to false when cosmonic-system enforces Pod Security baseline or restricted
v0.14.0	The LLM app serves OpenAI's API	None; deploy the published component on a GPU host group to use it
v0.14.0	Wasmtime 48.0.7 and Go 1.27.3 security updates	None; applied on chart bump
v0.13.0	HTTPTrigger moves to control.cosmonic.io/v1alpha2 and no longer carries the workload	Migration required. v1alpha1 is no longer served. Rewrite every trigger as a WorkloadDeployment plus a v1alpha2 HTTPTrigger, and move http-trigger chart installs to chart 0.2.0, before or during the upgrade
v0.13.0	Route conflicts: the oldest trigger now wins	Check before upgrading if you shift traffic by deploying a second trigger on the same host and path. Use backendRefs weights instead
v0.13.0	Autoscalers must target the WorkloadDeployment	Change any HPA or KEDA scaleTargetRef that names an HTTPTrigger
v0.13.0	Hosts check their NATS permissions at startup	Check before upgrading if hosts connect to an external NATS with per-user permissions or no_echo
v0.13.0	HostGroup terminationGracePeriodSeconds default raised from 30 to 150	None unless you override it; raise it further when you add hostPlugins
v0.13.0	Envoy can verify host certificates (envoy.upstreamTLS)	None; opt-in. Recommended for host groups with plugins.http.tls.enabled
v0.13.0	HTTPS on the bundled Traefik (ingress.traefik.tls)	None; opt-in
v0.13.0	Request tracing from the ingress and new Traces dashboards	Set envoy.tracing.enabled: true to populate the Traces dashboards and the ingress panels
v0.12.2	WASI P3 components built on wit-bindgen 0.58 or earlier trap on Wasmtime 48	Rebuild required for affected P3 components, in v0.12.2 and every later release. Check your components before upgrading; the host does not repair them
v0.12.2	Service workloads whose host includes a port are now dropped	Check before upgrading — if any Service workload's host is name:port, drop the port, or the route disappears without an error on the workload
v0.12.2	Outbound client identity for host egress (clientIdentity)	None; opt-in mTLS for components calling a service that requires a client certificate
v0.12.2	Same-host local routing (plugins.http.localBypassRouting)	None; off by default. Read what a locally routed call bypasses before enabling
v0.12.2	Host-plugin egress ceilings and allowedHostLoopbackPorts	None unless you name an egress field on a hostPlugins entry — naming one declares a ceiling the plugin must then enforce
v0.12.2	Guests can no longer reach the host's own ingress through host.wasmcloud.internal	None; security fix
v0.12.2	Envoy replicas and xDS upstream tuning; operator.xdsRefreshInterval deprecated	Move operator.xdsRefreshInterval to operator.xds.refreshInterval; set envoy.replicas above 1 for HA
v0.12.2	HTTPTrigger now serves wasiP3 components	None; bugfix
v0.12.1	xDS route handling hardened	None; a stale route now clears instead of lingering
v0.12.0	cosmonic:kafka moved to binding-scoped clients	Rebuild required for any component using Kafka — the producer resource is gone and consumer.open() takes no arguments
v0.11.0	Socket egress policy (egressPolicy), shipped in preview count mode	None on upgrade — nothing is enforced until you set socketEgress: enforce; plan the count→enforce rollout (and denySpecialRanges for the metadata endpoint) then
v0.11.0	Chart-wide image registry override (global.image.registry); Traefik and its busybox init are now mirrored	Air-gapped/mirrored installs: remove any per-component registry: ghcr.io so the global applies, and mirror the new traefik/busybox tags
v0.11.0	imagePullSecrets now attached to Traefik and the HostGroup pod	None unless your registry needs auth — then set global.image.pullSecrets
v0.11.0	Host OCI registry controls (ociRegistry: allowInsecure, pullTimeout, cacheDir)	None; opt-in
v0.11.0	Connection quotas (connectionQuotas)	None; the host derives a ceiling from its descriptor budget. Watch /readyz for http_ingress_saturated
v0.11.0	TLS for the NATS connections (nexus.tls / dataNats.tls)	None; opt-in for an external or managed NATS
v0.11.0	Native cosmonic:kafka host plugin	Configure on the HostGroup to grant workloads Kafka
v0.9.3	CA-bundle trust extended to component egress (caBundle.egress)	None unless components call a service behind a private CA
v0.9.3	wasmtime 47.0.4 security update (RUSTSEC-2026-0268/0269)	None — security patch, applied on chart bump
v0.9.3	HostGroup memory defaults raised; new hostMemory knobs	Review sizing before upgrade; the larger default request can leave host pods Pending
v0.9.3	Host health-probe listener (/livez, /readyz on 9192)	None needed; set health.enabled: false to disable
v0.9.3	caBundle now trusts a private CA for image pulls	None unless pulling from a registry behind a private CA
v0.9.3	wasmcloud:nats host plugin (core, JetStream, KV)	Configure on the HostGroup to grant workloads NATS access
v0.9.3	Warm-pool idle reclaim (reclaimWindowSeconds, reclaimMinInstances)	None needed; additive and opt-in per component
v0.9.2	control-host image is now a glibc build (unblocks GPU and native host plugins)	None — a host group picks it up on chart bump; only matters if you run the host on a custom base (needs glibc ≥ 2.34)
v0.7.0	Default resource requests and limits on every pod	Review sizing after upgrade; raise or remove limits that don't fit your cluster
v0.7.0	Hardened pod and container securityContexts by default	None unless a container must write outside its mounted volumes
v0.7.0	Baseline NetworkPolicies (networkPolicy.enabled, on by default)	Set networkPolicy.enabled: false if a service mesh manages network policy
v0.7.0	Named host interfaces and hostInterfaces admission validation	Deduplicate hostInterfaces entries that share a namespace:package before re-applying manifests
v0.6.x and earlier	Nexus StatefulSet PVC migration, Traefik-as-default ingress, Console/Cloud removal, CA bundle, unpinned trigger environment	Prefer a fresh install of the current chart; upgrading in place, migrate the nexus-data PVC and drop removed values

Upgrading from v0.6.x or earlier

These are large jumps to the current chart, and each intervening release carried its own CRD and values changes. For a gap this size, a fresh install of the current chart is usually simpler than an in-place upgrade; the cosmonic/control release notes carry the per-version detail. If you do upgrade in place, apply the current CRDs first (see Standard upgrade) and account for the load-bearing changes from that era:

• Nexus persistence moved to a StatefulSet (v0.6.0). The nexus.persistence.existingClaim value was removed. Migrate any pre-v0.6.0 nexus-data PVC onto data-nexus-0 before upgrading, or you lose durable wasi:keyvalue and Artifact JetStream state. Nexus also gained HA through nexus.replicas (an odd number, for RAFT quorum).
• Traefik became the default ingress (v0.4.0 and v0.4.1). Set ingress.provider: istio if you rely on Istio, or ingress.enabled: false to expose Envoy directly.
• Console and Cloud components were removed (v0.4.0). Delete any console: and console_ui: blocks from your values.
• CA bundle support was added (v0.5.0, opt-in), and HTTPTrigger.spec.environment became unpinned by default (v0.5.2); set environment explicitly on triggers that must stay on a tenant-scoped host group.

Then follow the v0.6.1 to v0.7.0 section below, which establishes the current baseline: default resource requests and limits, hardened pod securityContexts, baseline NetworkPolicies, and hostInterfaces admission validation.

Upgrading from v0.6.1 to v0.7.0

Chart 0.7.0 ships appVersion 0.7.0 and bumps the control-host image to wasmCloud v2.5.1. The release hardens every pod both charts deploy — securityContexts, resource limits, probes, and NetworkPolicies are now on by default — and adds admission-time validation plus named routing to hostInterfaces. All of the hardening is overridable through values, but an upgrade of an existing release changes running pods, so review the sections below before running helm upgrade.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.7.0 | kubectl apply -f -

The v0.7.0 CRDs add size bounds on hostInterfaces (at most 64 entries; name and version capped at 64 characters, namespace and package at 128) and two admission rules, enforced via CEL validation on create and update:

1. No two entries may be exact duplicates — the same namespace, package, name, and version.
2. At most one entry of a given namespace:package may be unnamed. The unnamed entry is the default route and cannot be shared; give additional entries of the same package distinct names.

Validation runs at admission, so objects already stored in the cluster are untouched until their next update. Audit any manifests that declare multiple hostInterfaces entries for the same namespace:package before re-applying them — entries that violate either rule are rejected with a descriptive error.

Named host interfaces and hostInterfaces admission validation

The name field on a hostInterfaces entry (present since v0.4.1) is now a routing identity end to end: the wasmCloud v2.5.1 host routes a component's named (implements <name>) imports by it, so a workload can import the same namespace:package more than once and resolve each import to a differently configured backend. Entries that share a namespace, package, and name with semver-compatible versions merge to the highest version, per the component model's canonical-version rules; semver-incompatible versions of the same package coexist as distinct interfaces.

See Named host interfaces for the manifest shape and a worked example.

Pods now ship default resource requests and limits

Every resources block in both charts previously defaulted to {} — pods ran unconstrained unless your values file said otherwise. As of v0.7.0, every container ships a memory request and limit plus a CPU request. CPU is deliberately request-only: a CPU limit only throttles, so none is set.

Upgrading an existing release imposes these limits on all pods. The defaults are sized conservatively for a small cluster; components under real load may need more:

• operator.resources — the informer cache grows with the number of watched objects (Hosts, Workloads, Pods, EndpointSlices). Raise the 192Mi default limit for large clusters.
• prometheus.resources — TSDB memory scales with active series; the default limit is 512Mi.
• HostGroup resources — host pods run Wasm components in-process, so memory scales with the workloads scheduled onto each host. Size limits.memory (default 512Mi) for your expected per-host workload density, or remove the limit to leave host memory uncapped.

Values you already set win over the new defaults. After upgrading, watch for OOMKilled restarts (kubectl get pods -n cosmonic-system will show them) and raise the affected component's limit. See Resource sizing.

Pods are hardened by default

Every container in both charts now runs with a hardened securityContext: runAsNonRoot (UID 65532 for first-party images, 10001 for bundled third-party images), the RuntimeDefault seccomp profile, all capabilities dropped, privilege escalation disabled, and readOnlyRootFilesystem with writable emptyDir volumes mounted where a process legitimately writes (a universal /tmp, the host's HOME, Prometheus's TSDB, Tempo's trace store). Liveness and readiness probes now cover every container, and automountServiceAccountToken is disabled everywhere the Kubernetes API isn't used — it stays on only for the operator, the Perses provisioning sidecar, and Traefik.

Stock installs need no changes. Override any field via values if an extension writes outside its mounted volumes — for example, a HostGroup plugin that writes to the container filesystem:

# hostgroup-values.yaml
securityContext:
  readOnlyRootFilesystem: false

Baseline NetworkPolicies

Both charts gain a networkPolicy.enabled value, on by default. Each enabled component gets a NetworkPolicy whose podSelector matches its pods, restricting ingress to the ports the component actually serves. Egress is intentionally left unrestricted — the operator talks to the Kubernetes API, hosts pull OCI images from arbitrary registries, and components export telemetry to one another — so tightening egress is left to environment-specific policy.

On a CNI without NetworkPolicy support the objects are inert. Set networkPolicy.enabled: false in either chart's values if a service mesh or cluster-level policy engine manages this instead.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.7.0.

Upgrading from v0.9.1 to v0.9.2

Chart 0.9.2 ships appVersion 0.9.2. This is a low-risk patch: there are no CRD or values changes, so stock installs upgrade with the Standard upgrade flow and --version 0.9.2. One behavior change is worth knowing.

note

This guide documents upgrades through v0.7.0 and then v0.9.2 through v0.13.0. If you are coming from v0.8.x, v0.9.0, or v0.9.1, review the cosmonic/control release notes for changes in those releases.

The control-host image is now a glibc build

The control-host release image is now built against glibc (floor 2.34) rather than statically against musl. A static musl build's dlopen is a stub that always fails, which blocked the GPU release image and any host component plugin that loads a native library at runtime. On glibc those load correctly, and this release also unblocks the GPU-enabled control-host image and corrects Perses dashboard routing in the observability stack.

The published image runs on any glibc ≥ 2.34 base (Ubuntu 22.04 and newer), and a host group picks it up automatically when you bump the chart version. This floor matters only if you build or run the host on your own base image, which must then meet it.

Run the Standard upgrade flow with --version 0.9.2.

Upgrading from v0.9.2 to v0.9.3

Chart 0.9.3 ships appVersion 0.9.3 and pins the control-host wash-runtime past wasmCloud v2.8.0 to a revision on main that backs the new native wasmcloud:nats host plugin. This release changes the HostGroup memory defaults, adds a host health-probe listener, extends CA-bundle trust to image pulls, and adds two optional CRD fields. Stock installs upgrade with the Standard upgrade flow and --version 0.9.3, but the memory defaults change what the scheduler places, so review the sections below before running helm upgrade on an install that relies on the chart defaults.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.9.3 | kubectl apply -f -

The v0.9.3 CRDs add two optional fields to a Workload component's warm instance pool: reclaimWindowSeconds (how long the pool watches its own peak concurrency before retiring the warm instances that peak did not need) and reclaimMinInstances (a floor an idle sweep never retires below). Both are additive and preserve today's behavior (a pool that sets neither never reclaims warm instances for idleness), so existing Workloads are untouched until you opt in. See warm instance pools.

HostGroup memory defaults changed

The HostGroup chart now sizes host memory explicitly, and the defaults are larger than before. resources.requests.memory moves from 64Mi to 768Mi and resources.limits.memory from 512Mi to 6Gi, and a new hostMemory block governs how much of the pod the host hands to Wasm guests:

• hostMemory.maxGuestMemory: total guest memory the host may hand out. Unset, the host derives it as three-quarters of resources.limits.memory, so under Kubernetes it tracks the pod automatically; set it only to override.
• hostMemory.defaultHeapMemory (the chart ships 1Gi): the ceiling on any single guest's linear memory, so one runaway component cannot exhaust the pod. Clear this value and the host falls back to wasmtime's 4GiB default, which exceeds the guest budget and makes the host warn at every start. Raising it is safe; lowering it is the change that needs testing.
• hostMemory.coreInstances: instance slots the pooling allocator keeps (wasmtime's default is 1000). Multiplied by defaultHeapMemory this is the pool's virtual reservation, mapped PROT_NONE and committed only as guests touch it, so the node needs nowhere near it and the pod's accounted memory is unaffected.

The host takes three-quarters of resources.limits.memory as its guest budget, reports the resolved numbers at startup (host memory resolved), and warns when defaultHeapMemory does not fit inside it. Because requests.memory is what the scheduler places on, the larger default request means an existing install that keeps the chart defaults will have its host pods request 768Mi and may reschedule onto nodes that can back it. Values you already set win over the new defaults, so an install that had sized resources for its workload density keeps those. Set hostMemory.defaultHeapMemory to fit within that guest budget. After upgrading, check for host pods stuck Pending (kubectl get pods -n cosmonic-system): a Pending host means its larger memory request does not fit the node, so lower resources.requests.memory to suit. The chart's values.yaml documents small/medium/large sizings; see also Resource sizing.

Host health-probe listener

The HostGroup gains a plaintext health listener, on by default, serving /livez and /readyz for the kubelet's liveness and readiness probes:

# hostgroup-values.yaml
health:
  enabled: true
  port: 9192

It is deliberately separate from the workload HTTP ingress, so probes never traverse TLS and a host with no ingress is still probed. Stock installs need no changes; set health.enabled: false only if the listener conflicts with your environment.

CA bundles now apply to image pulls

caBundle previously mounted your extra CA certificates over the container trust store, but the host's OCI client never read that store, so image pulls from a private registry fronted by an internal CA still failed. As of v0.9.3, enabling caBundle also passes --oci-ca-path to the host, so those CAs are honored for every image the host pulls: workload components and host component plugins alike. If you run a private registry with an internal CA, set caBundle.enabled: true and supply the bundle through caBundle.contents (or caBundle.mountPath); no other change is needed.

The wasmcloud:nats host plugin

This release adds a native wasmcloud:nats host plugin (NATS core pub/sub, JetStream, and KV) configured on the HostGroup with deny-by-default subject, stream, and bucket grants and per-binding overrides. It is operator-configured through hostPlugins and requires no new CRD. See The wasmcloud:nats host plugin.

Observability: the Envoy dashboard now populates

This release also fixes the bundled Envoy dashboard in the observability stack: the metrics collector can now reach the Envoy admin endpoint, so the dashboard renders data instead of coming up empty. No action is required, the dashboard definitions themselves are unchanged, and the fix applies on chart bump.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.9.3.

Upgrading from v0.9.3 to v0.10.0

Chart 0.10.0 is a small mainline release. Its two operator-relevant changes (component-egress CA trust and the wasmtime 47.0.4 security fix) were also backported to the release-0.9 line in v0.9.3, so an install already on v0.9.3 gains nothing new here. Both are documented below for anyone upgrading from v0.9.2 or earlier, and this release makes no CRD changes.

CA-bundle trust for component egress

v0.9.3 extended caBundle to the host's image pulls. That same release, and v0.10.0 on mainline, extend it one step further, to the outbound HTTPS that components make (wasi:http). That egress uses a trust store separate from both the OCI client and the container's system store, so enabling caBundle alone does not cover it: a component calling a service behind your private CA still fails its TLS handshake with InvalidCertificate(UnknownIssuer) until you opt in.

# hostgroup-values.yaml
caBundle:
  enabled: true
  contents: ""          # or existingConfigMap / existingSecret
  egress:
    enabled: true       # also trust the mounted bundle for component wasi:http
    trustRoots: ""      # webpki (default) | webpki-and-native | native | extra-only

trustRoots: extra-only trusts exactly the mounted bundle and nothing else; the default (webpki) layers your bundle under the compiled-in public roots. This is additive and off by default, so existing installs render the same egress trust as before.

wasmtime security update

The control-host runtime moves to wasmtime 47.0.4, picking up the fixes for RUSTSEC-2026-0268 and RUSTSEC-2026-0269. It reached both the release-0.9 line (v0.9.3) and mainline (v0.10.0); no configuration change, and the fix applies when you bump the chart.

Run the Standard upgrade flow with --version 0.10.0.

Upgrading from v0.10.0 to v0.11.0

Chart 0.11.0 ships appVersion 0.11.0 and bumps the control-host runtime to the latest wasmCloud. It adds a set of host security and air-gap controls (a raw-socket egress policy, connection quotas, OCI-registry settings, and NATS TLS), makes the chart's image-registry override work across every component, and adds a native Kafka host plugin. Most of it is opt-in. The change that can surprise an existing air-gapped install is the image-registry override, so review that section before running helm upgrade; the new egress policy ships in preview (count) mode and severs nothing until you enforce it.

Apply the updated CRDs

v0.11.0 makes no CRD schema changes; re-applying is a safe no-op, kept in the flow for consistency.

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.11.0 | kubectl apply -f -

Socket egress policy

The host now evaluates a policy on every raw-socket (wasi:sockets) connection a workload or host plugin opens, configured through a new egressPolicy block on the HostGroup. Two parts of it are on by default and change behavior on upgrade; the allowlist-enforcement part is deliberately not.

# hostgroup-values.yaml
egressPolicy:
  socketEgress: ""        # "" = host default (count) | count | enforce
  denySpecialRanges: ""   # on unless set to false
  denyPrivateRanges: ""   # off unless set to true
  allowHostLoopback: ""   # off unless set to true

• The policy ships in count mode, so nothing is severed on upgrade. With socketEgress unset (the default, count), the host evaluates the whole policy (the per-workload allowedHosts allowlist and the range denials below), logs each connection it would refuse (socket egress policy would refuse this connection, at debug level), and allows it anyway. Only enforce actually refuses. Roll it out gradually: run at RUST_LOG=debug for a representative period, confirm nothing legitimate appears in the would-refuse log, then set socketEgress: enforce. Under enforce, a hostPlugins entry that declares no allowedHosts has an empty allowlist and is denied every raw-socket connect, so check each plugin's allowedHosts first. wasi:http egress is gated by allowedHosts regardless of this setting.
• denySpecialRanges is on by default within the policy. Once you enforce, raw-socket connections to loopback, link-local (including the cloud metadata address 169.254.169.254), unspecified, multicast, and documentation ranges are refused. If a workload legitimately reaches the instance metadata endpoint over a raw socket (for example, for cloud IAM credentials), set denySpecialRanges: false before flipping to enforce; the count-period debug log will show those 169.254.169.254 hits so you can decide. Leaving it on is the safer default, since it closes the SSRF-to-metadata path.
• denyPrivateRanges (private IPv4/IPv6 and carrier-grade NAT ranges: RFC1918, ULA, CGNAT) is off by default, because reaching a sibling service on a cluster-private address is the ordinary in-cluster case. allowHostLoopback lets workloads reach the pod's own loopback via host.wasmcloud.internal, and additionally requires the workload's own allowedHostLoopbackPorts entry, so neither operator nor author can open it alone.

Connection quotas

A new connectionQuotas block bounds how many concurrent network connections the host lets its workloads hold (pooled outbound HTTP, raw wasi:sockets, and inbound published ports), rolling up into a host-wide ceiling.

# hostgroup-values.yaml
connectionQuotas:
  maxConnections: ""   # host-wide ceiling; unset derives from RLIMIT_NOFILE

These are bounds, not reservations: nothing is preallocated, so a generous number costs nothing until connections open. Left unset, the host derives the ceiling from its own RLIMIT_NOFILE (half of it, clamped to 64–32768), which tracks the container's real descriptor budget better than the chart can, so most installs should leave it empty. The host reports the resolved numbers at startup (connection quotas resolved), and a host that runs out of inbound slots reports http_ingress_saturated on /readyz rather than pushing the failure onto image pulls.

Host OCI registry controls

A new ociRegistry block governs how the host pulls workload components and host plugins. It is distinct from image.registry (which is where the host pod's own image comes from).

# hostgroup-values.yaml
ociRegistry:
  allowInsecure: ""   # off unless true; plain HTTP for EVERY pull, host-wide
  pullTimeout: ""     # e.g. "30s" / "2m"; unset = host default (30s)
  cacheDir: ""        # e.g. "/tmp/oci-cache"; unset = no caching

allowInsecure is whole-host and all-or-nothing: it downgrades every pull to plain HTTP, so turning it on to reach one in-cluster registry also sends ghcr.io pulls to http://ghcr.io/... (which fail). For the ordinary case of a registry behind a private or in-cluster CA, leave allowInsecure off and use caBundle instead, which keeps verification on and works per-registry. cacheDir should be a directory of its own (the host evicts everything it finds there older than an hour); /tmp/oci-cache is a good choice, and /tmp itself is refused at startup.

Image registry overrides apply to every component

Before v0.11.0, every component in both charts shipped an explicit registry: ghcr.io in values.yaml. Because a non-empty per-component registry always wins over global.image.registry, setting the global had no effect: every component pulled from ghcr.io regardless. The Traefik edge proxy and its busybox init container were worse off: both were hardcoded to docker.io and ignored overrides entirely.

As of v0.11.0, each component's per-component registry defaults to empty ("") in both charts, and the registry a component resolves to follows a clear precedence: the component's own image.registry, then global.image.registry, then the chart default ghcr.io. A fresh install, or one whose values set no component registry, renders exactly the same ghcr.io/... images as before; this only takes effect when you set global.image.registry.

Action for air-gapped and mirrored installs. If your values file was seeded from helm show values on v0.10.0 or earlier (or you copied a component block from an older chart), it carries an explicit registry: ghcr.io on one or more components. That value continues to shadow global.image.registry after the upgrade, with no render-time warning, so pointing the global at your mirror silently leaves those components pulling from ghcr.io. Remove the per-component registry: lines (or set them to ""), then verify before upgrading; every line should show your mirror:

helm template cosmonic-control oci://ghcr.io/cosmonic/cosmonic-control \
  --version 0.14.0 -f cosmonic-control-values.yaml | grep 'image:'
helm template hostgroup oci://ghcr.io/cosmonic/cosmonic-control-hostgroup \
  --version 0.14.0 -f hostgroup-values.yaml | grep 'image:'

Check both charts: the control-host image an air-gapped install most needs to mirror lives in the HostGroup chart, and the registry override applies there too. The Traefik proxy and its init container are now pulled from the mirror like every other component (traefik and busybox under your configured registry) and honor global.image.registry. Air-gapped installs must mirror the two new tags (traefik:v3.6.6 and busybox:1.38.0) to their internal registry alongside the existing images. See Air-Gapped Installation.

imagePullSecrets reach Traefik and the HostGroup

The Traefik Deployment and the HostGroup pod now emit imagePullSecrets, matching the other components. A private mirror that requires authentication no longer leaves those two pods unable to pull. Set the secret once on global.image.pullSecrets (both charts read it); a component's own image.pullSecrets still overrides the global where present.

TLS for the NATS connections

The host can now dial NATS over TLS, configured through a tls block on nexus (the control-plane connection) and on dataNats (the data-plane connection workloads use for messaging). Both default to plaintext, which is what the in-cluster nexus serves, so existing installs are unaffected.

# hostgroup-values.yaml
nexus:
  url: "tls://nats.example.com:4222"
  tls:
    enabled: true
    existingSecret: ""   # Secret (release namespace) holding the material below
    caKey: ""            # key holding the CA-bundle PEM the server chains to (private CA)
    certKey: ""          # client cert, for a mutual-TLS server (set with keyKey)
    keyKey: ""           # client private key
    first: false         # true for a tls_first / handshake_first server

Point it at TLS material only when NATS is external or managed and reachable over a tls:// URL. Set caKey for a server behind a private CA and leave it empty for a publicly-signed one, since a bundle replaces the trust store rather than adding to it (and a kubernetes.io/tls secret carries no CA key). When dataNats.url is empty the data plane inherits nexus.tls; a dataNats.url that names its own server inherits nothing, so set dataNats.tls explicitly in that topology.

The cosmonic:kafka host plugin

v0.11.0 adds a native cosmonic:kafka host plugin, so workloads can produce to and consume from Kafka without a sidecar or bespoke provider. Like wasmcloud:nats, it is operator-configured through hostPlugins with deny-by-default grants and needs no new CRD. Configure it on the HostGroup to grant a workload Kafka access; see Host groups.

wasmCloud runtime bump

The control-host runtime is bumped to the latest wasmCloud release. No configuration change; it applies when you bump the chart.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.11.0.

Upgrading from v0.11.0 to v0.12.0

Kafka moved to binding-scoped clients

This is a breaking WIT change, and components that use cosmonic:kafka must be rebuilt. Kafka clients previously took their broker, credentials and group from arguments the guest supplied, which put those values on the wrong side of the capability boundary. They now come from the workload's resolved hostInterfaces binding, and the guest names none of them.

The interface, now fetched from the digest-pinned ghcr.io/cosmonic-labs/cosmonic/kafka:0.5.0 package:

• The producer resource is gone, along with open, flush and its destructor. send, send-batch, send-stream and the metadata functions are now functions on the binding itself.
• consumer.open(config) becomes consumer.open() — a session with no arguments.
• Transactional sends and offset enlistment move onto transaction.transaction, and send-offsets loses its group argument.
• Ordinary and transactional clients need separate bindings, and use separate native producers even against the same broker.
• Handler subscriptions and dead-letter topics must sit inside the binding's topics grant.
• The default producer queue is 32 MiB, under the existing 100 MiB per-property ceiling.

Client and group configuration moves from guest code to the workload's hostInterfaces binding; see the cosmonic:kafka host plugin for the binding shape.

Upgrading from v0.12.0 to v0.12.2

v0.12.1 and v0.12.2 both carried operator-visible changes, and this section covers both. v0.12.2 also fixes HTTPTrigger so it serves wasiP3 components, which needs no action.

WASI P3 components built on wit-bindgen 0.58 or earlier must be rebuilt

Check this before upgrading. v0.12.2 moves control-host to wasmCloud v2.10.0, which runs on Wasmtime 48, the WebAssembly runtime inside the host. Every later release (v0.12.3, v0.12.4, v0.13.0 and on) carries the same change.

Wasmtime 48 enforces a component model rule (component-model#647, wasmtime#13708): a synchronous cancel of a stream, future or subtask that is still in a waitable set (the group of handles an async component waits on together) now traps. The async runtime in wit-bindgen 0.58 and earlier does exactly that when its inter-task-wakeup feature is on. wit-bindgen 0.59.0 fixed it (wit-bindgen#1638).

Who is affected. WASI P3 components (components that export the async wasi:http/handler@0.3.0 and other WASI 0.3 interfaces) whose async runtime comes from wit-bindgen 0.58 or earlier with inter-task-wakeup enabled. The common case is wasip3 0.7.x with the http-compat feature plus a direct wit-bindgen = "0.57.1" dependency that turns on inter-task-wakeup. WASI P2 components, P3 components on wit-bindgen 0.59 or later, and P3 components without inter-task-wakeup are not affected.

Symptom. The component traps on the first request that waits on I/O, such as reading the request body. Callers get a 500, or a truncated response if the trap fires after the response starts. The host log shows:

wasm trap: waitable cannot be used synchronously while added to a waitable set
failed to invoke component: pooled instance dropped the request

The second line appears only for components that use an instance pool (poolSize).

How to tell before upgrading. Don't rely on the wit-bindgen version in the component's producers metadata (what wasm-tools metadata show prints). A component can report wit-bindgen 0.60 while embedding the 0.57.1 runtime through wasip3 0.7. Check the source project instead:

cargo tree --target all -e features -i wit-bindgen

The component is affected if this lists a wit-bindgen version of 0.58 or earlier with the feature inter-task-wakeup. If the lockfile carries more than one wit-bindgen, name the version, for example -i wit-bindgen@0.57.1. A P2 component that pulls wit-bindgen 0.57 in through wasip2 is not affected.

Fix. Rebuild on wasip3 0.9.0 and wit-bindgen 0.62.0, and keep the direct wit-bindgen dependency at the same version wasip3 uses so Cargo builds one runtime with those features:

wasip3 = { version = "0.9.0", features = ["http-compat"] }
wit-bindgen = { version = "0.62.0", default-features = false, features = [
    "async",
    "async-spawn",
    "inter-task-wakeup",
] }

Coming from wit-bindgen 0.57, rename wit_bindgen::spawn to spawn_local; that is the only source change most components need. A rebuilt component also runs on hosts still on Wasmtime 47, so you can rebuild and redeploy before upgrading the chart.

Components published by others need a fixed release from their publisher. For example, wasmCloud's oci-registry 0.1.0 is affected and 0.2.0 is fixed.

Service hosts must not carry a port

Check this before upgrading. A Service workload whose host is name:port is now dropped by hostname validation. The route disappears and nothing on the workload reports an error. Use a plain hostname.

Outbound client identity

The host can present a client certificate on outbound HTTPS, so a component can call a service that requires mutual TLS:

# hostgroup-values.yaml
clientIdentity:
  secret: ""            # a kubernetes.io/tls secret
  refreshInterval: 30s  # re-read the pair on this interval

The identity is host-wide: every workload on that host group presents the same certificate, and there is no per-workload client identity. A missing half of the pair, a refresh interval with no certificate, or a zero interval each fail host startup rather than starting without the credential.

Same-host local routing

plugins.http.localBypassRouting lets an HTTP call between two workloads on the same host skip the network and dispatch in memory. It is off by default, and the host logs a warning while it is on.

Read the trade before enabling. A locally routed call bypasses ingress auth, rate limits, mesh mTLS and NetworkPolicy, and because dispatch is in-memory there is no TLS on an https:// target. A localRoute entry is a claim rather than a proof of ownership — any workload on the host may claim any hostname. Locally routed calls still count against the host's connection quotas.

Host-plugin egress ceilings

A hostPlugins entry now accepts allowedHosts, allowedIpNameLookups and the new allowedHostLoopbackPorts. Naming any of them declares a ceiling the plugin is then required to enforce, and a plugin that cannot enforce one fails host startup with a message naming it. An entry that names none stays unmanaged, so this changes nothing on upgrade unless you opt in.

Note what the ceiling is and is not: a native plugin checks the endpoint it is about to dial. That is a declaration handshake, not runtime interception, so a native client's own DNS and socket operations are not filtered.

Host-owned ports are closed to guests

The socket policy now carries the host's own ports, so a workload granted allowHostLoopback can no longer reach the host's ingress listener through host.wasmcloud.internal. Security fix; no configuration.

Envoy HA and xDS tuning

# control-values.yaml
envoy:
  replicas: 1                      # >1 also creates a PodDisruptionBudget
operator:
  xdsRefreshInterval: ""           # DEPRECATED - moved to xds.refreshInterval
  xds:
    refreshInterval: ""            # default 30s
    upstreamConnectTimeout: ""     # blank = Envoy's default of 5s
    upstreamIdleTimeout: ""        # default 20s
    upstreamMaxConnections: ""     # default 1024
    upstreamMaxPendingRequests: "" # default 1024
    upstreamMaxRequests: ""        # default 1024

operator.xdsRefreshInterval still works but is deprecated. Both it and its replacement default to 30 seconds, so a blank value needs no migration; a set value should move to operator.xds.refreshInterval.

xDS route handling

Route registration was hardened in v0.12.1 and again in v0.12.2. When two workloads claim the same route and path, the newer one wins by timestamp. A route is now unregistered when the workload or host changes state, and when a host carries a deletion timestamp, rather than waiting on the host finalizer. The effect is that a stale route clears instead of lingering and serving a workload that has gone.

Upgrading from v0.12.2 to v0.13.0

Chart 0.13.0 ships appVersion 0.13.0. This section also covers v0.12.3 and v0.12.4, which changed no CRDs or chart values: they moved the control-host runtime to wasmCloud v2.10.1 and then v2.10.3, and updated dependencies.

This release changes the HTTPTrigger API in a way that breaks existing manifests. HTTPTrigger is now served only as control.cosmonic.io/v1alpha2, and a trigger no longer carries the workload it routes to. Plan the migration below before running helm upgrade. The rest of the release is opt-in, apart from two changes on the HostGroup chart: a NATS permission check at startup and a longer default shutdown grace period.

If you are upgrading from v0.12.1 or earlier, also read WASI P3 components built on wit-bindgen 0.58 or earlier must be rebuilt. It applies to v0.13.0 unchanged.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.13.0 | kubectl apply -f -

The HTTPTrigger CRD gains the v1alpha2 version and makes it the storage version. v1alpha1 stays in the CRD but is no longer served, so once these CRDs are applied the API server rejects any v1alpha1 HTTPTrigger manifest. Work through Migrate your triggers before you apply them: export your existing triggers and delete them first, or follow Migrate without dropping traffic. No other CRD changes.

The first kubectl get httptriggers you run right after the apply can fail with the server could not find the requested resource, because kubectl's cached API discovery still lists v1alpha1. Run it again.

HTTPTrigger v1alpha2 references a WorkloadDeployment

In v0.12 and earlier, an HTTPTrigger held a whole workload: its template, replicas and environment sat next to the ingress rule, and the operator created a WorkloadDeployment from them and added the inbound wasi:http interface for you. In v0.13.0 an HTTPTrigger is a routing rule only, shaped like the Gateway API HTTPRoute (the Kubernetes standard resource for HTTP routing). It lists hostnames and path matches, and sends matching requests to one or more WorkloadDeployments that you create yourself, by name, in the same namespace.

This gives you:

• Weighted traffic splits. A rule can list several backendRefs with a weight each, for blue/green and canary rollouts.
• Independent lifecycles. A WorkloadDeployment can be scheduled before or without a trigger, and deleting a v1alpha2 trigger never deletes a workload.
• One workload behind several routes. Two triggers can point at the same WorkloadDeployment.

The same workload, before and after:

# v0.12.x: no longer accepted
apiVersion: control.cosmonic.io/v1alpha1
kind: HTTPTrigger
metadata:
  name: hello-world
  namespace: default
spec:
  replicas: 1
  ingress:
    host: hello.localhost.cosmonic.sh
    paths:
      - path: /
        pathType: Prefix
  template:
    spec:
      components:
        - name: http
          image: ghcr.io/cosmonic-labs/control-demos/hello-world:0.1.2

# v0.13.0
apiVersion: runtime.wasmcloud.dev/v1alpha1
kind: WorkloadDeployment
metadata:
  name: hello-world
  namespace: default
spec:
  replicas: 1
  template:
    spec:
      hostInterfaces:
        - namespace: wasi
          package: http
          interfaces:
            - incoming-handler
          config:
            host: hello.localhost.cosmonic.sh
      components:
        - name: http
          image: ghcr.io/cosmonic-labs/control-demos/hello-world:0.1.2
---
apiVersion: control.cosmonic.io/v1alpha2
kind: HTTPTrigger
metadata:
  name: hello-world
  namespace: default
spec:
  hostnames:
    - hello.localhost.cosmonic.sh
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: hello-world

Where each v1alpha1 field goes:

v1alpha1 HTTPTrigger	In v0.13.0
spec.replicas, spec.template, spec.deployPolicy, spec.artifacts, spec.kubernetes	The same fields on the WorkloadDeployment spec
spec.environment	WorkloadDeployment spec.template.spec.environment
The inbound wasi:http interface the operator added	Declare it yourself under spec.template.spec.hostInterfaces: incoming-handler for a WASI P2 component, handler for a WASI P3 component, with no version. Set config.host to the hostname the trigger serves
spec.ingress.host	HTTPTrigger spec.hostnames (a list of up to 16; empty matches every host, and *.example.com wildcards are allowed)
spec.ingress.paths[] with pathType: Prefix or Exact	spec.rules[].matches[].path with type: PathPrefix or Exact. A rule with no matches matches PathPrefix /
spec.timeout (Go duration such as 300s)	spec.rules[].timeouts.request, in Gateway API duration format: up to four of h, m, s and ms, for example 300s or 5m. Unset still means Envoy's 15 second default. timeouts.backendRequest bounds each try

Cosmonic Control v0.13.0 accepts either interface name for either kind of component, and it does not match the entry's version against the component. A WorkloadDeployment generated by a v0.12.x trigger always declares incoming-handler, even for a WASI P3 component, and it keeps serving after the upgrade. A misspelled interface name, or no wasi:http entry at all, leaves the workload ready but answering every request with 404.

v1alpha2 also accepts the path type RegularExpression, but Control does not route it yet: a rule that uses it is dropped and reported in the trigger's status.

Migrate your triggers

1. Before applying the v0.13.0 CRDs, save every trigger as v1alpha1. After the CRDs are applied this read no longer works.

   kubectl get httptriggers.v1alpha1.control.cosmonic.io -A -o yaml > httptriggers-v1alpha1.yaml

2. Write a WorkloadDeployment and a v1alpha2 HTTPTrigger for each one, using the table above.

3. Replace the old triggers, deleting each one before you apply the v0.13.0 CRDs. A v1alpha1 trigger owns the WorkloadDeployment it created (named <trigger-name>-<suffix>), so deleting the trigger also deletes that workload. Expect each workload to stop serving between deleting its old trigger and its new WorkloadDeployment becoming ready. To avoid that gap, follow Migrate without dropping traffic instead.

   • Installed with the http-trigger Helm chart: run helm uninstall <release>, upgrade Control, then install again with chart 0.2.0 and the same values file. Chart 0.2.0 renders the WorkloadDeployment and the v1alpha2 trigger from the same values as 0.1.x. It needs Control v0.13.0 or later.
   • Applied with kubectl or a GitOps tool: commit the new manifests, delete the old triggers (kubectl delete httptrigger <name> -n <namespace>), upgrade Control, then apply the new manifests. With Argo CD, update the Application to the new manifests (or to http-trigger chart 0.2.0) in the same change that you upgrade Control.

Anything that still applies v1alpha1 manifests after the upgrade fails, including Argo CD applications that pin http-trigger chart 0.1.x.

Triggers left in place during the upgrade

A v1alpha1 trigger that is still stored when the v0.13.0 CRDs are applied is not converted:

• The API server returns it only as v1alpha2, with an empty spec. None of its v1alpha1 fields carry over. Reading it as v1alpha1 fails with the server could not find the requested resource.
• The v0.12.x operator keeps serving the routes it had already programmed until the control chart is upgraded. Within seconds of the v0.13.0 operator starting, the trigger reports Accepted as False with reason UnsupportedValue, its routes are removed, and requests for its hostname get a 404 from Envoy.
• The WorkloadDeployment it generated, and that workload, keep running without a route.
• Deleting the trigger now no longer deletes the generated WorkloadDeployment. Its owner reference names control.cosmonic.io/v1alpha1, which the API server no longer serves, so the Kubernetes garbage collector cannot look the owner up. It leaves the WorkloadDeployment in place and keeps logging unable to get REST mapping for control.cosmonic.io/v1alpha1/HTTPTrigger in kube-controller-manager.

List the WorkloadDeployments that triggers generated, with the trigger that owns each one:

kubectl get workloaddeployments -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,TRIGGER:.metadata.ownerReferences[0].name'

Delete the ones you no longer need with kubectl delete workloaddeployment <name> -n <namespace>. To keep one, remove its owner reference as shown in step 5 below.

Migrate without dropping traffic

For triggers applied with kubectl or a GitOps tool, you can keep every route serving through the upgrade by pointing a v1alpha2 trigger at the WorkloadDeployment the old trigger generated. That WorkloadDeployment already declares the inbound wasi:http interface and config.host, so it needs no edits. This path has not been tested with http-trigger chart releases.

1. Export your triggers as in step 1 above, and note the generated WorkloadDeployment for each one (named <trigger-name>-<suffix>) with the kubectl get workloaddeployments command from Triggers left in place during the upgrade.

2. Apply the v0.13.0 CRDs. Do not delete the old triggers. The running v0.12.x operator keeps serving their routes.

3. For each trigger, apply a v1alpha2 HTTPTrigger with the same name and namespace, whose backendRefs name the generated WorkloadDeployment. This replaces the stored trigger's empty spec:

   apiVersion: control.cosmonic.io/v1alpha2
   kind: HTTPTrigger
   metadata:
     name: hello-world
     namespace: default
   spec:
     hostnames:
       - hello.localhost.cosmonic.sh
     rules:
       - matches:
           - path:
               type: PathPrefix
               value: /
         backendRefs:
           - name: hello-world-5d5f765957

4. Upgrade the control chart, then the HostGroup chart, as in Standard upgrade. The v0.13.0 operator programs the v1alpha2 routes as soon as it starts.

5. Remove the owner reference from each generated WorkloadDeployment, so the garbage collector stops retrying it and deleting the trigger later leaves the workload alone:

   kubectl patch workloaddeployment <trigger-name>-<suffix> -n <namespace> \
     --type=json -p='[{"op":"remove","path":"/metadata/ownerReferences"}]'

   Do this only after step 4. While the v0.12.x operator is running it puts the owner reference back within seconds, even after the v0.13.0 CRDs are applied.

Tested on a single-node cluster with WASI P2 and P3 components and a request every second, the trigger change itself failed no requests. Restarting pods still can: with the default single Envoy replica, a request in flight while the Envoy pod is replaced can fail (set envoy.replicas above 1, see Envoy HA and xDS tuning), and with a single-replica HostGroup every workload answered 503 for about 30 seconds while the replacement host took them over. That HostGroup gap applies to any upgrade that replaces the host pod.

Once the upgrade is done you can move each route to a WorkloadDeployment you manage yourself: deploy it under its own name, add it to the trigger's backendRefs (see Splitting traffic between workloads), and delete the generated one when the new one is serving.

Reading trigger status

kubectl get httptriggers now prints HOSTNAMES, ACCEPTED, RESOLVED, PROGRAMMED and AGE. The conditions follow HTTPRoute:

• Accepted is False when no rule can be routed. When only some rules are dropped (a conflict, or a path Envoy would reject), the trigger stays accepted and gains a PartiallyInvalid condition naming them.
• ResolvedRefs is False when a backendRefs entry names a WorkloadDeployment that does not exist. Its share of traffic gets a 500.
• Programmed is True once every Envoy replica has accepted the configuration that contains the trigger's current routes, and Pending until then.

status.backends lists each referenced WorkloadDeployment with its weight and readyWorkloads. A backend with no ready workloads answers its share of traffic with a 503 rather than shifting it to the other backends, so a broken canary shows up at its own weight.

Operator replicas now record which Envoys they serve, and what each accepted, in Leases in the operator namespace. The chart's Role already allows this. If you install with rbac.create=false and supply your own Role, add get, list, watch, create, patch and delete on coordination.k8s.io leases in the operator namespace.

Route conflicts now go to the oldest trigger

When two triggers claim the same hostname and path, v0.12.x gave the route to the newest one. v0.13.0 follows HTTPRoute precedence: the oldest trigger keeps it, and the newer trigger reports the match as Conflicted in its status. If your rollout process deploys a second trigger to take over a route, change it to edit backendRefs weights on the existing trigger instead.

A trigger with no hostnames, or a wildcard hostname, now also matches hosts that another trigger names specifically, for paths that trigger does not claim.

Autoscalers target the WorkloadDeployment

A v1alpha2 HTTPTrigger has no /scale subresource, because it no longer has replicas. Point any HorizontalPodAutoscaler or KEDA ScaledObject at the WorkloadDeployment:

scaleTargetRef:
  apiVersion: runtime.wasmcloud.dev/v1alpha1
  kind: WorkloadDeployment
  name: hello-world

See Autoscaling.

Hosts check their NATS permissions at startup

Check this before upgrading if your hosts connect to an external or managed NATS with per-user permissions. At startup a host now publishes a readiness marker and waits for the server to echo it back, to prove its command subscription works before it takes commands. The host's NATS user must be allowed to:

• subscribe to runtime.host.<host-id>.>
• publish to runtime.host.<host-id>.__control.ready and runtime.operator.heartbeat.<host-id>
• publish replies to requests

and must not connect with no_echo. Denying the subscription or the readiness marker, or disabling echo, stops the host from starting. Heartbeat and reply permissions are not checked at startup, but denying them leaves a running host undiscoverable or unable to answer commands. The in-cluster Nexus that the control chart installs sets no per-user permissions and needs no change.

HostGroup shutdown grace period raised

The HostGroup chart's terminationGracePeriodSeconds default moves from 30 to 150. On SIGTERM the host now drains in phases and then stops each plugin in turn. Each cleanup wait and each plugin stop can take up to WASH_PLUGIN_STOP_TIMEOUT_SECS plus one second (6 seconds by default), and telemetry flushing can take another 15 seconds. The chart's sizing rule is at least 9 + (plugin count + 2) × (plugin stop timeout + 1) + 15 seconds plus scheduling margin. The new default covers the built-in plugins. Raise it if you add hostPlugins or raise WASH_PLUGIN_STOP_TIMEOUT_SECS. A value you already set wins over the new default, so check that a lower override still fits the rule. Host pods take longer to terminate during rollouts as a result.

Envoy can verify host certificates

When a host group serves TLS (plugins.http.tls.enabled on the HostGroup chart), Envoy has so far encrypted traffic to its hosts without checking their certificates, so anything that could answer on a host's address could read and change that traffic. A new envoy.upstreamTLS block on the control chart turns verification on:

# cosmonic-control-values.yaml
envoy:
  upstreamTLS:
    existingSecret: ""            # Secret holding the CA that issues host certificates
    existingConfigMap: ""         # or a ConfigMap, such as a trust-manager Bundle target
    contents: ""                  # or inline PEM, with --set-file envoy.upstreamTLS.contents=ca.pem
    key: "ca.crt"
    mountPath: "/etc/envoy/upstream-ca"
    serverName: "internal.cosmonic.sh"

Setting one source (exactly one of existingSecret, existingConfigMap or contents) is what enables verification; with none, behavior is unchanged. serverName is the DNS name (SAN) every host certificate must carry, sent as SNI, and must match the HostGroup's plugins.http.tls.hostname. Hosts advertise pod IPs that their certificates never name, so without it Envoy would accept any host's certificate for any other host. The Secret or ConfigMap must be in the control release namespace, and only key is mounted, so a Secret that also holds a private key does not expose it to Envoy. This is separate from caBundle and envoy.caBundleEnabled, which govern Envoy's other outbound TLS. operator.xds.upstreamTLSTrustedCA and operator.xds.upstreamTLSServerName exist for a bundle that reaches the Envoy pod some other way, and win when set.

Verification stays off by default because each host group's certificate is self-signed by cert-manager, and the control chart cannot learn that CA on its own.

HTTPS on the bundled Traefik ingress

The bundled Traefik can now serve HTTPS on its websecure entrypoint (port 443, NodePort 30443) for the Perses host and every ingress.hosts entry, next to the existing HTTP routes. TLS ends at Traefik, and Envoy still receives plain HTTP.

# cosmonic-control-values.yaml
ingress:
  traefik:
    tls:
      enabled: true
      existingSecret: ""     # a kubernetes.io/tls Secret in cosmonic-system
      hosts: []              # empty means every ingress.hosts entry
      autogenerate:
        issuerRef:
          name: ""           # empty creates a self-signed Issuer, cosmonic-ingress-selfsigned
          kind: "Issuer"
        dnsNames: []         # extra SANs; the Perses host and ingress.hosts are always included

With existingSecret empty, cert-manager issues the certificate, so cert-manager must be installed. The Secret must live in cosmonic-system, because Traefik only reads an Ingress's TLS Secret from that Ingress's own namespace. If the same Traefik also passes TLS through for another host under the same domain, list hosts explicitly: a wildcard here terminates that host's TLS too. The Traefik image also moves from v3.6.6 to v3.7.13, and the chart's Traefik CRDs are updated with it.

Request tracing and new dashboards

Envoy can now start an OpenTelemetry trace for each request it routes, or join the one the client sent. Hosts continue the trace, so one request can be followed from the ingress through the component in Tempo. Spans are tagged with the trigger, rule, path and WorkloadDeployment they matched. It is off by default:

# cosmonic-control-values.yaml
envoy:
  tracing:
    enabled: true
    samplingPercent: 100       # share of requests Envoy traces
opentelemetryCollector:
  tracesStoredPercent: 100     # share of traces kept in Tempo

The request rate, error and latency panels on the Envoy, Workload Activity and Host Activity dashboards are derived from these spans. Envoy decides at the edge whether to trace a request, and the panels are not scaled back up, so a samplingPercent below 100 undercounts them by the same factor. To store fewer traces without skewing the panels, lower opentelemetryCollector.tracesStoredPercent instead: the collector samples after it derives its metrics, and keeps or drops whole traces. operator.xds.routeStats: true adds per-rule Envoy stats (envoy_vhost_route_*) that count every request whatever the sampling rate, at the cost of extra series per rule.

The dashboards change with it. See Built-in dashboards.

• Traces and Trace Detail are new. They list recent, failed and slow requests from Tempo, and open one as a timeline across Envoy, the host and the component. They need envoy.tracing.enabled.
• Workload Activity can filter by HTTPTrigger and compare a WorkloadDeployment against a baseline. Its internal name changed from detail to workload-activity, and Host Activity from host to host-activity, so update bookmarked URLs.
• Envoy names clusters by host instead of UUID and lists each trigger's domains and paths.
• Host Activity now selects logs on the indexed k8s_pod_name Loki label. The old query scanned every log line and could run Loki out of memory.

Observability image updates

The bundled observability images move to Prometheus v3.15.0, Loki 3.7.8, Tempo 3.1.0, OpenTelemetry Collector contrib 0.162.0, Jaeger 2.21.0 and the Perses provisioning sidecar 2.11.2, alongside Traefik v3.7.13. Envoy stays on v1.39.1. Air-gapped installs pick up the new tags by re-running the helm template mirroring step in Air-Gapped Installation with CHART_VERSION=0.13.0.

wasmCloud runtime bump

control-host now runs wash-runtime past wasmCloud v2.10.3 on Wasmtime 48.0.5. Apart from the NATS readiness check above, there is no configuration change. It applies when you bump the chart.

With your triggers migrated, the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.13.0.

Upgrading from v0.13.0 to v0.14.0

Chart 0.14.0 ships appVersion 0.14.0 and moves the control-host runtime to wasmCloud v2.11.0 on Wasmtime 48.0.7.

This release turns on host limits that earlier releases only counted. Every host group now enforces three sandbox boundaries by default, matching what wash host does:

• a raw socket (wasi:sockets) connection that the workload's allowedHosts does not cover is refused,
• a guest that grows past the host's guest memory budget gets an allocation failure,
• a workload with a hostPath volume fails to start unless the host group allows that directory.

Named wasi:keyvalue, wasi:blobstore and wasmcloud:messaging bindings also stop accepting a backend address from the workload. Each change has an opt-out on the HostGroup chart. Read the four sections below and run their checks against your v0.13.0 cluster before you run helm upgrade. The rest of the release is opt-in: admission policies for tenant namespaces, a Traefik setting for clusters that enforce Pod Security, image updates and security fixes.

Apply the updated CRDs

v0.14.0 makes no CRD schema changes. The allowedHosts and allowedIpNameLookups field descriptions now explain how a host name applies to raw sockets (see Raw-socket egress is enforced by default). Re-applying is a safe no-op, kept in the flow for consistency.

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.14.0 | kubectl apply -f -

Keep the v0.13.0 behavior while you check

If you cannot finish the checks below before the upgrade, these HostGroup values restore the v0.13.0 defaults for egress and memory. hostPath volumes have no counting mode, so list every directory your workloads mount:

# hostgroup-values.yaml
egressPolicy:
  socketEgress: count       # log what would be refused, refuse nothing
hostMemory:
  guestMemoryMode: count    # charge guest growth to the budget, refuse nothing
  maxProcessMemory: 'off'   # no whole-process limit
hostVolumes:
  allowedHostPaths:
    - /var/models           # each in-pod directory a workload hostPath volume uses

Quote 'off'. The chart reads an unquoted off as intended, but an unquoted on is a YAML boolean and fails the render. Treat these values as a temporary setting: each one removes a boundary the sandbox otherwise holds.

Raw-socket egress is enforced by default

egressPolicy.socketEgress left unset now means enforce. In v0.11.0 through v0.13.0 it meant count, which logged a connection the policy would refuse and allowed it anyway. Under enforce, every raw-socket connection a workload opens must match an entry in its allowedHosts. wasi:http requests are unaffected: they have always been gated by allowedHosts, whatever this setting says.

Enforcement also activates the range denials that were already on inside the policy. egressPolicy.denySpecialRanges defaults to on, so raw-socket connections to loopback, link-local (including the cloud metadata address 169.254.169.254), unspecified, multicast and documentation ranges are now refused. denyPrivateRanges stays off, so cluster-private addresses such as Service ClusterIPs remain reachable.

Who is affected: workloads that use raw TCP or UDP sockets, such as database drivers, Redis clients or custom protocols, and declare no allowedHosts entry for the destination. A hostPlugins entry that loads a plugin component and declares no allowedHosts has an empty allowlist, so it is refused every raw-socket connection too.

Check before upgrading. On v0.13.0 the host already evaluates the policy in count mode and logs each connection it would refuse at debug level. Set RUST_LOG=debug through the HostGroup chart's env value, let the workloads run for a representative period, and search the host logs:

# hostgroup-values.yaml (temporarily, on v0.13.0)
env:
  - name: RUST_LOG
    value: debug

kubectl logs -n cosmonic-system -l app.kubernetes.io/instance=hostgroup --tail=-1 \
  | grep 'socket egress policy would refuse this connection'

Each line carries the destination address and the reason. Match each address to the service the workload meant to reach, and add that destination to the workload's allowedHosts as host:port:

components:
  - name: api
    image: ghcr.io/example/api:0.1.0
    localResources:
      allowedHosts:
        - postgres.db.svc.cluster.local:5432

A raw socket carries no host name, so the host matches a name entry through the workload's own DNS lookups. An entry naming a host permits a connection only to an address that a wasi:sockets lookup of that name returned to the workload, on the entry's port. An entry with a scheme and no port permits that scheme's default port. As of v0.14.0, a host named in allowedHosts may also be resolved without repeating it in allowedIpNameLookups, which is now only for names beyond those. allowedHosts: ['*'] permits connections but opens no lookups.

A new HostGroup value bounds how long a lookup stays valid for connecting:

# hostgroup-values.yaml
egressPolicy:
  ipNameLookupGrantLifetime: 5m   # unset: as long as the workload remembers the lookup; 0s turns name entries off for raw sockets

This is not a DNS TTL. Set it where addresses are reused, such as pod IPs, so an address that a name pointed to earlier does not stay connectable after it moves to another service. A workload that caches an address for longer must resolve the name again before opening a new connection. Open connections are unaffected.

After the upgrade, the host publishes the policy counters as OpenTelemetry metrics: socket_policy.denied counts refused connections under enforce, and socket_policy.would_deny counts what count mode would refuse. Both carry a reason attribute.

To opt out, set egressPolicy.socketEgress: count. The policy keeps logging what it would refuse, so you can return to enforce once the log is quiet.

Guest memory is enforced by default

hostMemory.guestMemoryMode left unset now means enforce. In v0.13.0 it meant count. The guest budget is hostMemory.maxGuestMemory, which defaults to three quarters of resources.limits.memory (4.5Gi with the chart's 6Gi limit). Under enforce, a guest that tries to grow its memory past the budget sees memory.grow return -1, which the component's language runtime reports as an out-of-memory failure in that component. The per-guest ceiling, hostMemory.defaultHeapMemory (chart default 1Gi), is unchanged.

v0.14.0 also adds a limit on the whole host process, guests and host together:

Value	Default	Effect
hostMemory.guestMemoryMode	enforce (was count)	count charges guest growth to the budget and reports what it would refuse, but allows it
hostMemory.maxProcessMemory	90% of the cgroup limit (resources.limits.memory)	A Kubernetes quantity, or 'off' for no limit. Past it, the host refuses guest growth and new instances until its footprint falls back. Enforced in count mode too

The host samples its footprint periodically, so memory can overshoot maxProcessMemory between samples. Set it below the container limit, never equal to it.

Who is affected: host groups whose guests together use more than three quarters of the pod's memory limit, and pods running near their limit.

Check before upgrading. On v0.13.0, the host already charges guest growth to the budget in count mode and reports a high-water mark (guest_memory.high_water, and the host's periodic memory report in its logs). Compare the high-water mark with three quarters of resources.limits.memory. If it comes close, raise resources.limits.memory (the budget follows it) or set hostMemory.maxGuestMemory explicitly before you upgrade. kubectl top pod -n cosmonic-system shows how close each host pod runs to its limit.

To opt out, set hostMemory.guestMemoryMode: count and hostMemory.maxProcessMemory: 'off'. See HostGroup memory defaults changed for how the budget is derived.

hostPath volumes need an allowlist

A workload's hostPath volume mounts a directory from the host pod into the component. Until v0.14.0 the host accepted any directory it could read. It now accepts a hostPath volume only inside a directory listed in the new hostVolumes.allowedHostPaths value, which is empty by default. With the default, no hostPath volume is permitted and a workload that declares one fails to start. ephemeral volumes are unaffected. There is no counting mode for this check.

Find the affected workloads:

kubectl get workloaddeployments,workloads -A -o json | jq -r '.items[]
  | select(((.spec.template.spec // .spec).volumes // []) | any(has("hostPath")))
  | "\(.kind) \(.metadata.namespace)/\(.metadata.name)"'

For each one, add the directory to the HostGroup values that host group runs with. List the in-pod path (the mountPath of the HostGroup's volumeMounts), not the path on the node:

# hostgroup-values.yaml
volumes:
  - name: models
    hostPath:
      path: /opt/models       # on the node
      type: Directory
volumeMounts:
  - name: models
    mountPath: /var/models    # inside the host pod
    readOnly: true
hostVolumes:
  allowedHostPaths:
    - /var/models             # what workload hostPath volumes may lie within

The host judges each volume by its canonical path, so .. and symlinks cannot step outside an allowed directory. Even inside one, it refuses a volume that exposes a kernel filesystem (/proc, /sys, /dev), the emptyDir scratch root, or a reserved path. The host reserves the TLS, CA, cache, plugin, config and secret files its own settings name; add anything else that holds a credential to hostVolumes.reservedHostPaths. hostVolumes.scratchRoot sets where ephemeral volumes live (unset, wasmcloud-scratch under /tmp, the chart's writable emptyDir). The host counts refusals in the host_path.denied metric.

allowedHostPaths is a host-wide guardrail, not a tenant storage policy: every workload on the host group may mount any directory it lists. To keep tenants from using hostPath at all, add the COSMONIC-HOSTPATH admission policy.

Named bindings take their backend address from the HostGroup

A workload can declare the same interface more than once under distinct names and point each named host interface at its own backend. The wasmCloud v2.11.0 runtime makes the backend address a host-owned key on the plugins that serve these named wasi:keyvalue, wasi:blobstore and wasmcloud:messaging bindings. Under the default workloadConfig: deny, a workload that sets one of these keys, whether in config or through secretFrom, is refused when its binding is set up:

Plugin ids	Host-owned keys
wasi-keyvalue-multiplexed, wasmcloud-keyvalue-multiplexed	url, root, prefix
wasi-blobstore-multiplexed, wasmcloud-blobstore-multiplexed	url, root
wasmcloud-messaging-multiplexed, wasmcloud-messaging-async-multiplexed	url

The workload keeps backend, which selects the backend type. Unnamed hostInterfaces entries are not served by these plugins and are unaffected.

Find named entries that set a backend address:

kubectl get workloaddeployments,workloads -A -o json | jq -r '.items[]
  | ((.spec.template.spec // .spec).hostInterfaces // [])[] as $i
  | select($i.name != null and ($i.package | IN("keyvalue", "blobstore", "messaging"))
      and ((($i.config // {}) | has("url") or has("root") or has("prefix")) or $i.secretFrom != null))
  | "\(.kind) \(.metadata.namespace)/\(.metadata.name): \($i.namespace):\($i.package) \($i.name)"'

Move each address to a source-less hostPlugins entry on the HostGroup, under bindings keyed by the entry's name, and remove it from the workload:

# hostgroup-values.yaml
hostPlugins:
  - id: wasi-keyvalue-multiplexed
    bindings:
      cache:
        config:
          url: nats://workload-nats.shared.svc.cluster.local:4222
      sessions:
        secretFrom:
          - sessions-redis   # a Secret whose `url` key holds redis://... with credentials

A value on the entry's own config, rather than under bindings, is shared by every workload that binds any label, so one root or Redis prefix there becomes one store for all of them. Declare bindings per label to keep workloads apart.

warning

Once a hostPlugins entry declares any bindings, its named bindings become an allowlist under workloadConfig: deny. A workload that uses a binding name the entry doesn't list is refused, even if it only sets backend (for example backend: in-memory) and no address. List every binding name your workloads use with that plugin, or set workloadConfig: warn first and check the host logs before you switch back to deny.

To find affected workloads without refusing them, set workloadConfig: warn on the entry: the host logs everything deny would refuse. workloadConfig: allow restores the v0.13.0 behavior for that plugin.

Admission policies for tenant namespaces

The control chart can now install a set of ValidatingAdmissionPolicies (Kubernetes' built-in admission rules written in CEL) that cap what a tenant can grant its own workloads: no hostPath volumes, digest-pinned images from allowed registries, egress and interface allowlists, placement on the tenant's own hosts, replica and pool caps, ingress hostnames under the tenant's domain, and protection for the policy parameters themselves. It is off by default:

helm upgrade cosmonic-control oci://ghcr.io/cosmonic/cosmonic-control \
  --version 0.14.0 \
  --namespace cosmonic-system \
  -f cosmonic-control-values.yaml \
  --set policies.enabled=true

policies.enabled needs Kubernetes 1.30 or later, and the chart refuses to render it on an older cluster. Enabling it changes nothing on its own: the policies act only in namespaces labeled cosmonic.io/policy-profile: restricted, and read their parameters from a cosmonic-policy ConfigMap in that namespace. A labeled namespace without the ConfigMap refuses every workload, so create the ConfigMap before adding the label. The chart installs each binding with validationActions: ['Deny']; to start in Warn and Audit, apply the policy files yourself instead. Securing a Cluster lists every policy, with an equivalent Kyverno set, the tenant namespace they expect, and a staged rollout.

Traefik host ports and Pod Security

The bundled Traefik binds node ports 80 and 443, which the Pod Security baseline and restricted levels forbid, so it could not run in a cosmonic-system namespace that enforced either. The new ingress.traefik.hostPorts value (default true, no change on upgrade) turns the host ports off:

# cosmonic-control-values.yaml
ingress:
  traefik:
    hostPorts: false   # Traefik stays reachable through its NodePort Service (30080 and 30443)

Everything else in both charts already runs under restricted, and the v0.14.0 end-to-end suite runs with every namespace, cosmonic-system included, labeled restricted. The kind and k3d cluster configurations in Get Started map host ports 80 and 443 to those NodePorts, so they keep working with hostPorts: false. Turning host ports off also avoids the single-node Traefik rollout stall described in Standard upgrade. See Pod Security for labeling the namespace.

The LLM app serves OpenAI's API

Cosmonic's LLM app now serves OpenAI's HTTP API (chat completions, the Responses API, embeddings and the model list) from open-weight models running on a GPU host group, so OpenAI SDKs and tools can use it by changing their base URL. It ships as the component ghcr.io/cosmonic/components/openai-llama-server:0.4.0, published separately from the charts, and replaces llm-llama-server. It is not installed by either chart. See OpenAI-Compatible LLM API to deploy and call it.

Ingress and observability image updates

The bundled images move to Envoy v1.40.0, Traefik v3.7.15, OpenTelemetry Collector contrib 0.163.1, Jaeger 2.22.0 and the Perses provisioning sidecar (k8s-sidecar) 2.14.1. Air-gapped installs pick up the new tags by re-running the helm template mirroring step in Air-Gapped Installation with CHART_VERSION=0.14.0.

Security updates

• Wasmtime 48.0.5 to 48.0.7 in control-host. 48.0.6 fixes RUSTSEC-2026-0350 through -0359 and -0361 through -0364, including a wasi:filesystem sandbox escape when renaming symlinks with trailing slashes (RUSTSEC-2026-0352). 48.0.7 fixes RUSTSEC-2026-0375 through -0379, including a critical Cranelift bug on aarch64 that can jump to the wrong target on a large br_table (RUSTSEC-2026-0379).
• Go 1.27.3 and golang.org/x/net v0.60.0 for the operator and Nexus images, which fix the reachable standard library and HTTP/2 findings GO-2026-6599 through GO-2026-6617 (net/http, html/template, crypto/tls, net/textproto).

All of these apply when you bump the chart; there is no configuration change.

With the checks done, the values files updated and CRDs applied, run the Standard upgrade flow with --version 0.14.0.