Observability

Cosmonic Control ships a fully integrated observability stack. No external dependencies are required.

Architecture

All components export telemetry via OTLP to the opentelemetry-collector, which fans out to:

• Prometheus — metrics
• Loki — logs
• Tempo — distributed traces
• Perses — unified dashboard over all three backends

Workload telemetry

HostGroups emit OpenTelemetry data for Wasm workloads when the --wasi-otel flag is set on the control-host container. The flag is enabled by default via the opentelemetry.workload: true value on the cosmonic-control-hostgroup chart.

With the flag enabled, workloads on the HostGroup can import the wasi:otel WIT package to emit traces, metrics, and logs:

import wasi:otel/types@0.2.0-rc.1;
import wasi:otel/tracing@0.2.0-rc.1;
import wasi:otel/logs@0.2.0-rc.1;
import wasi:otel/metrics@0.2.0-rc.1;

Emitted data flows to the endpoint in opentelemetry.endpoint (default http://opentelemetry-collector:4317), which is the same OTel collector the control plane uses. Traces, metrics, and logs land in Tempo, Prometheus, and Loki and appear in the Perses dashboards alongside control-plane telemetry.

To opt out per HostGroup, set opentelemetry.workload: false:

# hostgroup-values.yaml
opentelemetry:
  workload: false

To send workload telemetry to a different collector, override opentelemetry.endpoint:

# hostgroup-values.yaml
opentelemetry:
  endpoint: https://my-collector.corp.com:4317
  insecure: false

See the wasi:otel package spec and the otel-config example for reference usage.

Getting a signal off the host

A component can report what it is doing three ways:

Stdout or stderr (does not reach the stack): Writing to stdout or stderr sends the bytes to the host process's own output, where kubectl logs can see them. The collector this chart ships receives OTLP only, with no file-tailing receiver and no log-shipping DaemonSet, so nothing written that way is stored, indexed, or searchable after the pod restarts. Stdout is fine while you are watching a pod, but it is not a record.

wasi:otel/logs and wasi:logging (reach Loki): Both interfaces emit structured log records over OTLP through the collector. A log-record carries attributes as well as a message, and Loki is configured to index structured metadata, so those attributes may be queried.

wasi:otel/tracing (reaches Tempo): A span is a timed record of one unit of work, with a start, an end, an outcome, and attributes. Where a log line tells you something happened, a span tells you how long it took and what it was part of.

Choosing between a span and a log record

Use a span when your target has a duration or an outcome: a handler invocation, an outbound call, or a batch, for example. Use a log record for cases such as startup and configuration, and errors that have no span around them.

There are a couple of properties to consider when choosing your approach:

Trace context is extracted for you, but not injected for you. The host reads W3C traceparent off an inbound HTTP request and makes it available to the component, so a request arriving from outside is already parented correctly. Outbound is your job: wasi:otel is deliberately a thin interface rather than an SDK, so propagating context on a call you make means minting span IDs, writing the traceparent header, and setting the parent yourself. The otel-config example demonstrates the plumbing. Budget for it before planning a trace-first rollout, and expect to use a guest-side OpenTelemetry SDK rather than the raw interface.

Traces and logs are not retained for the same length of time. Tempo is configured to compact trace blocks after 24 hours. Loki is installed with no retention limit. A trace answers what happened recently; a log record provides a longer-lived reference. We recommend keeping errors and state transitions in log records even once you are emitting spans well.

Span events are assembled, not appended

span-data carries a list of events, but there is no call that appends one to a running span. The guest accumulates events and ships them with the span when it ends. This is a property of the interface, not of a particular SDK.

Accessing the Perses dashboard

Perses is deployed as a ClusterIP service and is not exposed externally by default. Use kubectl port-forward to access it locally:

kubectl port-forward svc/perses 8080:8080 -n cosmonic-system

Open http://localhost:8080 in your browser.

To expose Perses externally (for example, behind an ingress controller), change the service type in your values file:

perses:
  service:
    type: LoadBalancer   # or NodePort, or configure your own ingress

Built-in dashboards

Cosmonic Control provisions the following Perses dashboards automatically, in the cosmonic-control project. The Perses home page shows the main ones.

Workload Activity

One workload at a time. Namespace, Trigger and Workload pick it: Trigger narrows the list to the WorkloadDeployments an HTTPTrigger routes to. Compare with picks a baseline workload for side-by-side request rate, error rate, p95 latency and time in the component, which is useful during a canary. Host narrows the host-side panels to one host pod. Rows cover hosts, HTTP, Blobstore, Keyvalue, Messaging, steps the component reports through wasi:otel, and logs. The comparison panels are measured at the ingress, so they need the workload behind an HTTPTrigger and ingress tracing turned on.

Host Activity

One host at a time: the workloads running on it (count, request rate, error rate, sorted table), host and workload lifecycle spans (connect_nats, component preparation, plugin bind and unbind), and a logs panel selected on the k8s_pod_name label.

Envoy

The ingress proxy: live servers, memory, connections and traffic, xDS control-plane health (connected proxies, configuration updates accepted and rejected, dropped route claims by reason, snapshot build time), and a Routes row with requests and error rate per HTTPTrigger, domain and path. The Routes row needs ingress tracing.

Traces and Trace Detail

Recent, failed and slow requests that came in through the ingress, filtered by trigger and WorkloadDeployment. Select one to open it in Trace Detail as a single timeline across Envoy, the host, and any spans the component reports through wasi:otel. Both need ingress tracing.

Workloads

The control plane, in four rows:

• Host Infrastructure: host reconciliation activity, host controller errors, and workqueue depth by controller
• Workloads: workload reconciliation rate, workload errors by type, and active workers by controller
• Operator Resource Usage: operator memory, CPU and goroutines
• Control Plane Components: ProjectEnvironment reconciliation activity, and HTTPTriggers left out of routing

As of v0.13.0 the Workload Activity dashboard's internal name is workload-activity (previously detail) and Host Activity's is host-activity (previously host). Display names are unchanged, but saved links of the form /projects/cosmonic-control/dashboards/detail need the new name.

Tracing requests from the ingress

Envoy traces requests only when the control chart enables it (v0.13.0 and later):

# cosmonic-control-values.yaml
envoy:
  tracing:
    enabled: true
    samplingPercent: 100       # share of requests Envoy traces
opentelemetryCollector:
  tracesStoredPercent: 100     # share of traces kept in Tempo

Hosts continue the trace Envoy starts, so one request can be followed from the ingress through the component. The ingress panels on the Envoy, Workload Activity and Host Activity dashboards are derived from Envoy's spans and are not scaled up for sampling, so a samplingPercent below 100 undercounts them. Keep samplingPercent at 100 and lower opentelemetryCollector.tracesStoredPercent to store fewer traces.

Host identity on telemetry

Every span, log, and metric emitted by a HostGroup pod carries the following OpenTelemetry resource attributes, set on the host via the Kubernetes downward API:

• k8s.pod.name
• k8s.pod.uid
• k8s.node.name
• k8s.namespace.name
• cosmonic.io/hostgroup

Use these to scope queries to a specific pod, node, or HostGroup without joining against external cluster state. The Host Activity dashboard uses k8s_pod_name (the Loki structured-metadata field copied from k8s.pod.name) as its host selector.

Accessing backends directly

Each backend is available as a ClusterIP service in the cosmonic-system namespace for direct access or integration with external tooling (e.g. an existing Grafana instance):

Service	Port	Protocol
prometheus	9090	HTTP
loki	3100	HTTP (Loki API)
tempo	3200	HTTP / 4317 gRPC (OTLP)
opentelemetry-collector	4317 (gRPC) / 4318 (HTTP)	OTLP

To disable the built-in Perses dashboard (for example, when integrating with an existing Grafana deployment):

perses:
  uiEnabled: false

Custom dashboards

Perses supports a Dashboard-as-Code approach via provisioning. Add custom dashboards with the perses.provisioning.extraProvisioningFiles Helm value. See the Perses documentation for the dashboard file format.

Further reading

• Perses documentation
• Prometheus documentation
• OpenTelemetry Collector documentation