---
title: "Kafka as a Capability with Cosmonic Control | Cosmonic"
description: "Cosmonic Control makes Kafka a least-privilege capability, with a host-driven handler built for serverless and elastic event processing."
canonical: "https://cosmonic.com/blog/cosmonic-kafka-native-and-in-sandbox/"
---

[Bailey Hayes](https://cosmonic.com/blog/team/bailey-hayes/)

| Oct 7, 2026| 7 min read

# Kafka as a Capability with Cosmonic Control

![Kafka as a Capability with Cosmonic Control](https://cosmonic.com/blog/assets/images/header-be7ebfeeb725e8f784c87f1a47262f5a.webp)

Cosmonic Control v0.12 ships native Kafka through `cosmonic:kafka@0.5.0`. Workloads can publish records, process events, use a stateful pull consumer, or commit Kafka transactions without carrying a broker address or credential in component code.

For serverless event processing, start with the `handler` interface. The host owns the Kafka consumer and keeps its group membership stable. It creates component instances when partitions have work, runs different partitions concurrently, and can reclaim every guest instance when the stream is quiet. Kafka stays connected while your compute scales independently, all the way to zero.

## Start with the handler

A handler exports one function. The host polls Kafka and calls that function with a partition-ordered batch. This deployment permits up to 32 concurrent component calls and lets the warm pool return to zero after 30 seconds:

```yaml
apiVersion: runtime.wasmcloud.dev/v1alpha1
kind: WorkloadDeployment
metadata:
  name: order-transformer
spec:
  replicas: 1
  template:
    spec:
      hostInterfaces:
        - namespace: cosmonic
          package: kafka
          version: '0.5.0'
          interfaces: [handler, producer]
          config:
            bootstrap.servers: kafka.kafka.svc:9092
            topics: orders.input,orders.output,orders.dlq
            handler.topics: orders.input
            handler.group.id: order-transformer
            handler.batch.size: '100'
            dead-letter.topic: orders.dlq
            acks: all
          secretFrom:
            - name: kafka-orders-credentials
      components:
        - name: processor
          image: ghcr.io/example/order-transformer:1.0.0
          poolSize: 32
          maxConcurrency: 1
          reclaimWindowSeconds: 30
          reclaimMinInstances: 0
```

The component exports `cosmonic:kafka/handler@0.5.0`. This example also imports `producer`, so a handler can transform an input record and publish its result without opening another Kafka client.

Useful concurrency for one deployment replica is bounded by:

```text
min(assigned partitions, 64, poolSize x maxConcurrency)
```

`poolSize` controls elastic guest compute, not Kafka group membership. Increasing it does not add consumers or trigger a rebalance. Increase workload replicas when you need another host or group member for availability or more assigned partitions.

That separation makes `handler` the best default for serverless and elastic workloads. A burst can use more component instances without waiting for pod scheduling, and those instances can disappear while the host preserves the consumer session.

## Delivery is explicit

Handler delivery is at least once. The host advances offsets only after the workload reports progress, so a crash between an external side effect and an offset commit can deliver a record again. Make handlers idempotent, or use a Kafka transaction when output records and consumed offsets must commit atomically.

A handler can report exactly how much of a batch succeeded:

- `ok(none)` accepts the full batch.
- `ok(some(offset))` accepts records through that offset and retries the remaining suffix.
- A `transient` error retries with exponential backoff.
- A `permanent` error isolates the head record and sends it to the configured dead-letter topic.
- A trap or timeout retries the head record and dead-letters it after five consecutive failures.

Calls contain records from one partition in offset order. The default maximum batch size is 100 records, and different partitions can run concurrently. Treat malformed input as permanent rather than trapping, and keep required state outside the component instance: the next call may run on a different instance.

## Least privilege is part of the binding

In 0.5.0, workloads do not pass connection or client settings to `open`. Producer functions are binding-scoped, and `consumer.open()` takes no arguments. Broker addresses, credentials, group IDs, transaction IDs, limits, and client policy come from `hostInterfaces`, outside the guest.

The `topics` setting is an authorization boundary, not only a convenience. It must grant every subscription, dead-letter topic, and topic named by a producer call. `handler.topics` selects the subscription from that grant. A component cannot redirect itself to another broker, substitute a credential, or publish to an ungranted topic.

Keep plain policy in `config` and credentials in a Kubernetes Secret referenced by `secretFrom`. The host resolves the Secret and supplies it to the native client; the component never receives the password or TLS material.

You can also use named bindings to give one component deliberately separate Kafka authorities. For example, two producer imports can route to different clusters or principals, each with its own topic grant.

## Choose the interface that matches the workload

The 0.5.0 package has four distinct application patterns:

- **`handler` is the default for serverless event processing.** The host owns polling, group membership, retries, offsets, and dead-letter delivery while guest compute scales independently.
- **`producer` is for publishing.** `send`, `send-batch`, and `send-stream` call a producer attached to the binding; there is no guest-owned producer resource, `open`, `close`, or global `flush`.
- **`transaction` is for atomic Kafka work.** Its resource can publish records and commit consumed offsets together. Each live replica needs a distinct, stable `transactional.id`.
- **`consumer` is for long-running services that need direct session control.** Choose it for manual assignment, rebalance events, pause, seek, explicit commits, or guest-controlled pull pace. Those operations make the workload stateful, so this is not the default serverless shape.

## The native backend

Cosmonic Control's production backend runs on librdkafka through the Rust `rdkafka` crate. The host reuses a native producer for each binding, so concurrent `send` calls can benefit from librdkafka's own batching. Blocking metadata and transaction operations stay off the async execution path.

The backend supports TLS, SASL, transactions, pull consumers, and host-driven handler delivery. Its broker-backed test suite covers authentication success and rejection, unavailable brokers, disconnects, queue backpressure, transactions, group rebalancing, retry behavior, and dead-letter delivery.

Native Kafka allocations live outside Wasm memory, so the plugin bounds clients, streams, queues, and handler buffers per component. It also emits bounded-cardinality OpenTelemetry metrics for active clients, guest operations, operation duration, and pull-consumer records. Dedicated handler throughput and outcome metrics are not part of 0.5.0 yet.

## The second backend: a Kafka client inside the sandbox

Because Kafka is exposed as a component interface, we can explore a very different implementation. The experimental backend is a **host component plugin**: a Wasm component that speaks the Kafka wire protocol over `wasi:sockets`. There is no native Kafka client in the host process. The long-lived poller is itself sandboxed, versioned, and shipped like any other component.

That placement matters for serverless Kafka. Broker connections, metadata, heartbeats, and group membership must outlive an ephemeral handler invocation. Most platforms put that state in trusted native infrastructure. This experiment preserves the same split between stable polling and elastic work while moving the poller into a sandboxed component.

The plugin's push path dispatches partition-ordered batches to a workload exporting `cosmonic:kafka/handler`, with retries, partial progress, offset commits, and dead-letter delivery. Its Kafka protocol implementation is written directly in Rust and tested end to end from Wasm against Redpanda and Apache Kafka.

This backend remains experimental. It has no TLS, SASL, or transactions, and its current `cosmonic:kafka@0.2.0` contract is not drop-in compatible with the native 0.5.0 interface. The [source and runnable example](https://github.com/cosmonic-labs/awesome-cosmonic/tree/feat/kafka-s3-host-plugins/host-plugins/component/kafka) show what becomes possible when even stateful infrastructure can run as a sandboxed component.

## Try it out

Native Kafka is available in [Cosmonic Control v0.12 and later](https://cosmonic.com/docs/install-cosmonic-control/). The Kafka interface is published at `ghcr.io/cosmonic-labs/cosmonic/kafka:0.5.0`. The [`awesome-cosmonic` Kafka templates](https://github.com/cosmonic-labs/awesome-cosmonic/tree/main/components/kafka) include complete Rust projects and manifests for all four patterns. Start with [`kafka-handler-consumer`](https://github.com/cosmonic-labs/awesome-cosmonic/tree/main/components/kafka/rust/kafka-handler-consumer) for serverless or elastic event processing, then reach for the pull service or transaction template only when the workload needs the state they own.

The larger point is not merely that WebAssembly can call Kafka. It is that Kafka access can be a least-privilege capability: platform policy owns the connection and authority, the application sees a typed interface, and the handler keeps stable broker state separate from compute that can scale up in milliseconds and return to zero.

> About the author
>
> ### [Bailey Hayes](https://cosmonic.com/blog/team/bailey-hayes/)
>
> Chief Technology Officer, Co-Founder
>
> Bailey Hayes is the CTO at Cosmonic. She believes the future is in distributed systems and WebAssembly (Wasm). She wears many hats in the open source ecosystem, from standards to implementations as the W3C WebAssembly WASI Subgroup co-chair, Bytecode Alliance TSC Director, and maintainer of CNCF wasmCloud.
>
> [More from Bailey](https://cosmonic.com/blog/team/bailey-hayes/)
