OpenAI-Compatible LLM API

Cosmonic publishes an LLM app that serves OpenAI's HTTP API from your own cluster. It runs open-weight models with llama.cpp on the host group's GPU and answers the same requests as the OpenAI API: chat completions, the Responses API, embeddings and the model list. Any OpenAI SDK or tool that lets you set a base URL can use it.

The app is a single Wasm component, so it runs in the same sandbox as any other workload. It reaches only the hosts its allowedHosts names, sees only the volume you mount for its model cache, and uses the GPU through the wasi:webgpu interface the host group grants. It holds no other access to the host pod or the node.

What is published

Artifact	Contents	Version
ghcr.io/cosmonic/components/openai-llama-server	The complete app: the API server, the llama.cpp inference provider and the engine, composed into one component. Deploy this one	0.4.0
ghcr.io/cosmonic/components/openai-server	The API server alone, for composing with a different inference backend	0.4.0
ghcr.io/cosmonic/components/llama-inference	The llama.cpp inference provider alone	0.2.0

The app is published on its own schedule, separately from the Cosmonic Control charts. Version 0.4.0 is the release tested with Cosmonic Control v0.14.0. Each artifact carries a build provenance attestation, which you can check with the GitHub CLI:

gh attestation verify oci://ghcr.io/cosmonic/components/openai-llama-server:0.4.0 --owner cosmonic

Before you begin

• Cosmonic Control v0.14.0 or later. See Get Started.
• A host group with WebGPU enabled (gpu: true on the HostGroup chart) on nodes with a GPU. The engine imports wasi:webgpu, and a workload that declares an interface its host does not provide never schedules (see hostInterfaces is a scheduling filter). See GPU HostGroups to set one up.
• Outbound access from the host pods to Hugging Face, where the app downloads model weights on first use.

Deploy the app

1. Create a Secret holding the API keys clients must send. The app reads them from the API_KEYS environment variable as a comma-separated list:

   kubectl create namespace llm
   kubectl create secret generic llm-api-keys -n llm \
     --from-literal=API_KEYS="$(openssl rand -hex 32)"

2. Apply a WorkloadDeployment for the app and an HTTPTrigger that routes a hostname to it. Replace llm.example.com with your hostname, and set environment if your GPU host group serves another namespace:

   apiVersion: runtime.wasmcloud.dev/v1alpha1
   kind: WorkloadDeployment
   metadata:
     name: llm
     namespace: llm
   spec:
     replicas: 1
     template:
       spec:
         hostInterfaces:
           - namespace: wasi
             package: http
             interfaces:
               - handler
             config:
               host: llm.example.com
           - namespace: wasi
             package: webgpu
             version: 0.3.0-rc.2
             interfaces:
               - webgpu
         components:
           - name: llm
             image: ghcr.io/cosmonic/components/openai-llama-server:0.4.0
             poolSize: 1
             maxConcurrency: 8
             localResources:
               environment:
                 secretFrom:
                   - name: llm-api-keys
               allowedHosts:
                 - '*'
               volumeMounts:
                 - name: model-cache
                   mountPath: /
         volumes:
           - name: model-cache
             ephemeral: {}
   ---
   apiVersion: control.cosmonic.io/v1alpha2
   kind: HTTPTrigger
   metadata:
     name: llm
     namespace: llm
   spec:
     hostnames:
       - llm.example.com
     rules:
       - backendRefs:
           - name: llm
         timeouts:
           request: 600s

   kubectl apply -f llm.yaml

3. Send the hostname to Envoy. With the bundled Traefik, add a Kubernetes Ingress for it as shown in Ingress and Workloads.

4. Check that the app answers and lists its models:

   export LLM_API_KEY=$(kubectl get secret llm-api-keys -n llm -o jsonpath='{.data.API_KEYS}' | base64 -d)
   curl -s https://llm.example.com/v1/models -H "Authorization: Bearer $LLM_API_KEY"

The app also serves a small chat page at / for trying a model in a browser. It asks for an API key the first time the server refuses it.

Why the manifest looks like this

• poolSize: 1 and maxConcurrency: 8 keep one warm instance and let it serve overlapping requests. Model weights live in the component instance, so with the default of one call per instance, a request that arrives while a reply is streaming goes to a cold instance that loads the model all over again.
• timeouts.request: 600s on the trigger rule. It bounds the whole response, streamed or not, and the default of 15 seconds is shorter than a long reply or a first model download.
• A volume at / holds downloaded weights. The app creates a models directory in it. An ephemeral volume keeps the weights for the workload's lifetime. To keep them across restarts, mount a node directory into the host pods and use a hostPath volume instead, which the host group must allow in hostVolumes.allowedHostPaths (see hostPath volumes).
• allowedHosts: ['*'] lets the app download weights from Hugging Face over wasi:http. Narrow it to the hosts Hugging Face serves your models from if your egress policy requires it.
• secretFrom turns each key of the Secret into an environment variable, so the API_KEYS key sets the accepted keys without putting them in the manifest.

Model weights pass through the component, so larger models need more memory. If a model fails to load with an allocation error, raise hostMemory.defaultHeapMemory and resources.limits.memory on the GPU host group. See Guest memory is enforced by default.

Call the API

Point an OpenAI client's base URL at https://<host>/v1 and set its API key to one of the keys in the Secret. A model's id is its name, as /v1/models lists it.

curl -s https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen2.5 0.5B",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

With the official OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url='https://llm.example.com/v1', api_key='<key>')

reply = client.chat.completions.create(
    model='Qwen2.5 0.5B',
    messages=[{'role': 'user', 'content': 'Say hello in five words.'}],
)
print(reply.choices[0].message.content)

vectors = client.embeddings.create(model='bge-small-en-v1.5', input=['first text', 'second text'])
print(len(vectors.data[0].embedding))  # 384

The first request for a model downloads its weights, which can take minutes. To load a model ahead of time and watch the download, call POST /load. It streams server-sent events with a phase of reading_cache, downloading, writing_cache or initializing, then event: loaded:

curl -N https://llm.example.com/load \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen2.5 0.5B"}'

Endpoints

Endpoint	Notes
GET /v1/models	Chat and embedding models together
POST /v1/chat/completions	Streaming (server-sent events) and non-streaming
POST /v1/responses	OpenAI's Responses API, streaming and non-streaming
GET /v1/responses/{id}	A stored response. DELETE removes it
GET /v1/responses/{id}/input_items	What the model saw for that response, paged
POST /v1/embeddings	Vectors for text, with an embedding model
POST /load	Not part of OpenAI's API. Loads a model ahead of time and streams progress
GET /	A chat page for trying a model in a browser

Models

Model id	Kind	Notes
Qwen2.5 0.5B	Chat	Q4_K_M quantization
Llama 3.2 1B	Chat	Q4_K_M quantization
SmolLM2 1.7B	Chat	Q4_K_M quantization
bge-small-en-v1.5	Embeddings	English, 384 dimensions, up to 512 tokens per input

Using a model for something it cannot do, such as chatting with an embedding model, returns a 400 that names model.

Configuration

Set these as environment variables on the component, under localResources.environment (config for plain values, secretFrom for Secrets):

Variable	Default	Effect
API_KEYS	unset	Comma-separated keys. When set, every request except the chat page and CORS preflights needs Authorization: Bearer <key>; a missing or wrong key is a 401 in OpenAI's error shape. List a new key next to the old one to rotate without downtime. Unset or empty, the server checks nothing
CORS_ALLOWED_ORIGINS	any origin	Which browser origins may call the API: unset or * for any, a comma-separated list such as https://chat.example.com for only those, or empty to send no CORS headers
LLAMA_MODEL_IDLE_SECONDS	300	How long an idle model stays loaded. 0 releases it once its last request ends. Downloaded weights stay on disk either way

Without API_KEYS, allowing any origin lets any web page a user visits send requests to the server. Set keys, or a list of origins, whenever the server is reachable from a browser that also visits untrusted sites.

Behavior and limits

The API is a text-only subset of OpenAI's.

• Supported request fields include stream (with stream_options.include_usage), max_completion_tokens and max_tokens, temperature, top_p, frequency_penalty, presence_penalty, logit_bias, seed, stop and response_format (text, json_object or json_schema, enforced with a llama.cpp grammar). The Responses API takes the equivalents, plus instructions, previous_response_id, store, metadata and text.format. llama.cpp's top_k, min_p, repeat_penalty and repeat_last_n are accepted too. Unset sampling fields use llama.cpp's defaults: temperature 0.8, top-k 40, top-p 0.95, min-p 0.05.
• Refused with a 400 naming the field: tools, n above 1, logprobs and top_logprobs, tool calls in the message history, and image or other non-text content. On the Responses API also background, conversation, prompt, truncation: auto and a tool_choice other than auto or none. On embeddings, dimensions: vectors always have the model's own size.
• Ignored: other fields, such as user and reasoning_effort on chat completions, or reasoning and include on the Responses API. An ignored field is a known gap, not a setting that takes effect.
• Context length. A prompt too long for the model's context returns a 400 with code context_length_exceeded, before any event when streaming. A reply that runs out of tokens or context ends with finish_reason: length (on the Responses API, status incomplete). A JSON reply cut off this way is incomplete JSON, and the model is not told about the requested format, so ask for JSON in the prompt as well.
• Stored responses live in the component instance's memory: the latest 256, up to 64 MiB, oldest removed first, and all of them are lost when the instance restarts. Send store: false and the whole conversation as input if your client keeps the conversation itself.
• Prompt caching. The context a reply finishes with stays loaded, so the next turn of the same conversation processes only what it adds; usage.prompt_tokens_details.cached_tokens reports the reuse. There is one cached context per model, so overlapping or alternating conversations reprocess more of their prompts.
• Model residency. The app keeps one chat model and one embedding model loaded at a time. A request for another model of the same kind replaces an idle one; while the loaded model is busy, the request gets a 503 that clients can retry.
• Embeddings take up to 2048 non-empty strings per request, and return unit-length vectors in input order, as float or base64.

Compose a different backend

openai-server imports the cosmonic:agent inference interfaces (inference-types, model-admin, chat and embeddings), so any component that exports them can sit behind the same API, whether it runs local weights or calls a remote provider. Compose one with wac and check that nothing is left unplugged:

wash oci pull ghcr.io/cosmonic/components/openai-server:0.4.0 openai-server.wasm
wac plug openai-server.wasm --plug your-backend.wasm -o app.wasm
wash inspect app.wasm

Any imports the backend still has must be provided by the host, or by another component composed in.