Skip to main content

Securing a Cluster

A Wasm component on Cosmonic Control starts with no access to anything: no network, no filesystem, no host interfaces. Everything it can reach is granted in its manifest: the hosts it may call, the volumes it mounts, the interfaces it imports, and the hosts it runs on. Whoever can create a WorkloadDeployment in a namespace decides those grants for that namespace.

On a shared cluster, that makes the manifest the place to enforce limits. This page sets up tenant isolation in two layers:

  1. Namespace isolation. Each tenant gets a namespace with its own hostgroup, Pod Security enforcement, and object quotas.
  2. Admission policies. Policies cap what a tenant can grant its own workloads, whatever its RBAC allows it to create.

Every manifest on this page is tested against a live cluster, with both policy engines.

Namespace isolation​

Multitenancy maps each tenant to a Kubernetes namespace, and Tenant RBAC limits each team to its own. A tenant's boundary is made of these pieces together:

  • A hostgroup in the tenant's namespace. A hostgroup reports its namespace as its environment, so the tenant's workloads have hosts of their own. See Host Groups.
  • Placement pinned to that hostgroup. By default, a workload that names no environment can be placed on any matching host in the cluster, and one that names another namespace's environment runs on that namespace's hosts. Set operator.allowSharedHosts=false on the Cosmonic Control chart to have the operator keep each workload on its own namespace's hosts, and add the COSMONIC-PLACEMENT policy below to refuse workloads that ask for anything else.
  • Pod Security on the namespace, which governs the host pods.
  • A ResourceQuota on the Cosmonic objects the tenant can create.
  • The NetworkPolicies the Cosmonic Control charts install.
Shared hosts

Components from tenants that share a hostgroup run in the same host process. The Wasm sandbox still isolates them from each other, but the namespace no longer separates them at the process or node level. Give each tenant that needs that separation its own hostgroup.

Pod Security​

Wasm components run inside the host pod rather than in pods of their own, so Pod Security Admission applies to the host pods and the control plane. The Cosmonic Control and hostgroup charts run under the restricted level by default.

Label each namespace before installing anything into it. Enforcement applies to pods as they are admitted, so labeling a namespace after the install leaves its running pods unchecked.

kubectl create namespace cosmonic-system
kubectl label namespace cosmonic-system \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

The one exception is the bundled Traefik ingress, which binds host ports 80 and 443. The baseline and restricted levels both forbid host ports. To run it under either, set ingress.traefik.hostPorts=false, and Traefik stays reachable through its NodePort Service:

helm install cosmonic-control oci://ghcr.io/cosmonic/cosmonic-control \
  --namespace cosmonic-system \
  --set ingress.traefik.hostPorts=false
note

ingress.traefik.hostPorts is available in Cosmonic Control v0.14.0 and later.

Resource quotas​

A ResourceQuota can count Cosmonic objects like any other resource. Capping workloads.runtime.wasmcloud.dev caps the replicas a tenant runs across all of its deployments, since every replica is a Workload.

The tenant namespace​

This manifest creates a tenant namespace with all of the above, plus the label and ConfigMap the admission policies below read. Replace tenant-a with the tenant's namespace, then install the tenant's hostgroup into it.

# A tenant namespace for the policies in ../vap or ../kyverno.
#
# Apply this before installing the tenant's hostgroup: Pod Security only
# checks pods admitted after the label is set. When adding the policy-profile
# label to a namespace that already exists, create the cosmonic-policy
# ConfigMap first. The parameterized policies deny every write in a labeled
# namespace that has none.
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
    cosmonic.io/policy-profile: restricted
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cosmonic-policy
  namespace: tenant-a
data:
  # Comma-separated. Domains also admit their subdomains.
  allowedEgressDomains: 'api.example.com'
  allowedRegistries: 'ghcr.io/wasmcloud/, registry.example.com/tenant-a/'
  allowedInterfaces: 'wasi:http, wasi:config, wasi:logging, wasi:keyvalue'
  ingressDomain: 'tenant-a.apps.example.com'
  maxReplicas: '5'
  maxPoolSize: '10'
  maxConcurrency: '16'
---
# Workloads counts every replica the tenant runs, across all deployments.
apiVersion: v1
kind: ResourceQuota
metadata:
  name: cosmonic-objects
  namespace: tenant-a
spec:
  hard:
    count/workloaddeployments.runtime.wasmcloud.dev: '20'
    count/workloads.runtime.wasmcloud.dev: '50'
    count/httptriggers.control.cosmonic.io: '20'
    count/artifacts.runtime.wasmcloud.dev: '20'

Admission policies​

The policies come in two equivalent sets:

  • ValidatingAdmissionPolicy (VAP) is built into Kubernetes 1.30 and later and needs nothing else installed. Start here unless you already run Kyverno.
  • Kyverno ValidatingPolicy (Kyverno 1.18 and later) is for clusters that already run Kyverno and want these policies in its reports and exceptions.

Both sets use the same CEL expressions. Each rejection message starts with the policy's ID, such as [COSMONIC-HOSTPATH], whichever engine refused it.

IDRefusesConfigMap keys
COSMONIC-HOSTPATHhostPath volumes
COSMONIC-LOOPBACKAccess to the host's loopback interface
COSMONIC-EGRESSOutbound hosts and DNS lookups outside an allowlistallowedEgressDomains
COSMONIC-REGISTRYImages from untrusted registriesallowedRegistries
COSMONIC-DIGESTImages not pinned by digest
COSMONIC-INTERFACESHost interfaces the namespace was not grantedallowedInterfaces
COSMONIC-PLACEMENTWorkloads that run outside the namespace's own hosts
COSMONIC-LIMITSReplica, pool, and concurrency settings above a capmaxReplicas, maxPoolSize, maxConcurrency
COSMONIC-INGRESSHTTPTrigger hostnames outside the namespace's domainingressDomain
COSMONIC-PARAMSChanges to cosmonic-policy by anyone who cannot update the namespace

How the policies are scoped​

The policies only act in namespaces labeled cosmonic.io/policy-profile: restricted. Policies with parameters read them from a ConfigMap named cosmonic-policy in the namespace of the object being checked, so each tenant can have its own allowlists.

A labeled namespace without that ConfigMap refuses every workload, so the policies fail closed rather than open. When adding the label to a namespace that already exists, create the ConfigMap first.

Every kind that carries a workload​

A workload's spec appears in three kinds. You write a WorkloadDeployment; the operator creates a WorkloadReplicaSet from it, and a Workload for each replica. A tenant who can create any of the three can run a workload, so a policy that checks only WorkloadDeployment is bypassed by creating a Workload directly. Each policy below checks all three, and the image policies also check Artifact.

Checking the WorkloadDeployment also gives a better error. If a policy only refuses the Workload the operator creates, the refusal appears on the WorkloadReplicaSet's ScaleUp condition, and the WorkloadDeployment just never becomes ready.

Existing workloads​

Admission policies also run on updates, including the operator's routine updates and the finalizer removal that lets a deleted workload go away. A policy that checked those against a workload created before the policy would leave the workload stuck: it could not be updated, and could not finish deleting.

Each policy therefore skips objects that are being deleted, and updates that leave spec unchanged. An existing workload that breaks a policy keeps running, and is refused the next time its spec changes.

To find those workloads before enforcing, roll the policies out in stages:

Set each binding's validationActions to ["Warn", "Audit"]. Violations come back as warnings to whoever applies the object and land in the API server's audit log, without refusing anything. Once they stop, switch to ["Deny"].

Applying the policies​

The Cosmonic Control chart (v0.14.0 and later) installs every policy below with its binding when policies.enabled is set. It needs Kubernetes 1.30 or later, and refuses to render on an older cluster:

helm upgrade cosmonic-control oci://ghcr.io/cosmonic/cosmonic-control \
  --version 0.14.0 \
  --namespace cosmonic-system \
  -f cosmonic-control-values.yaml \
  --set policies.enabled=true

The chart installs each binding with validationActions: ['Deny'] and offers no value to change it. For the staged rollout in Existing workloads, apply the files yourself instead: each file below holds a ValidatingAdmissionPolicy and its binding, so edit the binding and apply it with kubectl apply -f.

COSMONIC-HOSTPATH​

A hostPath volume mounts a directory from the host's filesystem into the component, read-write unless the mount sets readOnly. Since v0.14.0, a host accepts a hostPath volume only inside a directory its HostGroup lists in hostVolumes.allowedHostPaths, which is empty by default (see hostPath volumes need an allowlist). That list is host-wide: once a directory is on it, every workload on the host group can mount it. This policy refuses hostPath volumes in tenant namespaces whatever the HostGroup allows; ephemeral volumes, which give a component scratch space of its own, stay allowed.

# COSMONIC-HOSTPATH: no host filesystem mounts.
# A hostPath volume mounts any directory the host process can see, read-write
# unless the mount sets readOnly. Ephemeral volumes stay allowed.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-hostpath
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
  validations:
    - expression: '!variables.ws.?volumes.orValue([]).exists(v, has(v.hostPath))'
      messageExpression: >-
        '[COSMONIC-HOSTPATH] ' + object.kind + '/' + object.metadata.name +
        ': hostPath volumes are not allowed; use an ephemeral volume'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-hostpath
spec:
  policyName: cosmonic-hostpath
  validationActions: ['Deny']
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

A refused workload looks like this:

Error from server (Forbidden): workloaddeployments.runtime.wasmcloud.dev "app" is forbidden:
ValidatingAdmissionPolicy 'cosmonic-hostpath' with binding 'cosmonic-hostpath' denied request:
[COSMONIC-HOSTPATH] WorkloadDeployment/app: hostPath volumes are not allowed; use an ephemeral volume

COSMONIC-LOOPBACK​

allowedHostLoopbackPorts lets a component connect to services listening on the host's own loopback interface, through host.wasmcloud.internal. Hosts only honor it when started with host loopback enabled, which is off by default, so this policy guards against a host that has it turned on.

# COSMONIC-LOOPBACK: no access to the host's loopback interface.
# allowedHostLoopbackPorts reaches whatever listens on the host's localhost
# through host.wasmcloud.internal. It only works on hosts started with
# allow_host_loopback, so this policy is defense in depth.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-loopback
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
  validations:
    - expression: >-
        variables.ws.?components.orValue([]).all(c,
          size(c.?localResources.?allowedHostLoopbackPorts.orValue([])) == 0) &&
        size(variables.ws.?service.?localResources.?allowedHostLoopbackPorts.orValue([])) == 0
      messageExpression: >-
        '[COSMONIC-LOOPBACK] ' + object.kind + '/' + object.metadata.name +
        ': allowedHostLoopbackPorts is not allowed'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-loopback
spec:
  policyName: cosmonic-loopback
  validationActions: ['Deny']
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-EGRESS​

allowedHosts lists the hosts a component may make outbound requests to, and allowedIpNameLookups the names it may resolve. A bare * in either allows everything. This policy requires every entry to name a domain in the namespace's allowedEgressDomains, or a subdomain of one. api.example.com in the allowlist admits https://api.example.com, eu.api.example.com:443, and *.api.example.com, but not api.example.com.evil.org or evilapi.example.com.

# COSMONIC-EGRESS: outbound hosts and DNS lookups stay inside an allowlist.
# Every allowedHosts and allowedIpNameLookups entry must name a domain listed
# in the namespace's cosmonic-policy ConfigMap (allowedEgressDomains), or a
# subdomain of one. A bare `*` never passes.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-egress
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
    - name: domains
      expression: >-
        params.data.?allowedEgressDomains.orValue('').split(',')
          .map(d, d.trim()).filter(d, d != '')
    # Every resource block the workload declares: each component's and the service's.
    - name: resources
      expression: >-
        variables.ws.?components.orValue([]).map(c, c.?localResources.orValue({})) +
        [variables.ws.?service.?localResources.orValue({})]
    # allowedHosts entries may carry a scheme, path, and port; only the host
    # left after removing them is compared.
    - name: allowed
      expression: >-
        variables.resources.all(r,
          r.?allowedHosts.orValue([]).all(e,
            variables.domains.exists(d,
              (e.contains('://') ? e.split('://')[1] : e).split('/')[0].split(':')[0] == d ||
              (e.contains('://') ? e.split('://')[1] : e).split('/')[0].split(':')[0].endsWith('.' + d))) &&
          r.?allowedIpNameLookups.orValue([]).all(e,
            variables.domains.exists(d, e == d || e.endsWith('.' + d))))
  validations:
    - expression: 'variables.allowed'
      messageExpression: >-
        '[COSMONIC-EGRESS] ' + object.kind + '/' + object.metadata.name +
        ': allowedHosts and allowedIpNameLookups may only name these domains or their subdomains: ' +
        variables.domains.join(', ')
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-egress
spec:
  policyName: cosmonic-egress
  validationActions: ['Deny']
  paramRef:
    name: cosmonic-policy
    parameterNotFoundAction: Deny
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-REGISTRY​

This policy requires every component image, service image, and Artifact image to start with a prefix in the namespace's allowedRegistries. End each prefix with /: ghcr.io/acme would also admit ghcr.io/acme-evil/app.

# COSMONIC-REGISTRY: component images come only from trusted registries.
# Every component, service, and Artifact image must start with a prefix listed
# in the namespace's cosmonic-policy ConfigMap (allowedRegistries). End each
# prefix with `/` so `ghcr.io/acme/` does not also admit `ghcr.io/acme-evil/`.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-registries
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments', 'artifacts']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
    - name: prefixes
      expression: >-
        params.data.?allowedRegistries.orValue('').split(',')
          .map(p, p.trim()).filter(p, p != '')
    - name: images
      expression: >-
        (object.kind == 'Artifact' ? [object.spec.image] : []) +
        variables.ws.?components.orValue([]).map(c, c.image) +
        (has(variables.ws.service) ? [variables.ws.service.image] : [])
    - name: untrusted
      expression: >-
        variables.images.filter(i, !variables.prefixes.exists(p, i.startsWith(p)))
  validations:
    - expression: 'size(variables.untrusted) == 0'
      messageExpression: >-
        '[COSMONIC-REGISTRY] ' + object.kind + '/' + object.metadata.name + ': ' +
        variables.untrusted.join(', ') + ' is not from an allowed registry (' +
        variables.prefixes.join(', ') + ')'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-registries
spec:
  policyName: cosmonic-registries
  validationActions: ['Deny']
  paramRef:
    name: cosmonic-policy
    parameterNotFoundAction: Deny
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-DIGEST​

A tag can be moved to point at different content after the workload that uses it was reviewed; a digest cannot. This policy requires every image to be pinned by digest, as image@sha256:....

# COSMONIC-DIGEST: images are pinned by digest.
# A tag can be moved to different content after review; a digest cannot.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-digests
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments', 'artifacts']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
    - name: images
      expression: >-
        (object.kind == 'Artifact' ? [object.spec.image] : []) +
        variables.ws.?components.orValue([]).map(c, c.image) +
        (has(variables.ws.service) ? [variables.ws.service.image] : [])
    - name: unpinned
      expression: "variables.images.filter(i, !i.contains('@sha256:'))"
  validations:
    - expression: 'size(variables.unpinned) == 0'
      messageExpression: >-
        '[COSMONIC-DIGEST] ' + object.kind + '/' + object.metadata.name + ': ' +
        variables.unpinned.join(', ') + ' must be pinned by digest (image@sha256:...)'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-digests
spec:
  policyName: cosmonic-digests
  validationActions: ['Deny']
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-INTERFACES​

hostInterfaces are the capabilities a workload imports from the host, such as wasi:http or wasmcloud:messaging. This policy grants capabilities per namespace: each entry's namespace:package must be listed in the namespace's allowedInterfaces. See Principle of Least Authority for why capabilities are granted this way.

# COSMONIC-INTERFACES: workloads only import capabilities the namespace was granted.
# Each hostInterfaces entry's namespace:package must be listed in the
# namespace's cosmonic-policy ConfigMap (allowedInterfaces).
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-interfaces
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
    - name: granted
      expression: >-
        params.data.?allowedInterfaces.orValue('').split(',')
          .map(i, i.trim()).filter(i, i != '')
    - name: ungranted
      expression: >-
        variables.ws.?hostInterfaces.orValue([])
          .map(i, i.namespace + ':' + i.package)
          .filter(i, !(i in variables.granted))
  validations:
    - expression: 'size(variables.ungranted) == 0'
      messageExpression: >-
        '[COSMONIC-INTERFACES] ' + object.kind + '/' + object.metadata.name + ': ' +
        variables.ungranted.join(', ') + ' is not granted to this namespace (' +
        variables.granted.join(', ') + ')'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-interfaces
spec:
  policyName: cosmonic-interfaces
  validationActions: ['Deny']
  paramRef:
    name: cosmonic-policy
    parameterNotFoundAction: Deny
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-PLACEMENT​

With the chart's default operator.allowSharedHosts=true, a workload that names no environment can be placed on any matching host in the cluster, and one that names another namespace's environment runs on that namespace's hosts. hostId pins a workload to one specific host. This policy requires environment to be the workload's own namespace and refuses hostId, so a tenant's workloads only run on the tenant's own hosts.

Setting operator.allowSharedHosts=false has the operator apply the same environment rule when it schedules. Use both: the policy refuses a misplaced workload when it is applied, with a message saying why, where the operator can only leave it unscheduled.

# COSMONIC-PLACEMENT: workloads run only on their own namespace's hosts.
# A hostgroup reports its namespace as its environment, and the scheduler only
# places a workload on hosts in the environment it names. Requiring that to be
# the workload's own namespace, and refusing a pinned hostId, keeps a tenant
# off every other tenant's hosts.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-placement
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
  validations:
    - expression: "variables.ws.?environment.orValue('') == namespaceObject.metadata.name"
      messageExpression: >-
        '[COSMONIC-PLACEMENT] ' + object.kind + '/' + object.metadata.name +
        ': environment must be set to this namespace (' + namespaceObject.metadata.name + ')'
      reason: Forbidden
    - expression: "variables.ws.?hostId.orValue('') == ''"
      messageExpression: >-
        '[COSMONIC-PLACEMENT] ' + object.kind + '/' + object.metadata.name +
        ': hostId may not pin a workload to a specific host'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-placement
spec:
  policyName: cosmonic-placement
  validationActions: ['Deny']
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-LIMITS​

Workloads from many deployments share each host, so one oversized deployment can starve the others. This policy caps replicas, poolSize (warm instances kept per component), and maxConcurrency (calls one instance serves at once) at the namespace's maxReplicas, maxPoolSize, and maxConcurrency. A key left out of the ConfigMap is not capped.

# COSMONIC-LIMITS: caps on how much of a shared host one workload can claim.
# maxReplicas, maxPoolSize, and maxConcurrency come from the namespace's
# cosmonic-policy ConfigMap; a key left out there is not capped. Unset fields on
# the workload take their defaults (1 replica, no pool, 1 concurrent call), so
# only fields that are set need checking.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-limits
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
      - apiGroups: ['runtime.wasmcloud.dev']
        apiVersions: ['v1alpha1']
        operations: ['CREATE', 'UPDATE']
        resources: ['workloads', 'workloadreplicasets', 'workloaddeployments']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: ws
      expression: 'object.spec.?template.?spec.orValue(object.spec)'
    - name: components
      expression: 'variables.ws.?components.orValue([])'
  validations:
    - expression: >-
        !has(params.data.maxReplicas) ||
        dyn(object.spec).?replicas.orValue(1) <= int(params.data.maxReplicas)
      messageExpression: >-
        '[COSMONIC-LIMITS] ' + object.kind + '/' + object.metadata.name +
        ': replicas may not exceed ' + params.data.maxReplicas
      reason: Forbidden
    - expression: >-
        !has(params.data.maxPoolSize) ||
        variables.components.all(c, c.?poolSize.orValue(0) <= int(params.data.maxPoolSize))
      messageExpression: >-
        '[COSMONIC-LIMITS] ' + object.kind + '/' + object.metadata.name +
        ': poolSize may not exceed ' + params.data.maxPoolSize
      reason: Forbidden
    - expression: >-
        !has(params.data.maxConcurrency) ||
        variables.components.all(c, c.?maxConcurrency.orValue(1) <= int(params.data.maxConcurrency))
      messageExpression: >-
        '[COSMONIC-LIMITS] ' + object.kind + '/' + object.metadata.name +
        ': maxConcurrency may not exceed ' + params.data.maxConcurrency
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-limits
spec:
  policyName: cosmonic-limits
  validationActions: ['Deny']
  paramRef:
    name: cosmonic-policy
    parameterNotFoundAction: Deny
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-INGRESS​

An HTTPTrigger claims hostnames on the shared ingress. Without a check, a tenant can claim a hostname another tenant serves and receive its traffic. This policy requires every hostname to be the namespace's ingressDomain or a subdomain of it, and refuses wildcard hostnames.

# COSMONIC-INGRESS: HTTPTriggers only claim hostnames the namespace owns.
# Every hostname must sit under the ingressDomain in the namespace's
# cosmonic-policy ConfigMap, so one tenant cannot route another tenant's
# traffic to itself. Wildcard hostnames are refused outright.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-ingress
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
      - apiGroups: ['control.cosmonic.io']
        apiVersions: ['v1alpha2']
        operations: ['CREATE', 'UPDATE']
        resources: ['httptriggers']
  # Without these, objects that predate the policy could no longer be updated
  # by the operator or have their finalizer removed. On create, oldObject is
  # null under VAP and empty under Kyverno.
  matchConditions:
    - name: not-being-deleted
      expression: '!has(object.metadata.deletionTimestamp)'
    - name: spec-changed
      expression: 'oldObject == null || !has(oldObject.spec) || object.spec != oldObject.spec'
  variables:
    - name: domain
      expression: "params.data.?ingressDomain.orValue('')"
    - name: unowned
      expression: >-
        object.spec.?hostnames.orValue([]).filter(h,
          h.startsWith('*') || variables.domain == '' ||
          !(h == variables.domain || h.endsWith('.' + variables.domain)))
  validations:
    - expression: 'size(variables.unowned) == 0'
      messageExpression: >-
        '[COSMONIC-INGRESS] ' + object.kind + '/' + object.metadata.name + ': ' +
        variables.unowned.join(', ') + ' is not under this namespace\'s domain (' +
        variables.domain + ')'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-ingress
spec:
  policyName: cosmonic-ingress
  validationActions: ['Deny']
  paramRef:
    name: cosmonic-policy
    parameterNotFoundAction: Deny
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

COSMONIC-PARAMS​

The cosmonic-policy ConfigMap sets what every other policy allows, and the namespace's standard edit and admin roles can write ConfigMaps. Without a check, a tenant could add its own registry to allowedRegistries, or any other key, and then deploy what that admits. This policy lets only a user who can update or patch the namespace create or change the ConfigMap. That user could remove the cosmonic.io/policy-profile label anyway, so the policy takes nothing from them. Deleting the ConfigMap stays allowed: without it, the other policies refuse every workload in the namespace, so a tenant who deletes it only locks itself out.

# COSMONIC-PARAMS: tenants cannot loosen their own policy parameters.
# The cosmonic-policy ConfigMap sets what the other policies allow, and a
# namespace's edit and admin roles can write ConfigMaps. Only someone who may
# change the namespace, and so could remove its policy-profile label anyway,
# may create or change it. Deleting it is left alone: the other policies then
# refuse every workload in the namespace.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: cosmonic-params
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: ['CREATE', 'UPDATE']
        resources: ['configmaps']
  matchConditions:
    # A request with no name, such as a generateName create, would otherwise
    # fail the expression and be refused.
    - name: policy-configmap
      expression: "has(request.name) && request.name == 'cosmonic-policy'"
  variables:
    - name: ns
      expression: "authorizer.group('').resource('namespaces').name(request.namespace)"
  validations:
    - expression: >-
        variables.ns.check('update').allowed() ||
        variables.ns.check('patch').allowed()
      messageExpression: >-
        '[COSMONIC-PARAMS] ConfigMap/cosmonic-policy: only those who can update namespace ' +
        request.namespace + ' may change its policy parameters'
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: cosmonic-params
spec:
  policyName: cosmonic-params
  validationActions: ['Deny']
  matchResources:
    namespaceSelector:
      matchLabels:
        cosmonic.io/policy-profile: restricted

Checking a manifest​

A server-side dry run sends a manifest through admission without creating anything, which makes it a quick way to check a workload against the policies, or to test a policy change:

kubectl apply --dry-run=server -n tenant-a -f workload.yaml

Writing your own policies​

The policies above are a starting point. When writing more, keep the conventions that make these safe to roll out:

  • Match workloads, workloadreplicasets, and workloaddeployments, and read the workload spec with object.spec.?template.?spec.orValue(object.spec), which finds it in all three.
  • Keep the two matchConditions, so existing objects can still be updated and deleted.
  • Name API versions explicitly rather than with *. Kyverno's reports controller cannot resolve a wildcard version for custom resources, and silently scans nothing.
  • Read the namespace from namespaceObject.metadata.name. On Kubernetes 1.30, CEL does not expose metadata.namespace for custom resources.
Last updated on by github-actions[bot]