Netreap: A Practical Guide to Running Cilium in Nomad

In our last post, we looked at some of the challenges inherent in running a highly distributed, microservices-centric infrastructure and how to overcome issues of networking and security in this novel environment.

In particular, we looked at some of the limitations Kubernetes has, especially at the edge, and why this was a key reason for selecting HashiCorp Nomad as our container orchestrator for WebAssembly and wasmCloud.

We love working with Cilium. In our view, it's the best way to manage network policies and secure workloads running on Kubernetes clusters. We wanted to find a way to capitalize on Cilium within our Nomad clusters: and so Netreap was born. Check out the last post for the background.

In this post, we'll guide you through how to run Cilium on a Nomad node, and give you a sense of how Netreap performs in practice.

This should work with any Nomad task driver that supports CNI, but in this example we'll use Docker. You can follow along with my demo here - the practical section starts at 8.35.

A few tips to get started

To get started you'll need the following:

• Our newly open sourced project: Netreap! 🎉
• A Consul cluster. This can be a single node running in dev mode.
• A running Nomad cluster configured to use Consul service discovery. This can also be a single node running in dev mode.
• Cilium 1.12 or 1.13.
• You will also need to install the CNI plugins alongside Cilium.

A note on running Cilium

Due to the way Nomad fingerprinting currently works, you cannot run Cilium as a system job to provide the CNI plugin. This means you'll need to configure and run it yourself on every agent that you want to include in the Cilium mesh.

Make sure the default Docker bridge network has a CIDR that does not overlap with the Cilium network you want to use. See the documentation for the default-address-pools option in Docker's runtime configuration.

Before continuing, you'll need to ensure that CNI plugins are installed. Some distros have CNI plugins in their package repositories but it's often easier to install the latest version from the GitHub repo.

Lastly, you'll also need to be able to mount volumes from the host system so Netreap can use the Cilium API over a Unix socket. Make sure the following configuration is set in /etc/nomad/nomad.hcl:

plugin "docker" {
  config {
    volumes {
      enabled = true
    }
  }
}

iptables

Make sure that iptables is properly configured on the host:

cat <<'EOF' | sudo tee /etc/modules-load.d/iptables.conf
iptable_nat
iptable_mangle
iptable_raw
iptable_filter
ip6table_mangle
ip6table_raw
ip6table_filter
EOF

Cilium Agent

Ensure the Cilium Docker container is running on every node. It needs to be configured to use Consul as the backing store, since that is where Cilium stores its state. Since you can't run Cilium as a Nomad job right now, the easiest way to run it is to just use systemd. You can run and enable a service similar to the following:

[Unit]
Description=Cilium Agent
After=docker.service
Requires=docker.service
After=consul.service
Wants=consul.service
Before=nomad.service

[Service]
Restart=always
ExecStartPre=-/usr/bin/docker exec %n stop
ExecStartPre=-/usr/bin/docker rm %n
ExecStart=/usr/bin/docker run --rm --name %n \
  -v /var/run/cilium:/var/run/cilium \
  -v /sys/fs/bpf:/sys/fs/bpf \
  --net=host \
  --cap-add NET_ADMIN \
  --cap-add NET_RAW \
  --cap-add IPC_LOCK \
  --cap-add SYS_MODULE \
  --cap-add SYS_ADMIN \
  --cap-add SYS_RESOURCE \
  --privileged \
  cilium/cilium:v1.13.2 \
  cilium-agent --kvstore consul --kvstore-opt consul.address=127.0.0.1:8500 \
    --enable-ipv6=false -t geneve \
    --enable-l7-proxy=false  \
    --ipv4-range 172.16.0.0/16

[Install]
WantedBy=multi-user.target

Note that this actually runs Cilium with Docker! The reason for this is that Cilium uses forked versions of some key libraries and needs access to a C compiler. We found that it is much easier to just the container instead of installing all of Cilium's dependencies.

Configuring the CNI

The big thing to keep in mind is that you need to make sure that the IP CIDR you use for Cilium does not conflict with Docker's default IP range. If it does or if you want to change Docker's IP range, take a look at the default-address-pools option in daemon.json, ex.

{
  "default-address-pools": [
    {
      "base": "192.168.0.0/24",
      "size": 24
    }
  ]
}

You will then need to make sure you have a CNI configuration for Cilium in /opt/cni/conf.d named cilium.conflist:

{
  "name": "cilium",
  "cniVersion": "1.0.0",
  "plugins": [
    {
      "type": "cilium-cni",
      "enable-debug": false
    }
  ]
}

Ensure that the Cilium CNI binary is available in /opt/cni/bin:

sudo docker run --rm --entrypoint bash -v /tmp:/out cilium/cilium:v1.13.2 -c \
  'cp /usr/bin/cilium* /out; cp /opt/cni/bin/cilium-cni /out'
sudo mv /tmp/cilium-cni /opt/cni/bin/cilium-cni

Optionally install the other Cilium binaries to /usr/local/bin:

sudo mv /tmp/cilium* /usr/local/bin

Create a configuration file for Cilium (similar to what the Nomad docs suggest), ex:

{
  "name": "cilium",
  "cniVersion": "1.0.0",
  "plugins": [
    {
      "type": "cilium-cni"
    }
  ]
}

Once you've followed these steps, once you start the Nomad agent you should be able to see that the CNI is fingerprinted on the node:

$ nomad node status -verbose $NODE_ID | grep cilium
plugins.cni.version.cilium-cni           = 1.0.0

Running Netreap

You can run Netreap as a system job in your cluster similar to the following:

job "netreap" {
  datacenters = ["dc1"]
  priority    = 100
  type        = "system"

  constraint {
    attribute = "${attr.plugins.cni.version.cilium-cni}"
    operator = "is_set"
  }

  group "netreap" {
    restart {
      interval = "10m"
      attempts = 5
      delay = "15s"
      mode = "delay"
    }
    service {
      name = "netreap"
      tags = ["netreap"]
    }

    task "netreap" {
      driver = "docker"

      env {
        NETREAP_CILIUM_CIDR = "172.16.0.0/16"
      }

      config {
        image        = "ghcr.io/cosmonic/netreap:0.1.0"
        network_mode = "host"

        # You must be able to mount volumes from the host system so that
        # Netreap can use the Cilium API over a Unix socket.
        # See
        # https://developer.hashicorp.com/nomad/docs/drivers/docker#volumes-1
        # for more information.
        volumes = [
          "/var/run/cilium:/var/run/cilium"
        ]
      }
    }
  }
}

The job constraint ensures that Netreap will only run on nodes where the Cilium CNI is available.

Please note that to configure the Nomad and Consul clients that Netreap uses, we leverage the well defined environment variables for Nomad and Consul.

Netreap connects to Cilium's API using a Unix domain socket. This path must be made available to the job otherwise Netreap will not be able to communicate with Cilium

Cilium Policies

One of Netreap's key responsibilities is to sync Cilium policies to every node in your Cilium mesh. Normally Cilium policies are configured using Kubernetes CRDs, but is not an option when we're running Nomad. Normally Cilium combines all of the CRD values into a single JSON representation which is in turn imported by every agent. Instead, Netreap watches a single Consul key that stores the complete JSON representation of all of the Cilium policies in your cluster and sync that to the local Cilium agent.

The Cilium documentation has examples on how to write policies in JSON.

Whenever you want to update policies in your cluster, simply set the key in Consul:

consul kv put netreap.io/policy @policy.json

Netreap automatically picks up any updates to the value and updates the policy on every node where it is running.

Tutorial

Assuming you have Cilium and Netreap running in Nomad, you can test out Netreap by deploying a sample job and taking a look at the generated endpoints.

To try it out, we can use the stock Nomad job example with a few modifications:

job "example" {
  // This will end up as a label on the Cilium endpoint.
  meta = {
    "cosmonic.io/app_name" = "example"
  }

  group "cache" {
    network {
      port "db" {
        to = 6379
      }
      // This selects the CNI plugin to use.
      // The name after "cni/" should match the conflist that is configured on
      // the Nomad node.
      // See https://developer.hashicorp.com/nomad/docs/job-specification/network#cni.
      mode = "cni/cilium"
    }

    service {
      name         = "example"
      port         = "db"
      tags         = ["example"]
      address_mode = "alloc"
    }

    task "redis" {
      driver = "docker"

      config {
        image          = "redis:7"
        ports          = ["db"]
        auth_soft_fail = true
      }

      identity {
        env  = true
        file = true
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

This adds a few things we need to run a Nomad job with Cilium and Netreap:

• The job is configured to use the Cilium CNI (the mode=cni/cilium line in the network block). This instructs Nomad to use Cilium to provision an IP address for the group instead of using Docker's built in bridge network.
• It configures a service for the group with the address_mode set to use the IP address allocated to the container by Cilium. This service is what Netreap is able to use to look up details about the network later.
• It also adds a meta block to the job definition. Netreap is able to apply all of the key-value pairs in this block as labels on the Cilium endpoint so that you can tie information in Cilium back to running Nomad jobs.

Go ahead and save this file as example.nomad.hcl and run it your Nomad cluster:

nomad job run -check-index 0 example.nomad.hcl

One the job starts, you should see that Netreap applied some metadata as labels including the Nomad job ID and the key-value pair we had set in the meta block of the job:

$ cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])           IPv6   IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
575        Disabled           Disabled          1          reserved:host                                                ready
741        Enabled            Enabled           4          reserved:health                              172.16.32.95    ready
1203       Enabled            Enabled           20803      netreap:nomad.job_id=example                 172.16.31.165   ready
                                                           netreap:nomad.namespace=default
                                                           nomad:cosmonic.com/app_name=example

By default Cilium is configured to block all ingress and egress traffic from an endpoint. If you are using our systemd configuration then that is also the case, and you will need to apply a policy that allows your Nomad jobs to receive network traffic. Let's try applying a policy that allows for all ingress and egress traffic and see what happens.

First, let's apply a policy that allows for all egress and ingress traffic to your Nomad jobs.

Save the following file as policy.json where you happen to be running Nomad and Consul:

[
  {
    "labels": [
      {
        "key": "io.cosmonic.cilium_health"
      }
    ],
    "endpointSelector": {
      "matchLabels": {
        "reserved:health": ""
      }
    },
    "ingress": [
      {
        "fromEntities": ["remote-node", "host"]
      }
    ],
    "egress": [
      {
        "fromEntities": ["remote-node", "host"]
      }
    ]
  },
  {
    "endpointSelector": {},
    "labels": [
      {
        "key": "io.cosmonic.default_rule"
      }
    ],
    "ingress": [
      {
        "fromCIDRSet": [
          {
            "cidr": "0.0.0.0/0"
          }
        ]
      },
      {
        "fromEntities": ["host", "remote-node"]
      }
    ],
    "egress": [
      {
        "toEntity": ["host"],
        "toPorts": [
          {
            "ports": [
              {
                "port": "53"
              }
            ]
          }
        ]
      },
      {
        "toCIDRSet": [
          {
            "cidr": "0.0.0.0/0"
          }
        ]
      }
    ]
  }
]

This policy does a few things:

• It always allows for Cilium health checks to run
• It allows all ingress to your endpoints
• It allows all egress traffic from your endpoints
• It separately grants access to egress on port 53 for DNS

Next, set the policy as a key in Consul:

$ consul kv put netreap.io/policy @policy.json
Success! Data written to: netreap.io/policy

Now let's exec into the running example job and try pinging a host:

$ nomad exec -i -t $(curl localhost:4646/v1/job/example/allocations 2>/dev/null | jq -r '.[0].ID') bash
root@6fc56f808252:/data# apt update && apt install -y iputils-ping
...

root@6fc56f808252:/data# ping cosmonic.com
PING cosmonic.com (34.148.97.127) 56(84) bytes of data.
64 bytes from 127.97.148.34.bc.googleusercontent.com (34.148.97.127): icmp_seq=1 ttl=44 time=31.9 ms
64 bytes from 127.97.148.34.bc.googleusercontent.com (34.148.97.127): icmp_seq=2 ttl=44 time=31.0 ms
64 bytes from 127.97.148.34.bc.googleusercontent.com (34.148.97.127): icmp_seq=3 ttl=44 time=31.0 ms

Great! The policy allowed us unfettered access to the Internet. But what happens if you don't want to allow that? Let's apply a policy that blocks all egress traffic for your cluster except for DNS.

[
  {
    "labels": [
      {
        "key": "io.cosmonic.cilium_health"
      }
    ],
    "endpointSelector": {
      "matchLabels": {
        "reserved:health": ""
      }
    },
    "ingress": [
      {
        "fromEntities": ["remote-node", "host"]
      }
    ],
    "egress": [
      {
        "fromEntities": ["remote-node", "host"]
      }
    ]
  },
  {
    "endpointSelector": {},
    "labels": [
      {
        "key": "io.cosmonic.default_rule"
      }
    ],
    "ingress": [
      {
        "fromCIDRSet": [
          {
            "cidr": "0.0.0.0/0"
          }
        ]
      },
      {
        "fromEntities": ["host", "remote-node"]
      }
    ],
    "egress": [
      {
        "toEntity": ["host"],
        "toPorts": [
          {
            "ports": [
              {
                "port": "53"
              }
            ]
          }
        ]
      }
    ]
  }
]

Save the above policy as blocked-egress.json and update the key in Consul:

$ consul kv put netreap.io/policy @blocked-egress.json
Success! Data written to: netreap.io/policy

Now let's try pinging a host again:

$ nomad exec -i -t $(curl localhost:4646/v1/job/example/allocations 2>/dev/null | jq -r '.[0].ID') bash
root@6fc56f808252:/data# ping cosmonic.com
PING cosmonic.com (34.74.170.74) 56(84) bytes of data.

Success! Netreap observed that there was a change to the policy and applied it to Cilium, which blocked all egress traffic in your cluster. Of course that isn't very useful, but that should give you an idea of what you can accomplish with Cilium policies for your Nomad workloads.

Now let's stop the job and see if the endpoint disappears:

nomad stop -purge example

If you run cilium endpoint list, you should see that the endpoint for the job has been cleaned up.

$ cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6   IPv4           STATUS
           ENFORCEMENT        ENFORCEMENT
575        Disabled           Disabled          1          reserved:host                                       ready
741        Enabled            Enabled           4          reserved:health                      172.16.32.95   ready

Wrapping Up

In summary, in Netreap, Cilium retains the ability to transparently insert security visibility and enforcement, but does so in a way that is based on service / pod / container identity (in contrast to IP address identification in traditional systems) and can filter on application-layer (e.g. HTTP).

As a result, Cilium not only makes it simple to apply security policies in a highly dynamic environment by decoupling security from addressing, but can also provide stronger security isolation by operating at the HTTP-layer in addition to providing traditional Layer 3 and Layer 4 segmentation.

Again, take a look at my HashiTalks presentation for the more detailed demo on Netreap.

What's Next?

After open sourcing Netreap, we're excited to see what other engineers can do with the tool. When you get Netreap running, head over to our Discord and let us know how it's going! Our next avenue that we're exploring is the ability to build a bridge between capabilities provided by CNI plugins and WASI sockets so that we can bring all of the power of Cilium to WebAssembly.

If you're interested in diving in a little deeper, check out last year's HashiConf talk that discusses our vision, use of Nomad, working with Firecracker and creating Netreap. Distributed Flexibility: Nomad and Vault in a Post-Kubernetes World

Want to send your application into orbit?

Get in touch for a demo

Book Now