This is the first post in my Zero to DBaaS series, where I’m building a global, multi-tenant Database-as-a-Service platform from scratch. By the end of this post, we’ll have an 8-node Kubernetes cluster spanning three independent providers - two Fasthosts VPS servers running a high-availability control plane, three AWS EC2 instances, and three GCP Compute Engine instances - all connected by Cilium eBPF with WireGuard encrypting every packet that crosses the public internet.

No kube-proxy. No cloud VPNs. Just the Linux kernel doing the heavy lifting.

Architecture

                      ┌────────────────────────────────────────────────────────┐
                      │              FASTHOSTS (Management Hub)                │
                      │  ┌──────────────────────┐    ┌──────────────────────┐  │
                      │  │  k8s-cp-01 (Primary) │    │  k8s-cp-02 (HA Sync) │  │
                      │  └──────────┬───────────┘    └──────────┬───────────┘  │
                      └─────────────┼───────────────────────────┼──────────────┘
                                    │                           │
====================================│===========================│======================
                                    │ Cilium WireGuard Tunnel   │  (Public Internet)
                                    │      (UDP Port 51871)     │
====================================│===========================│======================
                                    │                           │
             ┌──────────────────────┴─────────────┬─────────────┘
             │                                    │ 
             ▼                                    ▼ 
┌─────────────────────────┐          ┌─────────────────────────┐ 
│       AWS POOL          │          │        GCP POOL         │ 
│  (Tenant A Workers)     │          │   (Tenant B Workers)    │ 
│  • k8s-worker-aws-01    │          │   • k8s-worker-gcp-01   │ 
│  • k8s-worker-aws-02    │          │   • k8s-worker-gcp-02   │ 
│  • k8s-worker-aws-03    │          │   • k8s-worker-gcp-03   │ 
└─────────────────────────┘          └─────────────────────────┘

The control plane lives on cost-effective VPS infrastructure. Each tenant’s worker pool is isolated within its own cloud provider’s VPC - database replication and client traffic never leave that provider’s network, so there are zero cross-cloud data-plane egress costs. The only traffic crossing the public internet is control-plane communication: API server <-> kubelet heartbeats, Cilium policy sync, Hubble telemetry. All of it encrypted.

Node inventory:

Role Provider Count Type Labels
Control Plane Fasthosts VPS 2 2 vCPU / 4 GB -
Tenant A Workers AWS EC2 3 t3.small (2 vCPU / 2 GB) cloud=aws,tenant=tenant-a
Tenant B Workers GCP Compute Engine 3 e2-small (2 vCPU / 2 GB) cloud=gcp,tenant=tenant-b

Port matrix:

Port Protocol Scope Purpose
6443 TCP Control plane nodes Kubernetes API
2379–2380 TCP Fasthosts CP nodes etcd HA sync
51871 UDP All nodes Cilium WireGuard
10250 TCP All worker nodes Kubelet API

Prerequisites

Before we touch Kubernetes, eight machines need to exist. I provisioned them using the native cloud CLIs - here’s how:

Fasthosts (2 nodes): Manual provisioning via their console. Two standard VPS instances running Ubuntu 26.04 LTS, 2 vCPU / 4 GB each. Record both public IPs - we’ll need them throughout.

AWS (3 nodes):

$ SG_ID=$(aws ec2 create-security-group --group-name k8s-multicloud-workers \
$   --description "K8s multi-cloud workers" --query "GroupId" --output text)

$ aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 10250 --cidr 0.0.0.0/0
$ aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol udp --port 51871 --cidr 0.0.0.0/0

$ for i in {1..3}; do
$   aws ec2 run-instances --image-id <UBUNTU_26.04_AMI> --count 1 \
$     --instance-type t3.small --key-name k8s-multicloud-key \
$     --security-group-ids $SG_ID --associate-public-ip-address \
$     --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=k8s-worker-aws-0$i},{Key=Tenant,Value=tenant-a}]"
$ done

Gotcha: Use --security-group-ids (plural, with the ID), not --security-groups (singular, with the name). Mixing the two gives you InvalidParameterCombination.

GCP (3 nodes):

$ gcloud compute firewall-rules create k8s-allow-kubelet \
$   --direction=INGRESS --action=ALLOW --rules=tcp:10250 \
$   --source-ranges=0.0.0.0/0 --target-tags=k8s-multicloud-workers

$ gcloud compute firewall-rules create k8s-allow-wireguard \
$   --direction=INGRESS --action=ALLOW --rules=udp:51871 \
$   --source-ranges=0.0.0.0/0 --target-tags=k8s-multicloud-workers

$ PUB_KEY=$(cat ~/.ssh/k8s-multicloud-key.pub)
$ for i in {1..3}; do
$   gcloud compute instances create k8s-worker-gcp-0$i \
$     --machine-type=e2-small --image-family=ubuntu-2604-lts-amd64 \
$     --image-project=ubuntu-os-cloud --tags=k8s-multicloud-workers \
$     --labels=tenant=tenant-b,cloud=gcp --metadata=ssh-keys="ubuntu:$PUB_KEY"
$ done

Step 1: OS Preparation (All 8 Nodes)

Every node needs the same kernel tweaks. SSH into each one and run:

# Kernel modules required by Cilium eBPF
$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
$ overlay
$ br_netfilter
$ EOF

$ sudo modprobe overlay
$ sudo modprobe br_netfilter

# IP forwarding - mandatory for pod networking
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
$ net.bridge.bridge-nf-call-iptables  = 1
$ net.bridge.bridge-nf-call-ip6tables = 1
$ net.ipv4.ip_forward                 = 1
$ EOF

$ sudo sysctl --system

# Kubernetes requires swap disabled
$ sudo swapoff -a
$ sudo sed -i '/swap/d' /etc/fstab

Step 2: Install containerd (All 8 Nodes)

$ sudo apt-get update -qq
$ sudo apt-get install -y -qq containerd

$ sudo mkdir -p /etc/containerd
$ containerd config default | sudo tee /etc/containerd/config.toml > /dev/null

# Cilium requires the systemd cgroup driver
$ sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

$ sudo systemctl restart containerd
$ sudo systemctl enable containerd

Step 3: Install kubeadm, kubelet, kubectl

On control plane nodes (kubectl is only needed here):

$ K8S_VERSION="1.36"
$ K8S_MAJOR=$(echo "$K8S_VERSION" | cut -d. -f1-2)

$ sudo mkdir -p /etc/apt/keyrings
$ curl -fsSL "https://pkgs.k8s.io/core:/stable:/v${K8S_MAJOR}/deb/Release.key" \
$   | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
$   https://pkgs.k8s.io/core:/stable:/v${K8S_MAJOR}/deb/ /" \
$   | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

$ sudo apt-get update -qq
$ sudo apt-get install -y -qq kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl

On worker nodes (no kubectl needed):

# Same steps, but:
$ sudo apt-get install -y -qq kubelet kubeadm

Configure kubelet to advertise the public IP

On Fasthosts CP nodes, the public IP is on a local interface, so --node-ip works:

$ LOCAL_IP=$(curl -s ifconfig.me)
$ echo "KUBELET_EXTRA_ARGS=--node-ip=${LOCAL_IP}" | sudo tee /etc/default/kubelet
$ sudo systemctl restart kubelet

On cloud workers, the public IP is NAT-based - not on any local interface - so kubelet silently ignores --node-ip. This means kubectl get nodes -o wide will show <none> for INTERNAL-IP on AWS and GCP nodes. That’s fine. Cilium maintains its own node registry with the correct public IPs for WireGuard. The Kubernetes Node object’s INTERNAL-IP field is cosmetic.

Step 4: Initialize the Control Plane (k8s-cp-01)

Here’s where we diverge from every Kubernetes tutorial: we’re skipping kube-proxy entirely. Cilium eBPF replaces it at the socket level.

$ CP1_PUBLIC_IP=$(curl -s ifconfig.me)

$ sudo kubeadm init \
$   --apiserver-advertise-address="${CP1_PUBLIC_IP}" \
$   --control-plane-endpoint="${CP1_PUBLIC_IP}:6443" \
$   --pod-network-cidr=10.244.0.0/16 \
$   --kubernetes-version=v1.36.0 \
$   --skip-phases=addon/kube-proxy

Output includes the join commands we’ll need. Save everything.

Set up kubectl

$ mkdir -p $HOME/.kube
$ sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Get join tokens

# Worker join command
$ kubeadm token create --print-join-command

# Certificate key for HA control plane join (needs sudo - this caught me out)
$ sudo kubeadm init phase upload-certs --upload-certs

My cluster’s values:

# Worker join:
kubeadm join 74.208.24.234:6443 --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

# CP join cert key:
<64-CHAR-HEX-KEY>

Step 5: Install Helm

$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
$ helm version

Step 6: Install Cilium CNI with WireGuard

This is the centerpiece of the post. Cilium replaces kube-proxy for service routing, and WireGuard transparently encrypts ALL node-to-node traffic crossing the public internet.

There are two gotchas here that cost me a couple of hours.

Gotcha #1: On a single-node cluster, Cilium tries to reach the API server via the Kubernetes ClusterIP (10.96.0.1:443). But Cilium IS the CNI - it hasn’t routed that IP yet. Deadlock. The fix is --set k8sServiceHost and --set k8sServicePort, which tell Cilium to connect to the API server directly via the public IP until it’s ready to route the ClusterIP itself.

Gotcha #2: The default 2-operator-replica deployment can’t schedule on a single node. Scale to 1 immediately after install.

$ helm repo add cilium https://helm.cilium.io/
$ helm repo update

$ CP1_PUBLIC_IP=$(curl -s ifconfig.me)

$ helm install cilium cilium/cilium \
$   --namespace kube-system \
$   --set kubeProxyReplacement=true \
$   --set encryption.enabled=true \
$   --set encryption.type=wireguard \
$   --set encryption.nodeEncryption=true \
$   --set cluster.name=multi-cloud-k8s \
$   --set ipam.mode=cluster-pool \
$   --set ipam.operator.clusterPoolIPv4PodCIDRList[0]="10.244.0.0/16" \
$   --set k8sServiceHost="${CP1_PUBLIC_IP}" \
$   --set k8sServicePort=6443

# Scale operator for single-node
$ kubectl -n kube-system scale deployment cilium-operator --replicas=1

Wait for Cilium to come up:

$ cilium status --wait

Why encryption.nodeEncryption=true matters: By default, Cilium’s WireGuard only encrypts pod-to-pod traffic. Node encryption ensures ALL traffic - kubelet health checks, host-network daemons, Hubble telemetry - gets encrypted when crossing the public internet.

Step 7: Join the Second Control Plane (k8s-cp-02)

On cp-02, run Steps 1–3 (OS prep, containerd, kubeadm/kubelet) first. Then:

$ sudo kubeadm join 74.208.24.234:6443 \
$   --token <TOKEN> \
$   --discovery-token-ca-cert-hash sha256:<HASH> \
$   --control-plane \
$   --certificate-key <CERT_KEY> \
$   --node-name k8s-cp-02

Both CP nodes have a control-plane taint that blocks regular pods from scheduling. Since we don’t have worker nodes yet, remove the taint from cp-02 so Prometheus and Grafana can land:

$ kubectl taint node k8s-cp-02 node-role.kubernetes.io/control-plane:NoSchedule-

Verify:

$ kubectl get nodes
NAME        STATUS   ROLES           AGE     VERSION
k8s-cp-02   Ready    control-plane   10m     v1.36.3
ubuntu      Ready    control-plane   142m    v1.36.3

Step 8: Verify WireGuard Encryption

With two nodes, WireGuard has a peer to talk to:

# Install tools
$ sudo apt-get install -y wireguard-tools

# Check the WireGuard interface
$ sudo wg show cilium_wg0

interface: cilium_wg0
  listening port: 51871

peer: jPHq6ZW1kD4i1zrbIaDbafi/SVtV7Yy3wrhZ5PWMAhg=
  endpoint: 198.251.78.107:51871
  allowed ips: 198.251.78.107/32
  latest handshake: 32 seconds ago
  transfer: 6.95 KiB received, 6.74 KiB sent

A peer with latest handshake: X seconds ago and actual transfer bytes - the encrypted tunnel is live. Now let’s prove it with tcpdump.

Find the network interface (Ubuntu 26.04 uses predictable names - ens6 in my case, not eth0):

$ ip link show | grep -E '^[0-9]+:' | grep -v lo | head -3

Then:

$ sudo tcpdump -n -i ens6 udp port 51871

00:01:12.292923 IP 198.251.78.107.51871 > 74.208.24.234.51871: UDP, length 144
00:01:12.293187 IP 74.208.24.234.51871 > 198.251.78.107.51871: UDP, length 144
00:01:24.709815 IP 198.251.78.107.51871 > 74.208.24.234.51871: UDP, length 144
00:01:24.710198 IP 74.208.24.234.51871 > 198.251.78.107.51871: UDP, length 144

Only encrypted UDP on port 51871. No plain-text pod traffic anywhere. Every packet between these two VPS instances - sitting on different physical hosts, crossing the open internet - is scrambled inside a WireGuard tunnel.

Step 9: Prometheus + Grafana Observability Stack

The kube-prometheus-stack Helm chart is powerful but heavy. With only 4 GB per CP node, we need to tune it way down. Learned this the hard way - the default install includes AlertManager (we don’t need it for a demo), admission webhooks (need TLS certs that break on small clusters), and a 10-day data retention policy that eats RAM for breakfast.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ helm install prometheus prometheus-community/kube-prometheus-stack \
$   --namespace monitoring \
$   --create-namespace \
$   --set alertmanager.enabled=false \
$   --set prometheus.prometheusSpec.retention=24h \
$   --set prometheus.prometheusSpec.resources.requests.memory=512Mi \
$   --set prometheus.prometheusSpec.resources.requests.cpu=150m \
$   --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
$   --set grafana.adminPassword=admin \
$   --set grafana.resources.requests.memory=128Mi \
$   --set nodeExporter.resources.requests.memory=64Mi \
$   --set kube-state-metrics.resources.requests.memory=128Mi \
$   --set prometheusOperator.admissionWebhooks.enabled=false \
$   --set prometheusOperator.admissionWebhooks.patch.enabled=false \
$   --set prometheusOperator.tls.enabled=false

The last three flags - admissionWebhooks.enabled=false, admissionWebhooks.patch.enabled=false, and tls.enabled=false - prevent the operator from trying to start a TLS-secured webhook server that would need certificates we haven’t provisioned. For a demo cluster, this is fine.

Verify:

$ kubectl -n monitoring get pods
NAME                                                   READY   STATUS    RESTARTS   AGE
prometheus-grafana-554f4fbc7b-ftbgq                    3/3     Running   0          2m
prometheus-kube-prometheus-operator-644d8979cd-xrnft   1/1     Running   0          2m
prometheus-kube-state-metrics-8995b68d6-gb68d          1/1     Running   0          2m
prometheus-prometheus-kube-prometheus-prometheus-0     2/2     Running   0          2m
prometheus-prometheus-node-exporter-r9t6d              1/1     Running   0          2m
prometheus-prometheus-node-exporter-rqvqk              1/1     Running   0          2m

Access Grafana:

$ kubectl -n monitoring port-forward svc/prometheus-grafana 3000:80
# From your local machine: ssh -L 3000:localhost:3000 clusteradmin@<CP1_IP>
# Open http://localhost:3000, login admin/admin

Pre-built dashboards for Kubernetes Compute Resources and Networking are ready to go. For Cilium-specific eBPF metrics, import dashboard 16611 from the Grafana community library and patch the Cilium agent service to expose metrics:

$ kubectl -n kube-system patch svc cilium-agent \
$   -p '{"spec":{"ports":[{"name":"metrics","port":9962,"protocol":"TCP","targetPort":9962}]}}'

Step 10: Enable Hubble

Hubble gives you real-time eBPF flow visualization - pod-to-pod traffic, dropped packets, L4/L7 policy decisions, all rendered as a service map:

$ cilium hubble enable --ui

$ kubectl -n kube-system get pods | grep hubble
hubble-relay-6fb864ff5d-skgbf       1/1     Running   0    26s
hubble-ui-677bcd7f96-jg9d8          2/2     Running   0    26s

Access the UI:

$ cilium hubble ui
# Or: kubectl -n kube-system port-forward svc/hubble-ui 12000:80

Step 11: Join the Worker Nodes

Now for the real multi-cloud part. Each worker needs the OS prep (Step 1), containerd (Step 2), and kubeadm/kubelet (Step 3). Then:

AWS Workers (Tenant A)

On each AWS instance:

$ sudo kubeadm join 74.208.24.234:6443 \
$   --token <TOKEN> \
$   --discovery-token-ca-cert-hash sha256:<HASH> \
$   --node-name k8s-worker-aws-0X    # Replace X with 1, 2, 3

Then on cp-01, label each:

$ kubectl label node k8s-worker-aws-01 cloud=aws tenant=tenant-a
$ kubectl label node k8s-worker-aws-02 cloud=aws tenant=tenant-a
$ kubectl label node k8s-worker-aws-03 cloud=aws tenant=tenant-a

GCP Workers (Tenant B)

On each GCP instance:

$ sudo kubeadm join 74.208.24.234:6443 \
$   --token <TOKEN> \
$   --discovery-token-ca-cert-hash sha256:<HASH> \
$   --node-name k8s-worker-gcp-0X    # Replace X with 1, 2, 3

On cp-01:

$ kubectl label node k8s-worker-gcp-01 cloud=gcp tenant=tenant-b
$ kubectl label node k8s-worker-gcp-02 cloud=gcp tenant=tenant-b
$ kubectl label node k8s-worker-gcp-03 cloud=gcp tenant=tenant-b

Note: --node-labels is not a kubeadm join flag (despite what some docs suggest). Labels are applied with kubectl label node after join.

Cilium DaemonSets auto-deploy to every new node. Node exporters appear automatically. The WireGuard mesh expands - each new node generates a keypair, registers its public key as a Kubernetes annotation, and every other node adds it as a peer.

Step 12: Full Validation

After all 6 workers join, let’s verify the complete platform:

# ── Nodes ──
$ kubectl get nodes -o wide
# Expected: 8 nodes, all Ready

# ── Cilium DaemonSets ──
$ cilium status | grep -E "Desired|Ready|Available"
# Expected: Desired: 8, Ready: 8/8, Available: 8/8

# ── WireGuard peers ──
$ sudo wg show cilium_wg0 | grep -c "latest handshake"
# Expected: 7 (one per other node)

# ── Tenant labels ──
$ kubectl get nodes -l cloud=aws --no-headers | wc -l  # → 3
$ kubectl get nodes -l cloud=gcp --no-headers | wc -l  # → 3

# ── Observability ──
$ kubectl -n monitoring get pods  # all Running
$ kubectl -n kube-system get pods | grep hubble  # relay + UI Running

My final cluster:

$ kubectl get nodes -o wide
NAME                STATUS   ROLES           AGE     VERSION   INTERNAL-IP
k8s-cp-02           Ready    control-plane   131m    v1.36.3   198.251.78.107
k8s-worker-aws-01   Ready    <none>          79m     v1.36.3   <none>
k8s-worker-aws-02   Ready    <none>          61m     v1.36.3   <none>
k8s-worker-aws-03   Ready    <none>          61m     v1.36.3   <none>
k8s-worker-gcp-01   Ready    <none>          38m     v1.36.3   <none>
k8s-worker-gcp-02   Ready    <none>          57m     v1.36.3   <none>
k8s-worker-gcp-03   Ready    <none>          58m     v1.36.3   <none>
ubuntu              Ready    control-plane   4h33m   v1.36.3   74.208.24.234

The <none> INTERNAL-IP on cloud workers is expected - their public IP is NAT-based, not on a local interface. The Kubernetes Node object doesn’t need it. Cilium manages its own node registry with the correct public IPs for WireGuard endpoints.

$ sudo wg show cilium_wg0 | grep -c "latest handshake"
7

Seven peers, all with recent handshakes and active data transfer. The control plane on Fasthosts, the AWS workers in us-east-1, and the GCP workers in us-east1-b - all part of one encrypted mesh.

What We Built

A single Kubernetes cluster spanning three independent infrastructure providers:

  • HA control plane on Fasthosts VPS with etcd quorum across two nodes
  • 3 AWS workers tagged for Tenant A, 3 GCP workers tagged for Tenant B
  • Cilium eBPF replacing kube-proxy for all service routing and load balancing
  • WireGuard encrypting every packet between every node - no plain-text traffic on the public internet
  • Prometheus + Grafana monitoring all 8 nodes with per-node exporters
  • Hubble providing real-time eBPF flow visualization

Every design decision was intentional. The control plane lives on commodity VPS. The data plane is isolated per cloud provider - when we deploy databases in Post 2, Tenant A’s replication traffic never leaves AWS, Tenant B’s never leaves GCP. The only cross-cloud traffic is lightweight control-plane communication, and WireGuard ensures even that is encrypted.

In the next post, I’ll write a custom Kubernetes operator in Go that manages Valkey database clusters. It’ll use those cloud=aws and cloud=gcp node labels for topology-aware scheduling, deploy Sentinel-based high availability, and handle master election and failover automatically. The Prometheus/Grafana stack from this post will show us exactly what’s happening when things go wrong - and I’ll make them go wrong on purpose.