Kubernetes Federation using Consul

As organizations are moving towards a multi-cloud approach, the need for cross platform tools has been on rise. In this article, I am going to explore cross cluster service discovery in kubernetes using Consul.

I will be setting up consul for a federation between GCP’s GKE and AWS’s EKS. Please ensure you have admin access to both these clusters.

Here is the architecture diagram of what we are creating

This article assumes that you already have created the Kubernetes Clusters in both AWS and GCP

Step 1: Installing Consul in Cluster-1, lets call it DC1

As mentioned in the Consul Documentation the best way to install consul in your cluster is via helm chart

Here are pre requisites that we need to run to install consul

#making sure that we are using DC1 cluster, I have renamed my contexts to DC1 and DC2 to avoid confusion
kubectl config use-context DC1
#creating namespace 
kubectl create ns consul
#adding helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com
#confirming 
helm search repo hashicorp/consul

Now that we have added the consul repo we need to work on creating config file

We can use the following file for a basic federation setup. but we can also encrypt the conversation between the Consul deployments and we can also use acls

global:
name: consul
datacenter: dc1
# TLS configures whether Consul components use TLS.
tls:
# TLS must be enabled for federation in Kubernetes.
enabled: true
federation:
enabled: true
# This will cause a Kubernetes secret to be created that
# can be imported by secondary datacenters to configure them
# for federation.
createFederationSecret: true
syncCatalog:
enabled: true
default: false
connectInject:
# Consul Connect service mesh must be enabled for federation.
enabled: true
controller:
enabled: true
server:
replicas: 2
meshGateway:
# Mesh gateways are gateways between datacenters. They must be enabled
# for federation in Kubernetes since the communication between datacenters
# goes through the mesh gateways.
enabled: true

Now that we have created the config file we need to install consul in the DC1 cluster by running the following command

helm install consul hashicorp/consul -f config.yaml -n consul

Once we have installed consul in DC1 it will create a federation kubernetes secret in namespace where consul is installed

This secret is important and has to be shared with the other consul deployment for federation

#grab the secret using 
kubectl get secret -n consul | grep consul-federation
#get the yaml file of the secret and store it in a file
kubectl get secret -n consul consul-federation -o yaml > consulfederation.yaml

Now that we have the secret we can remove resourceVersion, selfLink, and other things that to avoid any errors

Step 2: Installing Consul in Cluster-2 let’s call it DC2

#Switching contexts to now work on DC2
kubectl config use-context DC2

Now we need to add the secret to the DC2 cluster

#Creating Namepace
kubectl create ns consul
#Creating Secret
kubectl apply -f consulfederation.yaml

Once the Secret is created we are going to create the config file for DC2 so save this file as config2.yaml

global:
name: consul
datacenter: dc2
tls:
enabled: true
# Here we're using the shared certificate authority from the primary
# datacenter that was exported via the federation secret.
caCert:
secretName: consul-federation
secretKey: caCert
caKey:
secretName: consul-federation
secretKey: caKey
federation:
enabled: true
connectInject:
enabled: true
syncCatalog:
enabled: true
default: false
controller:
enabled: true
meshGateway:
enabled: true
server:
replicas: 2
# Here we're including the server config exported from the primary
# via the federation secret. This config includes the addresses of
# the primary datacenter's mesh gateways so Consul can begin federation.
extraVolumes:
- type: secret
name: consul-federation
items:
- key: serverConfigJSON
path: config.json
load: true

Install consul in DC2 cluster using this command

helm install consul hashicorp/consul -f config2.yaml -n consul

And voila!

We have create federation between the two consul clusters running on different Kubernetes cluster

#confirming federation
#this will show all the service registered using consul in both the clusters
kubectl exec statefulset/consul-server -n consul -- consul members -wan

Step 3: Accessing Consul UI

To show the UI we can port forward consul service

#port forwarding consul server
kubectl port-forward -n consul svc/consul-server 9090:9090

Now we can access the consul UI in our local browser using url: localhost:9090

which should look like this

Step 4: Calling consul services between clusters

After this step we will be able to call all the services using consul DNS, instead of using kuberentes service names

To achieve this we will use consul DNS instead of kube-dns or core-dns

Now that the consul service is installed in both the clusters, we are going to run the following command to get the cluster IP of the consul service

kubectl get svc -n consul consul-dns -o jsonpath='{.spec.clusterIP}'

This will give the cluster IP of the service

export CONSUL_DNS_IP=<the cluster IP here>

Now we create the ConfigMap for kube-dns

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"consul": ["$CONSUL_DNS_IP"]}

Ensure that the configmap is created

kubectl get configmap kube-dns -n kube-system -o yaml

If the cluster is using Core DNS

kubectl edit configmap coredns -n kube-system

Now we are going to edit coredns config map and add our clusterip

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: coredns
  namespace: kube-system
data:
  Corefile: |
   .:53 {
        <Existing CoreDNS definition>    
   }
  consul {
    errors
    cache 30
    forward . <consul-dns-service-cluster-ip>
  }

Now that the kubedns/coredns is updated the services can call cross cluster services using consul DNS

#Example: Kubernetes servicename
service.namespace.svc.cluster.local
#Example: Consul servicename
service1.service.dc1.consul
service2.service.dc2.consul

To read more about consul and how it works under the hood, refer below official articles

Multi-Cluster Federation

Consul DNS on Kubernetes

Consul DNS Interface

Consul by HashiCorp

Consul by HashiCorp

Consul by HashiCorp

You can always find me in the comments

PS,
Thanks for the suggestions Umesh Kumar