Build a K8S cluster of EMQ X MQTT servers from scratch

Posted Jun 16, 202017 min read

EMQ X Team provides a Helm chart to facilitate users to deploy EMQ X MQTT server on the kubernetes cluster, which is the most recommended EMQ X Team in kubernetes Or the method of deploying EMQ X MQTT server on k3s cluster. This article will use the method of handwritten yaml files to deploy a K8S cluster of EMQ X MQTT servers from scratch, analyze the details and skills in the deployment, and facilitate users to use it flexibly in actual deployment.

Reading this article requires users to understand the basic concepts of kubernetes and have an operational kubernetes cluster.

Deploy a single EMQ X MQTT server node on K8S

Directly deploy EMQ X Broker using Pod

In Kubernetes, the smallest management element is not a separate container, but Pod , Pod is a Kubernetes application The basic unit of execution, that is, it is the smallest and simplest unit created or deployed in the Kubernetes object model. Pod represents a process running on cluster .

EMQ X Broker provides an image on docker hub , so it is easy to deploy EMQ X Broker on a single pod, use kubectl run The command creates a Pod running EMQ X Broker:

$kubectl run emqx --image=emqx/emqx:v4.1-rc.1 --generator=run-pod/v1
pod/emqx created

Check the status of EMQ X Broker:

$kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE
emqx 1/1 Running 0 3m13s

$kubectl exec emqx - emqx_ctl status
Node'emqx@192.168.77.108' is started
emqx 4.1-rc.1 is running

Delete Pod:

$kubectl delete pods emqx
pod "emqx" deleted

Pod is not designed as a persistent resource, it will not survive scheduling failures, node crashes, or other recycling(such as lack of resources, or other maintenance), therefore, a controller is needed to Manage Pod.

Use Deoloyment to deploy pods

Deployment provides a declarative method for Pod and ReplicaSet to replace the previous ReplicationController to facilitate management applications. Typical application scenarios include:

  • Define Deployment to create Pod and ReplicaSet
  • Rolling upgrade and rollback application
  • Expansion and reduction
  • Pause and resume deployment

Use Deployment to deploy an EMQ X Broker Pod:

  • Define Deployment:

    $cat deployment.yaml
    
    apiVersion:apps/v1
    kind:Deployment
    metadata:
      name:emqx-deployment
      labels:
        app:emqx
    spec:
      replicas:1
      selector:
        matchLabels:
          app:emqx
      template:
        metadata:
          labels:
            app:emqx
        spec:
          containers:
          -name:emqx
            image:emqx/emqx:v4.1-rc.1
            ports:
            -name:mqtt
                containerPort:1883
            -name:mqttssl
                containerPort:8883
            -name:mgmt
                containerPort:8081
            -name:ws
                containerPort:8083
            -name:wss
                containerPort:8084
            -name:dashboard
                containerPort:18083
  • Deploy Deployment:

      $kubectl apply -f deployment.yaml
      deployment.apps/emqx-deployment created
  • View deployment status:

      $kubectl get deployment
      NAME READY UP-TO-DATE AVAILABLE AGE
      deployment.apps/emqx-deployment 3/3 3 3 74s
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      pod/emqx-deployment-7c44dbd68-8j77l 1/1 Running 0 74s
    
      $kubectl exec pod/emqx-deployment-7c44dbd68-8j77l - emqx_ctl status
      Node'emqx-deployment-7c44dbd68-8j77l@192.168.77.117' is started
      emqx 4.1-rc.1 is running
  • Try to delete the pod manually

      $kubectl delete pods emqx-deployment-7c44dbd68-8j77l
      pod "emqx-deployment-7c44dbd68-8j77l" deleted
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-deployment-68fcb4bfd6-2nhh6 1/1 Running 0 59s

    The output shows that the EMQ X Broker Pod has been successfully deployed with Deployment. Even if this Pod is terminated unexpectedly, the Deployment will recreate a new Pod.

Use Services to expose EMQ X Broker Pod services

Kubernetes Pods has a life cycle. They can be created, and destruction will not start again. If you use Deployment to run the application, it can dynamically create and destroy pods.

Each Pod has its own IP address, but in a Deployment, the set of Pods running at the same time may be different from the set of Pods running the application later.

This leads to a question:If EMQ X Broker Pod is used to provide services to MQTT clients, how should the client find and track the IP address to connect so that the client can use the EMQ X Broker service?

The answer is:Service

Service is an abstract method that exposes applications running on a set of Pods as network services.

Use Service to expose EMQ X Broker Pod as a network service:

  • Define Service:

      $cat service.yaml
    
      apiVersion:v1
      kind:Service
      metadata:
        name:emqx-service
      spec:
        selector:
          app:emqx
        ports:
          -name:mqtt
            port:1883
            protocol:TCP
            targetPort:mqtt
          -name:mqttssl
            port:8883
            protocol:TCP
            targetPort:mqttssl
          -name:mgmt
            port:8081
            protocol:TCP
            targetPort:mgmt
          -name:ws
            port:8083
            protocol:TCP
            targetPort:ws
          -name:wss
            port:8084
            protocol:TCP
            targetPort:wss
          -name:dashboard
            port:18083
            protocol:TCP
            targetPort:dashboard
  • Deploy Service:

      $kubectl apply -f service.yaml
      service/emqx-service created
  • View deployment

      $kubectl get svc
      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      emqx-service ClusterIP 10.96.54.205 <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP 58s
  • View the API of EMQ X Broker using the IP provided by Service

      $curl 10.96.54.205:8081/status
      Node emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120 is started
      emqx is running

At this point, a single EMQ X Broker node is deployed on kubernetes, EMQ X Broker Pod is managed through Deployment, and the EMQ X Broker service is exposed through Service.

Automatically cluster EMQ X MQTT servers via kubernetes

In the above, a single EMQ X Broker Pod is deployed through Deployment. It is extremely convenient to expand the number of Pods through Deployment. Execute the kubectl scale deployment ${deployment_name} --replicas ${numer} command to expand the number of Pods. Expand EMQ X Broker Pod to 3:

$kubectl scale deployment emqx-deployment --replicas 3
deployment.apps/emqx-deployment scaled

$kubectl get pods
NAME READY STATUS RESTARTS AGE
emqx-deployment-68fcb4bfd6-2nhh6 1/1 Running 0 18m
emqx-deployment-68fcb4bfd6-mpvch 1/1 Running 0 6s
emqx-deployment-68fcb4bfd6-mx55q 1/1 Running 0 6s

$kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 - emqx_ctl status
Node'emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120' is started
emqx 4.1-rc.1 is running

$kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 - emqx_ctl cluster status
Cluster status:#{running_nodes =>
                      ['emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120'],
                  stopped_nodes => []}

It can be seen that the number of EMQ X Broker Pods has been expanded to three, but each Pod is independent and there is no cluster. Next, try to automatically cluster EMQ X Broker Pod through kubernetes.

Modify the configuration of EMQ X Broker

Check the EMQ X Broker documentation about automatic clustering , you can see that you need to modify EMQ X Broker configuration:

cluster.discovery = kubernetes
cluster.kubernetes.apiserver = http://10.110.111.204:8080
cluster.kubernetes.service_name = ekka
cluster.kubernetes.address_type = ip
cluster.kubernetes.app_name = ekka

Where cluster.kubernetes.apiserver is the address of the kubernetes apiserver, which can be obtained through the kubectl cluster-info command, cluster.kubernetes.service_name is the name of the Service above, and cluster.kubernetes.app_name is EMQ X Broker Before the @ symbol in node.name, you need to set the EMQ X Broker in the cluster to a uniform prefix of node.name.

The docker image of EMQ X Broker provides the function of modifying the configuration through environment variables. For details, you can check docker hub or [Github]( https://github .com/emqx/emqx-rel/blob/master/deploy/docker/README.md).

  • Modify the deployment yaml file and add environment variables:

       $cat deployment.yaml
    
       apiVersion:apps/v1
       kind:Deployment
       metadata:
         name:emqx-deployment
         labels:
           app:emqx
       spec:
         replicas:3
         selector:
           matchLabels:
             app:emqx
         template:
           metadata:
             labels:
               app:emqx
           spec:
             containers:
             -name:emqx
               image:emqx/emqx:v4.1-rc.1
               ports:
               -name:mqtt
                 containerPort:1883
               -name:mqttssl
                 containerPort:8883
               -name:mgmt
                 containerPort:8081
               -name:ws
                 containerPort:8083
               -name:wss
                 containerPort:8084
               -name:dashboard
                 containerPort:18083
                 env:
               -name:EMQX_NAME
                   value:emqx
               -name:EMQX_CLUSTER__DISCOVERY
                 value:k8s
               -name:EMQX_CLUSTER__K8S__APP_NAME
                 value:emqx
               -name:EMQX_CLUSTER__K8S__SERVICE_NAME
                 value:emqx-service
               -name:EMQX_CLUSTER__K8S__APISERVER
                 value:"https://kubernetes.default.svc:443"
               -name:EMQX_CLUSTER__K8S__NAMESPACE
                 value:default

Because the `kubectl scale deployment ${deployment_name} --replicas ${numer} command will not modify the yaml file, you need to set spec.replicas:3 when modifying yaml.

The DNS rules for kubernetes are built into the Pod, so https://kubernetes.default.svc:443 will be resolved to the address of the kubernetes apiserver.

  • Delete the previous Deployment and redeploy:

      $kubectl delete deployment emqx-deployment
      deployment.apps "emqx-deployment" deleted
    
      $kubectl apply -f deployment.yaml
      deployment.apps/emqx-deployment created

Give Pod access to kubernetes apiserver

After deploying the deployment above, check the status of EMQ X Broker. You can see that although EMQ X Broker is successfully started, but there is still no cluster success, check the log of EMQ X Broker Pod:

$kubectl get pods
NAME READY STATUS RESTARTS AGE
emqx-deployment-5c8cfc4d75-67lmt 1/1 Running 0 5s
emqx-deployment-5c8cfc4d75-r6jgb 1/1 Running 0 5s
emqx-deployment-5c8cfc4d75-wv2hj 1/1 Running 0 5s

$kubectl exec emqx-deployment-5c8cfc4d75-67lmt - emqx_ctl status
Node'emqx@192.168.87.150' is started
emqx 4.1-rc.1 is running

$kubectl exec emqx-deployment-5c8cfc4d75-67lmt - emqx_ctl cluster status
Cluster status:#{running_nodes => ['emqx@192.168.87.150'],
                  stopped_nodes => []}

$kubectl logs emqx-deployment-76f6895c46-4684f

···

( emqx@192.168.87.150 )1> 2020-05-20 01:48:39.726 [error]Ekka(AutoCluster):Discovery error:{403,
"{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message ":"endpoints \"emqx-service\" is forbidden:User \"system:serviceaccount:default:default\" cannot get resource \"endpoints\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":\ "emqx-service","kind":"endpoints"},"code":403}\n"}
···

Pod was denied access to kubernetes apiserver due to permission issues and returned HTTP 403, so the cluster failed.

Ordinary Pods cannot access kubernetes apiserver. There are two methods to solve this problem. One is to open the http interface of kubernetes apiserver, but this method has certain security risks. The other is to configure RBAC authentication through ServiceAccount, Role, and RoleBinding. right.

  • Define ServiceAccount, Role and RoleBinding:

      $cat rbac.yaml
    
      apiVersion:v1
      kind:ServiceAccount
      metadata:
        namespace:default
        name:emqx
      ---
      kind:Role
      apiVersion:rbac.authorization.kubernetes.io/v1beta1
      metadata:
        namespace:default
        name:emqx
      rules:
      -apiGroups:
resources:
-endpoints
verbs:
-get
-watch
-list

---
kind:RoleBinding
apiVersion:rbac.authorization.kubernetes.io/v1beta1
metadata:

namespace:default
name:emqx

subjects:

-kind:ServiceAccount
  name:emqx
  namespace:default

roleRef:

kind:Role
name:emqx
apiGroup:rbac.authorization.kubernetes.io
  • Deploy the corresponding resources:

      $kubectl apply -f rbac.yaml
      serviceaccount/emqx created
      role.rbac.authorization.kubernetes.io/emqx created
      rolebinding.rbac.authorization.kubernetes.io/emqx created
  • Modify the yaml file of Deployment, add spec.template.spec.serviceAccountName, and redeploy:

      $cat deployment.yaml
    
      apiVersion:apps/v1
      kind:Deployment
      metadata:
        name:emqx-deployment
        labels:
          app:emqx
      spec:
        replicas:3
        selector:
          matchLabels:
            app:emqx
        template:
          metadata:
            labels:
              app:emqx
          spec:
            serviceAccountName:emqx
            containers:
            -name:emqx
              image:emqx/emqx:v4.1-rc.1
              ports:
              -name:mqtt
                containerPort:1883
              -name:mqttssl
                containerPort:8883
              -name:mgmt
                containerPort:8081
              -name:ws
                containerPort:8083
              -name:wss
                containerPort:8084
              -name:dashboard
                containerPort:18083
                env:
              -name:EMQX_NAME
                  value:emqx
              -name:EMQX_CLUSTER__DISCOVERY
                value:kubernetes
              -name:EMQX_CLUSTER__K8S__APP_NAME
                value:emqx
              -name:EMQX_CLUSTER__K8S__SERVICE_NAME
                value:emqx-service
              -name:EMQX_CLUSTER__K8S__APISERVER
                value:"https://kubernetes.default.svc:443"
              -name:EMQX_CLUSTER__K8S__NAMESPACE
                value:default
    
               $kubectl delete deployment emqx-deployment
               deployment.apps "emqx-deployment" deleted
               $kubectl apply -f deployment.yaml
               deployment.apps/emqx-deployment created
  • View status:

      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-deployment-6b854486c-dhd7p 1/1 Running 0 10s
      emqx-deployment-6b854486c-psv2r 1/1 Running 0 10s
      emqx-deployment-6b854486c-tdzld 1/1 Running 0 10s
    
      $kubectl exec emqx-deployment-6b854486c-dhd7p - emqx_ctl status
      Node'emqx@192.168.77.92' is started
      emqx 4.1-rc.1 is running
    
      $kubectl exec emqx-deployment-6b854486c-dhd7p - emqx_ctl cluster status
      Cluster status:#{running_nodes =>
                            ['emqx@192.168.77.115','emqx@192.168.77.92',
                             'emqx@192.168.87.157'],
                        stopped_nodes => []}
  • Abort a Pod:

      $kubectl delete pods emqx-deployment-6b854486c-dhd7p
      pod "emqx-deployment-6b854486c-dhd7p" deleted
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-deployment-6b854486c-846v7 1/1 Running 0 56s
      emqx-deployment-6b854486c-psv2r 1/1 Running 0 3m50s
      emqx-deployment-6b854486c-tdzld 1/1 Running 0 3m50s
    
      $kubectl exec emqx-deployment-6b854486c-846v7 - emqx_ctl cluster status
      Cluster status:#{running_nodes =>
                            ['emqx@192.168.77.115','emqx@192.168.77.84',
                             'emqx@192.168.87.157'],
                        stopped_nodes => ['emqx@192.168.77.92']}

    The output shows that EMQ X Broker will correctly display the stopped Pods and add the Pods created by Deployment to the cluster.

So far, EMQ X Broker has successfully established a cluster on kubernetes.

Persistent EMQ X Broker cluster

The Deployment used above manages the Pod, but the Pod network is constantly changing, and when the Pod is destroyed and rebuilt, the data and configuration stored in the EMQ X Broker will disappear, which is unacceptable in production. Next, try to make the cluster of EMQ X Broker persistent. Even if the Pod is destroyed and rebuilt, the data of EMQ X Broker can still be saved.

ConfigMap

ConfigMap Yes configMap is an API object used to save non-confidential data to health value pairs. When used, it can be used as environment variable, command line parameter or configuration file in storage volume.

ConfigMap decouples your environment configuration information from container image to facilitate application configuration modification.

ConfigMap does not provide confidentiality or encryption functions. If the data you want to store is confidential, please use Secret , or use other third-party tools to ensure the privacy of your data, and Not using ConfigMap.

Next, use ConfigMap to record the configuration of EMQ X Broker and import them into the deployment as environment variables.

  • Define Configmap and deploy:

      $cat configmap.yaml
    
      apiVersion:v1
      kind:ConfigMap
      metadata:
        name:emqx-config
      data:
        EMQX_CLUSTER__K8S__ADDRESS_TYPE:"hostname"
        EMQX_CLUSTER__K8S__APISERVER:"https://kubernetes.default.svc:443"
        EMQX_CLUSTER__K8S__SUFFIX:"svc.cluster.local"
    
      $kubectl apply -f configmap.yaml
      configmap/emqx-config created
  • Configure Deployment to use Configmap

      $cat deployment.yaml
    
      apiVersion:apps/v1
      kind:Deployment
      metadata:
        name:emqx-deployment
        labels:
          app:emqx
      spec:
        replicas:3
        selector:
          matchLabels:
            app:emqx
        template:
          metadata:
            labels:
              app:emqx
          spec:
            serviceAccountName:emqx
            containers:
            -name:emqx
              image:emqx/emqx:v4.1-rc.1
              ports:
              -name:mqtt
                containerPort:1883
              -name:mqttssl
                containerPort:8883
              -name:mgmt
                containerPort:8081
              -name:ws
                containerPort:8083
              -name:wss
                containerPort:8084
              -name:dashboard
                containerPort:18083
                envFrom:
                -configMapRef:
                    name:emqx-config
  • Redeploy the deployment and check the status

      $kubectl delete -f deployment.yaml
      deployment.apps "emqx-deployment" deleted
    
      $kubectl apply -f deployment.yaml
      deployment.apps/emqx-deployment created
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-deployment-5c7696b5d7-k9lzj 1/1 Running 0 3s
      emqx-deployment-5c7696b5d7-mdwkt 1/1 Running 0 3s
      emqx-deployment-5c7696b5d7-z57z7 1/1 Running 0 3s
    
      $kubectl exec emqx-deployment-5c7696b5d7-k9lzj - emqx_ctl status
      Node'emqx@192.168.87.149' is started
      emqx 4.1-rc.1 is running
    
      $kubectl exec emqx-deployment-5c7696b5d7-k9lzj - emqx_ctl cluster status
      Cluster status:#{running_nodes =>
                            ['emqx@192.168.77.106','emqx@192.168.77.107',
                             'emqx@192.168.87.149'],
                        stopped_nodes => []}

The configuration files of EMQ X Broker have been decoupled into Configmap. If necessary, one or more Configmaps can be freely configured and imported into the Pod as environment variables or files.

StatefulSet

StatefulSet is to solve the problem of stateful services(the corresponding Deployments and ReplicaSets are designed for stateless services), and its application scenarios include

  • Stable persistent storage, that is, Pod can still access the same persistent data after rescheduling, based on PVC
  • Stable network sign, that is, PodName and HostName remain unchanged after Pod rescheduling, based on Headless Service(that is, Service without Cluster IP)
  • Ordered deployment, orderly expansion, that is, Pods are in order, and must be carried out in sequence according to the defined order when deploying or expanding(that is, from 0 to N-1, all previous Pods must be before the next Pod runs) Is Running and Ready state), based on init containers
  • Orderly shrink, orderly delete(ie from N-1 to 0)

It can be found from the application scenario above that the StatefulSet consists of the following parts:

  • Headless Service for defining the network domain(DNS domain)
  • VolumeClaimTemplates for creating PersistentVolumes
  • Define StatefulSet for specific applications

The DNS format of each Pod in the StatefulSet is statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local, where

  • serviceName is the name of the headless service
  • 0..N-1 is the serial number where the Pod is located, starting from 0 to N-1
  • statefulSetName is the name of StatefulSet
  • namespace is the namespace where the service is located, Headless Servic and StatefulSet must be in the same namespace
  • .cluster.local is Cluster Domain

Next, use StatefulSet instead of Deployment to manage pods.

  • Delete Deployment:

      $kubectl delete deployment emqx-deployment
      deployment.apps "emqx-deployment" deleted
  • Define StatefulSet:

      $cat statefulset.yaml
    
      apiVersion:apps/v1
      kind:StatefulSet
      metadata:
        name:emqx-statefulset
        labels:
          app:emqx
      spec:
          serviceName:emqx-headless
        updateStrategy:
          type:RollingUpdate
        replicas:3
        selector:
          matchLabels:
            app:emqx
        template:
          metadata:
            labels:
              app:emqx
          spec:
            serviceAccountName:emqx
            containers:
            -name:emqx
              image:emqx/emqx:v4.1-rc.1
              ports:
              -name:mqtt
                containerPort:1883
              -name:mqttssl
                containerPort:8883
              -name:mgmt
                containerPort:8081
              -name:ws
                containerPort:8083
              -name:wss
                containerPort:8084
              -name:dashboard
                containerPort:18083
               envFrom:
                -configMapRef:
                    name:emqx-config

Note that StatefulSet needs Headless Service to achieve stable network logo, so you need to define a Service

$cat headless.yaml

apiVersion:v1
kind:Service
metadata:
  name:emqx-headless
spec:
  type:ClusterIP
  clusterIP:None
  selector:
    app:emqx
  ports:
  -name:mqtt
    port:1883
    protocol:TCP
    targetPort:1883
  -name:mqttssl
    port:8883
    protocol:TCP
    targetPort:8883
  -name:mgmt
    port:8081
    protocol:TCP
    targetPort:8081
  -name:websocket
    port:8083
    protocol:TCP
    targetPort:8083
  -name:wss
    port:8084
    protocol:TCP
    targetPort:8084
  -name:dashboard
    port:18083
    protocol:TCP
    targetPort:18083

Because the Headless Service does not require an IP, clusterIP:None is configured.

  • Deploy the corresponding resources:

      $kubectl apply -f headless-service.yaml
      service/emqx-headless created
    
      $kubectl apply -f statefulset.yaml
      statefulset.apps/emqx-deployment created
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-statefulset-0 1/1 Running 0 2m59s
      emqx-statefulset-1 1/1 Running 0 2m57s
      emqx-statefulset-2 1/1 Running 0 2m54s
    
      $kubectl exec emqx-statefulset-0 - emqx_ctl cluster status
      Cluster status:#{running_nodes =>
                            ['emqx@192.168.77.105','emqx@192.168.87.153',
                             'emqx@192.168.87.155'],
                        stopped_nodes => []}
  • Update Configmap:

    StatefulSet provides a stable network logo. EMQ X Broker supports the use of hostname and dns rules to implement IP clustering. Taking hostname as an example, you need to modify emqx.conf:

      cluster.kubernetes.address_type = hostname
      cluster.kubernetes.suffix = "svc.cluster.local"

    The DNS rules of Pods in the kubernetes cluster can be customized by users. EMQ X Broker provides cluster.kubernetes.suffix to facilitate users to match their own DNS rules. This article uses the default DNS rules:statefulSetName-{0..N- 1}.serviceName.namespace.svc.cluster.local, the serviceName in the DNS rule is the Headless Service used by StatefulSet, so you also need to modify cluster.kubernetes.service_name to Headless Service Name.

    To turn configuration items into environment variables, you need to configure them in Configmap:

      EMQX_CLUSTER__K8S__ADDRESS_TYPE:"hostname"
      EMQX_CLUSTER__K8S__SUFFIX:"svc.cluster.local"
      EMQX_CLUSTER__K8S__SERVICE_NAME:emqx-headless

    Configmap provides hot update function, execute $kubectl edit configmap emqx-config to hot update Configmap.

  • Redeploy StatefulSet:

    After the Configmap is updated, the Pod will not restart, and we need to manually update the StatefulSet

      $kubectl delete statefulset emqx-statefulset
      statefulset.apps "emqx-statefulset" deleted
    
      $kubectl apply -f statefulset.yaml
      statefulset.apps/emqx-statefulset created
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-statefulset-0 1/1 Running 0 115s
      emqx-statefulset-1 1/1 Running 0 112s
      emqx-statefulset-2 1/1 Running 0 110s
    
      $kubectl exec emqx-statefulset-2 - emqx_ctl cluster status
      Cluster status:#{running_nodes =>
                            ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local',
                             'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',
                             'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'],
                        stopped_nodes => []}

    You can see that the new EMQ X Broker cluster has been successfully established.

  • Abort a Pod:

    The PodName and HostName of the Pod in the StatefulSet will not change after rescheduling. Let's try it out as follows:

      $kubectl get pods
      kuNAME READY STATUS RESTARTS AGE
      emqx-statefulset-0 1/1 Running 0 6m20s
      emqx-statefulset-1 1/1 Running 0 6m17s
      emqx-statefulset-2 1/1 Running 0 6m15s
    
      $kubectl delete pod emqx-statefulset-0
      pod "emqx-statefulset-0" deleted
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-statefulset-0 1/1 Running 0 27s
      emqx-statefulset-1 1/1 Running 0 9m45s
      emqx-statefulset-2 1/1 Running 0 9m43s
    
      $kubectl exec emqx-statefulset-2 - emqx_ctl cluster status
      Cluster status:#{running_nodes =>
                            ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local',
                             'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',
                             'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'],
                        stopped_nodes => []}

    As expected, the StatefulSet rescheduled a Pod with the same network logo, and the EMQ X Broker in the Pod successfully joined the cluster.

StorageClasses, PersistentVolume and PersistentVolumeClaim
================================================== =====

PersistentVolume(PV) is a storage set by the administrator, it is part of the cluster. Just as nodes are resources in the cluster, PV is also a resource in the cluster. PV is a volume plug-in such as Volume, but has a life cycle independent of the pods that use PV. This API object contains storage implementation details, namely NFS, iSCSI, or cloud vendor-specific storage systems.

PersistentVolumeClaim(PVC) is a user storage request. It is similar to Pod. Pods consume node resources, and PVC consumes PV resources. Pods can request specific levels of resources(CPU and memory). The declaration can request a specific size and access mode(for example, it can be mounted in a read/write once or read-only multiple mode).

StorageClass provides administrators with a method to describe storage "class". Different classes may be mapped to different quality of service levels or backup strategies, or any strategy determined by the cluster administrator. Kubernetes itself is not clear what the various classes represent. This concept is sometimes referred to as a "profile" in other storage systems.

When deploying EMQ X Broker, you can create PV or StorageClass in advance, and then use PVC to mount the /opt/emqx/data/mnesia directory of EMQ X Broker. When Pods are rescheduled, EMQ X will start from Get the data in the /opt/emqx/data/mnesia directory and restore it to achieve the persistence of EMQ X Broker.

  • Define StatefulSet

      $cat statefulset.yaml
    
      apiVersion:apps/v1
      kind:StatefulSet
      metadata:
        name:emqx-statefulset
        labels:
          app:emqx
      spec:
        replicas:3
        serviceName:emqx-headless
        updateStrategy:
          type:RollingUpdate
        selector:
          matchLabels:
            app:emqx
        template:
          metadata:
            labels:
              app:emqx
          spec:
            volumes:
            -name:emqx-data
              persistentVolumeClaim:
                claimName:emqx-pvc
             serviceAccountName:emqx
             containers:
            -name:emqx
              image:emqx/emqx:v4.1-rc.1
              ports:
              -name:mqtt
                containerPort:1883
              -name:mqttssl
                containerPort:8883
              -name:mgmt
                containerPort:8081
              -name:ws
                containerPort:8083
              -name:wss
                containerPort:8084
              -name:dashboard
                containerPort:18083
               envFrom:
                -configMapRef:
                    name:emqx-config
                    volumeMounts:
              -name:emqx-data
                mountPath:"/opt/emqx/data/mnesia"
               volumeClaimTemplates:
        -metadata:
            name:emqx-pvc
            annotations:
              volume.alpha.kubernetes.io/storage-class:manual
          spec:
            accessModes:["ReadWriteOnce"]
            resources:
              requests:
                storage:1Gi

The file first specifies a PVC resource named emqx-pvc using the storage class whose name is manual through the VolumeClaimTemplates. The read and write mode of the PVC resource is ReadWriteOnce, which requires 1Gi of space, and then defines this PVC as The volume whose name is emqx-data, and mount the volume in the /opt/emqx/data/mnesia directory in the Pod.

  • Deployment resources:

    Before deploying StatefulSet, users or kubernetes cluster administrators need to create storage classes by themselves.

      $kubectl apply -f statefulset.yaml
      statefulset.apps/emqx-statefulset created
    
      $kubectl get pods
      NAME READY STATUS RESTARTS AGE
      emqx-statefulset-0 1/1 Running 0 27s
      emqx-statefulset-1 1/1 Running 0 9m45s
      emqx-statefulset-2 1/1 Running 0 9m43s
    
      $kubectl get pvc
      NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
      emqx-data-emqx-statefulset-0 Bound pvc-8094cd75-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 2m11s
      emqx-data-emqx-statefulset-0 Bound pvc-9325441d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 99s
      emqx-data-emqx-statefulset-0 Bound pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 56s

    The output shows that the status of the PVC is Bound, and the PVC storage has been successfully established. When the Pod is rescheduled, EMQ X Broker will read the data mounted on the PVC to achieve persistence.

EMQ X Broker has established a persistent cluster on kubernetes. This article skipped some details. The deployment process is also biased towards a simple Demo. Users can read kubernetes documentation and Helm chart source code provided by EMQ X Team to continue in-depth research, of course, welcome to Github contributes issues, pull requests, and start.

Copyright statement:This article is EMQ original, please indicate the source when reprinting.

Original link: https://www.emqx.io/cn/blog/e...