The kubernetes Operator Pattern is exactly what it sounds like. It is code or automation that acts like the set of runbooks a human operator of an application would use to deploy, monitor, troubleshoot, upgrade and decommission an application. The operator can do these functions because it has a deep understanding of the application it is operating on and knowns if the application is functioning correctly.
When Solace first introduced a Kubernetes solution back in the days of Kubernetes 1.5 and 1.6, Operator concepts where very much in flux. For this reason, we based our Solution on Helm to do initial deployments and rolling upgrades and config-maps as the glue that binds together the highly available cluster of brokers together. But today Kubernetes Operators are far more prevalent and stable. They make extending Kubernetes behaviour to manage stateful applications like event brokers and databases possible via custom resource and custom resource definitions. So for organizations that manage their deployments through custom resources and operators a Solace operator would be useful, here is why.
All Kubernetes deployments are a collection of resources. Services, Persistent Volumes, Secrets, Pods, are example of basic resources that are composed together to make up a deployment. With a Solace custom resources as defined in an operator, then a Solace highly available cluster could also be used as another resource to compose a deployment. Its life cycle tied to the deployment and maintained within the deployment. It would be possible for example to upgrade brokers and applications in one operation.
The easiest first step toward a Solace Operator is to wrap the existing Helm charts and Kubernetes templates into an operator based with the Operator SDK. So, lets see how to do that.
$ helm repo add solacecharts https://solaceproducts.github.io/pubsubplus-kubernetes-quickstart/helm-charts
Now that you have the Solace defined helm chart, use this to create the operator with the following workflow:
$ cd ~/"<go_workspace>"/src;
$ mkdir pubsubplus-ha-operator
$ cd pubsubplus-ha-operator
$ operator-sdk init --plugins=helm.sdk.operatorframework.io/v1 --domain=solace.com --helm-chart=solacecharts/pubsubplus-ha
You will recieve this warning and need to take action:
WARN[0000] "The RBAC rules generated in config/rbac/role.yaml are based on the chart's default manifest. Some rules may be missing for resources that are only enabled with custom values, and some existing rules may be overly broad. Double check the rules generated in config/rbac/role.yaml to ensure they meet the operator's permission requirements."
Ensure roles are correct to modify pod lables. Ensure these line exist in config/rbac/role.yaml
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["patch"]
Install the CRD
$ make install
/"<go_workspace>"/src/pubsubplus-ha-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/pubsubplushas.charts.solace.com created
Define docker repo and imager name:
$ export IMG="<docker_repo>"/pubsubplus-ha-operator:v0.0.1
Build the operator container and push to docker repo:
$ make docker-build docker-push IMG=$IMG
Deploy the operator container:
$ make deploy IMG=$IMG
cd config/manager && /<go_workspace>/src/pubsubplus-ha-operator/bin/kustomize edit set image controller=<docker_repo>/pubsubplus-ha-operator:v0.0.1
/<go_workspace>/src/pubsubplus-ha-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/pubsubplus-ha-operator-system created
customresourcedefinition.apiextensions.k8s.io/pubsubplushas.charts.solace.com unchanged
role.rbac.authorization.k8s.io/pubsubplus-ha-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/pubsubplus-ha-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/pubsubplus-ha-operator-proxy-role created
clusterrole.rbac.authorization.k8s.io/pubsubplus-ha-operator-metrics-reader created
rolebinding.rbac.authorization.k8s.io/pubsubplus-ha-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pubsubplus-ha-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pubsubplus-ha-operator-proxy-rolebinding created
service/pubsubplus-ha-operator-controller-manager-metrics-service created
deployment.apps/pubsubplus-ha-operator-controller-manager created
Validate the controller is running
$ kubectl get deployment --namespace pubsubplus-ha-operator-system
NAME READY UP-TO-DATE AVAILABLE AGE
pubsubplus-ha-operator-controller-manager 1/1 1 1 2m50s
Validate the crd loads into Kubernetes
$ kubectl get crd
NAME CREATED AT
pubsubplushas.charts.solace.com 2020-08-17T22:53:03Z
The next steps create a Solace HA cluster using the newly deployed operator:
Edit the opertor.yaml file to set admin password
$ vi config/samples/charts_v1alpha1_pubsubplusha.yaml
solace:
redundancy: true
size: prod100
usernameAdminPassword: "<new admin password>"
Finally run the operator to start the Solace cluster
$ kubectl apply -f config/samples/charts_v1alpha1_pubsubplusha.yaml
Validate the Solace brokers are live and ready.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pubsubplusha-sample-pubsubplus-ha-0 1/1 Running 0 96s
pubsubplusha-sample-pubsubplus-ha-1 1/1 Running 0 96s
pubsubplusha-sample-pubsubplus-ha-2 1/1 Running 0 95s
Go on the primary broker with node ordinal 0 and validate "Redundancy Status" = "Up". This means that all the brokers are syncronized and ready to recieve messages.
kubectl exec -it pubsubplusha-sample-pubsubplus-ha-0 cli
pubsubplusha-sample-pubsubplus-ha-0> show redundancy
Configuration Status : Enabled
Redundancy Status : Up
Operating Mode : Message Routing Node
Switchover Mechanism : Hostlist
Auto Revert : No
Redundancy Mode : Active/Standby
Active-Standby Role : Primary
Mate Router Name : pubsubplushasamplepubsubplusha1
ADB Link To Mate : Up
ADB Hello To Mate : Up
Get the name of the loadbalancing Service that fronts the Solace cluster. This will be the name used be applications connect to Solace.
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pubsubplusha-sample-pubsubplus-ha LoadBalancer 10.97.76.97 <pending> 2222:31348/TCP,8080:31717/TCP,1943:32170/TCP,55555:32597/TCP,55003:30106/TCP,55443:32318/TCP,8008:31054/TCP,1443:32713/TCP,5672:32610/TCP,1883:30046/TCP,9000:31317/TCP 5m4s
Using the test application found here to test, we see successful send and receipt of messages.
[21:13:32]
*** Subscriber to topic "tutorial/topic" is ready to connect ***
[21:13:32] Connecting to Solace message router using url: ws://pubsubplusha-sample-pubsubplus-ha:8008
[21:13:32] Client username: subscriber
[21:13:32] Solace message router VPN name: default
[21:13:32] Press Ctrl-C to exit
[21:13:32] === Successfully connected and ready to subscribe. ===
[21:13:32] Subscribing to topic: tutorial/topic
[21:13:32] Successfully subscribed to topic: tutorial/topic
[21:13:32] === Ready to receive messages. ===
[21:17:40] Received message: "Sample Message", details:
Destination: [Topic tutorial/topic]
Class Of Service: COS1
DeliveryMode: DIRECT
Binary Attachment: len=14
53 61 6d 70 6c 65 20 4d 65 73 73 61 67 65 Sample.Message
From here you should be able to stress the cluster be deleting nodes,(in public clouds), or pods and watch the Solace HA cluster heal itself.
Finally, to clean up and remove Solace HA cluster, the operator and the CRD.
kubectl delete -f config/samples/charts_v1alpha1_pubsubplusha.yaml
make undeploy
Note: this will not automatically delete the persistent volumes.
kubectl get pvc
kubectl delete -R pvc <no-longer-needed-pvc>