Try a Sample
This guide lets you quickly evaluate KusionStack Controller Mesh.
Install Controller Mesh Manager
Controller Mesh requires Kubernetes version >= 1.18
Install with helm
# Firstly add KusionStack charts repository if you haven't do this.
$ helm repo add kusionstack https://kusionstack.github.io/charts
# To update the kusionstack repo.
$ helm repo update kusionstack
# Install the latest version.
$ helm install ctrlmesh kusionstack/ctrlmesh
# Wait manager ready
$ kubectl -n ctrlmesh get po
NAME READY STATUS RESTARTS AGE
ctrlmesh-57d6b4df57-mdslc 1/1 Running 0 40s
ctrlmesh-57d6b4df57-mtv2s 1/1 Running 0 40s
Install manager with more options
Try with sample-operator
Here is an example of a Deployment enabling sharding.
Get and deploy sample-operator v0
# Clone and checkout branch sample-operator.
$ git clone -b sample-operator https://github.com/KusionStack/controller-mesh.git
$ cd sample
# Make sure you have kind or test cluster, and kubectl is available.
# Deploy default sample-operator v0.
$ IMAGE_TAG=v0.1.0 make deploy
namespace/kusionstack-sample created
serviceaccount/kusionstack-controller-manager created
role.rbac.authorization.k8s.io/kusionstack-leader-election-role created
clusterrole.rbac.authorization.k8s.io/kusionstack-manager-role created
rolebinding.rbac.authorization.k8s.io/kusionstack-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/kusionstack-sample-manager-rolebinding created
deployment.apps/kusionstack-sample-operator-v0 created
# kusionstack-sample-operator-v0 is created.
$ kubectl get deploy -n kusionstack-sample
NAME READY UP-TO-DATE AVAILABLE AGE
kusionstack-sample-operator-v0 2/2 2 2 14s
$ kubectl get po -n kusionstack-sample
NAME READY STATUS RESTARTS AGE
kusionstack-sample-operator-v0-66f7595c7b-n4c47 1/1 Running 0 50s
kusionstack-sample-operator-v0-66f7595c7b-wxwtv 1/1 Running 0 50s
# sample-operator uses leader-election. Only one leader pod reconciling.
$ kubectl -n kusionstack-sample get lease
NAME HOLDER AGE
sample-operator-leader kusionstack-sample-operator-v0-66f7595c7b-wxwtv_c0ed684d-f332-47f6-890c-dd7e489486f2 53
Play with ShardingConfig
By configuring ShardingConfig appropriately, you can achieve canary and sharding deploy.
Isolate canary namespaces
# Create some test namespaces([foo-01, foo-02, ..., foo-31]).
$ chmod +x ./scripts/create-ns-foo.sh && ./scripts/create-ns-foo.sh
# All namespaces are controlled by sample-operator v0.
$ kubectl get ns -l sample.kusionstack.io/control-by=kusionstack-sample-operator-v0-66f7595c7b-wxwtv
NAME STATUS AGE
default Active 12d
foo-01 Active 78s
foo-02 Active 78s
foo-03 Active 78s
... ... ...
foo-32 Active 78s
# There are more details in leader pod log.
$ kubectl logs kusionstack-sample-operator-v0-66f7595c7b-wxwtv -n kusionstack-sample | grep "hold namespaces"
I0110 09:32:50.950535 1 runner.go:101] hold namespaces [ctrlmesh default foo-01 foo-02 foo-03 foo-04 foo-05 foo-06 foo-07 foo-08 foo-09 foo-10 foo-11 foo-12 foo-13 foo-14 foo-15 foo-16 foo-17 foo-18 foo-19 foo-20 foo-21 foo-22 foo-23 foo-24 foo-25 foo-26 foo-27 foo-28 foo-29 foo-30 foo-31 foo-32 kusionstack-sample kusionstack-system local-path-storage]
# Apply sample ShardingConfigs
./bin/kustomize build config/shardingconfig/canary | kubectl apply -f -
shardingconfig.ctrlmesh.kusionstack.io/kusionstack-sample-operator-0-canary created
shardingconfig.ctrlmesh.kusionstack.io/kusionstack-sample-operator-1-normal created
The kusionstack-sample-operator-0-canary has restricted the scope of namespaces [foo-01, foo-02, foo-03] reconciled by version v1.
And kusionstack-sample-operator-1-normal decided that other namespaces will be reconciled by version v0.
# Patch labels to pod template to inject sidecar and ShardingConfig
$ kubectl -n kusionstack-sample patch deployment kusionstack-sample-operator-v0 --type=strategic --patch \
'spec:
template:
metadata:
labels:
ctrlmesh.kusionstack.io/enable-proxy: "true"
ctrlmesh.kusionstack.io/watching: "true"'
# Mesh proxy container was injected
$ kubectl get po -n kusionstack-sample
NAME READY STATUS RESTARTS AGE
kusionstack-sample-operator-v0-6944bb4bf5-gclqq 2/2 Running 0 30s
kusionstack-sample-operator-v0-6944bb4bf5-lfwdb 2/2 Running 0 41s
# Find current leader
# sharding lease format: ${leader-election-name}---${shardingconfig-name}
$ kubectl get lease -n kusionstack-sample
NAME HOLDER AGE
sample-operator-leader---kusionstack-sample-operator-1-normal kusionstack-sample-operator-v0-6944bb4bf5-lfwdb_497a7962-a5f1-465e-b8ef-6e35660c63f4 32s
# Namespaces [foo-1, foo-2, foo-3] are no longer under v0 control.
$ kubectl logs kusionstack-sample-operator-v0-6944bb4bf5-lfwdb -c manager | grep "namespaces"
... hold namespaces [default foo-04 foo-05 ... foo-32]
Deploy canary sample-operator v1
# Apply sample operator v1 which deployment already labeled
$ ./bin/kustomize build config/manager-v1 | kubectl apply -f -
deployment.apps/kusionstack-sample-operator-v1 created
# Two pods created
$ kubectl get po -n kusionstack-sample
NAME READY STATUS RESTARTS AGE
kusionstack-sample-operator-v0-6944bb4bf5-gclqq 2/2 Running 0 4m
kusionstack-sample-operator-v0-6944bb4bf5-lfwdb 2/2 Running 0 4m
kusionstack-sample-operator-v1-7b6bbb49c8-kbgww 0/2 ContainerCreating 0 3s
kusionstack-sample-operator-v1-7b6bbb49c8-qbzjj 0/2 ContainerCreating 0 3s
# The canary shard uses a separate lease
$ kubectl get lease -n kusionstack-sample
NAME HOLDER AGE
sample-operator-leader---kusionstack-sample-operator-0-canary kusionstack-sample-operator-v1-7b6bbb49c8-qbzjj_64272983-c59a-4574-933d-7d5fea7a1e35 15s
sample-operator-leader---kusionstack-sample-operator-1-normal kusionstack-sample-operator-v0-6944bb4bf5-lfwdb_497a7962-a5f1-465e-b8ef-6e35660c63f4 4m
# Only foo-01, foo-02, foo-03 controlled by v1
$ kubectl get ns -l sample.kusionstack.io/control-by=v1 -n kusionstack-sample
NAME STATUS AGE
foo-01 Active 4m
foo-02 Active 4m
foo-03 Active 4m
$ kubectl logs kusionstack-sample-operator-v1-7b6bbb49c8-qbzjj -c manager | grep namespaces
... hold namespaces [foo-01 foo-02 foo-03]
Similarly, if you want to have more shards, you need to do the following steps:
- Extract a portion of the namespace from the existing ShardingConfigs.
- Configure a new ShardingConfig and apply it.
- Recreate or restart the existing pods to make the new ShardingConfig take effect.
- Scale out the Pods for the new ShardingConfig.
tip
Beta: We try to support automatic sharding strategy. With automatic sharding configuration, there is no need to manually configure each shard's configuration. It manages multiple sub-shardingconfigs automatically through a root configuration.
Try with Operating
For StatefulSet case, you can use the Operating v0.1.1 available here.
Deploy the sample operator with ShardingConfig:
$ helm repo update
$ helm install sample-operating kusionstack/operating \
--version v0.2.0 \
--set sharding.enabled=true \
--set sharding.isDemo=true
$ kubectl -n kusionstack-system get sts
NAME READY AGE
kusionstack-operating 5/5 1m45s
# The proxy container will be automatically injected into the pod
$ kubectl -n kusionstack-system get po
NAME READY STATUS RESTARTS AGE
kusionstack-operating-0 2/2 Running 0 42s
kusionstack-operating-1 2/2 Running 0 32s
kusionstack-operating-2 2/2 Running 0 21s
kusionstack-operating-3 2/2 Running 0 12s
kusionstack-operating-4 0/2 ContainerCreating 0 1s
# Now we have three shards with three lease.
# operating-0-canary -> [kusionstack-operating-0]
# operating-1-normal -> [kusionstack-operating-1, kusionstack-operating-2]
# operating-2-normal -> [kusionstack-operating-3, kusionstack-operating-4]
$ kubectl -n kusionstack-system get lease
NAME HOLDER AGE
kusionstack-controller-manager---operating-0-canary kusionstack-operating-0_81b5bbae-be63-45ed-a939-e67e0c3d6326 12m
kusionstack-controller-manager---operating-1-normal kusionstack-operating-1_e4bbad49-e6ec-42fa-8ffd-caae82156a3e 12m
kusionstack-controller-manager---operating-2-normal kusionstack-operating-3_94f7f81a-f9e6-47d6-b72b-e16da479e9be 12m
Show the sample ShardingConfig:
$ helm template sample-operating kusionstack/operating \
--version v0.1.1 \
--set sharding.enabled=true \
--set sharding.isDemo=true \
--show-only templates/shardingconfig.yaml
Here is a sample ShardingConfig:
operating/templates/shardingconfig.yaml
apiVersion: ctrlmesh.kusionstack.io/v1alpha1
kind: ShardingConfig
metadata:
name: sharding-root
namespace: kusionstack-system
spec:
# Auto sharding config
root:
prefix: operating
targetStatefulSet: kusionstack-operating
canary:
replicas: 1
inNamespaces:
- kusionstack-system
auto:
everyShardReplicas: 2
shardingSize: 2
resourceSelector:
- relateResources:
- apiGroups:
- '*'
resources:
- configmaps
- pods
- endpoints
- services
- replicasets
- apiGroups:
- apps.kusionstack.io
resources:
- '*'
controller:
leaderElectionName: kusionstack-controller-manager
You can configure the ShardingConfig according to your requirements.
info
In order to enable the ShardingConfig, you also need to add the following label to the pod template.
ctrlmesh.kusionstack.io/watching: 'true'
ctrlmesh.kusionstack.io/enable-proxy: 'true'
We plan to deprecate it in future versions.