K8s on Samarthya

Adding Node to a cluster

Thu, 06 Jan 2022 06:58:31 +0000

In this blog, I will try add a new node to the already running cluster.

My Cluster
#

root@master>k get nodes
NAME STATUS ROLES AGE VERSION
mydevmachine001830.samarthya.me Ready <none> 22h v1.23.1
mydevmachine002687.samarthya.me Ready <none> 41d v1.23.1
mydevmachine003277.samarthya.me Ready <none> 131d v1.23.1
mydevmachine003278.samarthya.me Ready <none> 131d v1.23.1
mydevmachine003417.samarthya.me Ready <none> 4m55s v1.23.1
mydevmachine003968.samarthya.me Ready <none> 131d v1.23.1
mydevmachine003969.samarthya.me Ready control-plane,master 131d v1.23.1
mydevmachine003970.samarthya.me Ready <none> 131d v1.23.1
mydevmachine003979.samarthya.me Ready <none> 29m v1.23.1

Steps to follow
#

Install docker
Install kubeadm, kubectl & kubelet
Join the cluster using the token
Check the node has joined the cluster

Step 1: Install docker
#

sudo yum install -y yum-utils
sudo yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo

When adding repo make sure the repo has been added successfully to the machine

Upgrading K8S cluster

Wed, 05 Jan 2022 07:05:30 +0000

In this blog I will talk about how you can upgrade a node and the kubernetes cluster version to a specific version.

I have a 6 node local Kubernetes cluster like below

K8S RBAC

Mon, 13 Dec 2021 13:03:56 +0000

What?
#

Role Based Access Control(s) is a way of regulating access to entities like storage, network etc.
RBAC authorization uses rbac.authorization.k8s.io API group.

API Objects
#

There are 4 kind of kubernetes object

Secrets

Mon, 12 Apr 2021 09:35:03 +0000

To manage sensitive information you create secrets

k create secret --help

Create a secret using specified subcommand.

Available Commands:
 docker-registry Create a secret for use with a Docker registry
 generic Create a secret from a local file, directory or literal value
 tls Create a TLS secret

Usage:
 kubectl create secret [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).

type specifies the type of secret you are creating (options are as shown above). As I need to create only key value pairs for my dummy secret file I need to know a way of how to facilitate that, and the option -h helped me there

Pod & Volume: K8S

Fri, 22 Jan 2021 10:19:46 +0000

Everything in the same file

apiVersion: v1
kind: Pod
metadata:
 name: redispod
spec:
 containers:
 - image: redis
 name: redisdb
 volumeMounts:
 - name: redis-data
 mountPath: /data
 ports:
 - containerPort: 6379
 protocol: TCP
 volumes:
 - name: redis-data
 persistentVolumeClaim:
 claimName: redisdb-pvc
---
apiVersion: v1
kind: PersistentVolume
metadata:
 name: redis-pv
spec:
 storageClassName: ""
 capacity:
 storage: 1Gi
 accessModes:
 - ReadWriteOnce
 hostPath:
 path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: redisdb-pvc
spec:
 storageClassName: ""
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

If you deploy you will get something like

Taint Nodes

Tue, 12 Jan 2021 08:57:23 +0000

Reference
#

Before we understand the tainting we should know about the Node Affinity. It the by the virtue of Affinity that pods attracts nodes as a preference or as a basic requirement.

Quick: Cheat Sheet

Fri, 08 Jan 2021 11:04:34 +0000

A cheat sheet for Installing K8S cluster.

Step1:
#

Check your OS?

cat /etc/issue
lsb_release -a

Step 2:
#

Add the GPG key as described here

K8S: Deployment

Tue, 22 Dec 2020 12:45:09 +0000

Time to re-test how to create deployments. It has been sometime, I created some deployments so just checking on how much I remember.

The quick way to always get started is to check the HELP.

Installing K8S on Ubuntu

Mon, 14 Dec 2020 00:00:00 +0000

Install docker

cat /etc/issue
Ubuntu 18.04.4 LTS \n \l

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

sudo apt-get update

sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu

sudo apt-mark hold docker-ce

apt-mark --help
apt 1.6.12ubuntu0.2 (amd64)
Usage: apt-mark [options] {auto|manual} pkg1 [pkg2 ...]

apt-mark is a simple command line interface for marking packages
as manually or automatically installed. It can also be used to
manipulate the dpkg(1) selection states of packages, and to list
all packages with or without a certain marking.

lsb_release -c
Codename: bionic

lsb_release -cs
bionic

systemctl status docker
● docker.service - Docker Application Container Engine
 Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
 Active: active (running) since Mon 2020-12-14 09:05:39 UTC; 45s ago
 Docs: https://docs.docker.com
 Main PID: 662 (dockerd)
 Tasks: 21
 CGroup: /system.slice/docker.service
 ├─662 /usr/bin/dockerd -H fd://
 └─685 docker-containerd --config /var/run/docker/containerd/containerd.toml

Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.297237656Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.297281586Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7df0, CONNECTING" module=grpc
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.297797167Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7df0, READY" module=grpc
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.297852837Z" level=info msg="Loading containers: start."
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.596242360Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP addres
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.753685539Z" level=info msg="Loading containers: done."
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.822291121Z" level=info msg="Docker daemon" commit=e68fc7a graphdriver(s)=overlay2 version=18.06.1-ce
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.822430053Z" level=info msg="Daemon has completed initialization"
Dec 14 09:05:39 ip-10-0-1-101 dockerd[662]: time="2020-12-14T09:05:39.853660664Z" level=info msg="API listen on /var/run/docker.sock"
Dec 14 09:05:39 ip-10-0-1-101 systemd[1]: Started Docker Application Container Engine.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update
sudo apt-get install -y kubelet=1.14.5-00 kubeadm=1.14.5-00 kubectl=1.14.5-00
sudo apt-mark hold kubelet kubeadm kubectl

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

I1214 09:10:37.236647 3055 version.go:240] remote version is much newer: v1.20.0; falling back to: stable-1.14
[init] Using Kubernetes version: v1.14.10
[preflight] Running pre-flight checks
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ip-10-0-1-101 localhost] and IPs [10.0.1.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ip-10-0-1-101 localhost] and IPs [10.0.1.101 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ip-10-0-1-101 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.1.101]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.001809 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node ip-10-0-1-101 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ip-10-0-1-101 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: xtpkvj.e81gaeg0v18shctg
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
 https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.1.101:6443 --token xtpkvj.e81gaeg0v18shctg \
 --discovery-token-ca-cert-hash sha256:6c79fda4db86da060f083b35def6c39790c47f6533ab8f7ebb6ea30c2c2af6a3

Metric Server

Wed, 09 Dec 2020 08:01:49 +0000

For my local K8S cluster of 4 nodes I wanted to get the resource usage metrics. From the official documentation of kubernetes.io

These metrics can be accessed either directly by the user with the kubectl top command, or by a controller in the cluster, for example Horizontal Pod Autoscaler, to make decisions.

https://kubernetes.io

Writing Helm Charts

Mon, 14 Sep 2020 18:07:54 +0000

Helm is a dedicated package manager for Kubernetes.

Although the Kubernetes objects are deployed in YAML’s there was a need to package and club them together and customise the deployment (not the K8S deployment).

Helm Charts: Packaging

Wed, 02 Sep 2020 09:12:45 +0000

Every software cycle includes packaging; which essentially describes how the software written is packaged and shared.

For the software built for Containers Helm is a popular choice of packaging the scripts.

MultiContainer: K8S

Thu, 27 Aug 2020 11:18:17 +0000

POD
#

Smallest deployment unit you can manage in K8S, is a Pod. A pod may have one or more containers.

Helpful links for K8S

Wed, 26 Aug 2020 09:52:02 +0000

Always remember to refer the API and elements for the YAML structure

API Overview — official K8S documentation

The Kuberenetes documentation that has text descriptions and good explanatory examples.

Deployments: K8S

Mon, 24 Aug 2020 12:29:53 +0000

A declarative way of maintaining the desired state.

Continuing my example from the last blog

--- # Pod deployment
apiVersion: apps/v1
kind: Deployment
metadata:
 name: go-server-deployment
spec:
 replicas: 2
 selector:
 matchLabels:
 app: my-go-server
 template:
 metadata:
 labels:
 app: my-go-server
 spec:
 containers:
 - name: go-server
 image: samarthya/epserver:2.0
 resources:
 limits:
 memory: "256Mi"
 cpu: "1024m"
 env:
 - name: SERVERPORT
 value: "9090"
 ports:
 - containerPort: 9090
# Reference - https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

In my deployment, I have two replicas configured, which can be increased to three using the command like the one below.

NetworkPolicy: K8S

Mon, 24 Aug 2020 00:00:00 +0000

How can a pod communicate with the other pod? The answer to this question is through a network policy.

Network Policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints.

Probes: K8S

Sat, 22 Aug 2020 10:53:05 +0000

In this blog we will be talking about Liveness, Readiness and Startup probes of a container.

A Service in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them. — - Life without a service

What is Kubelet?
#

Kubelet is a primary node agent that runs on every node.
Works in term of PodSpec, that describes a POD (JSON or Yaml).

Liveness Probe
#

It is used by kubelet to decide when to restart the container.
periodSeconds defines when the kubelet check periodically.
initialDelaySeconds specifies the initial wait before checking.

I just created a single container pod that will run touch command to create a file and then it deletes that specific file after a delay, on which the livenessProbe is listening on and then once liveness fails we can check the container.

Services: K8S

Fri, 21 Aug 2020 17:34:29 +0000

There are multiple Kubernetes objects we will be talking about in this blog. Mainly Services, Selectors and Labels.

Service is an abstraction which defines a logical set of Pods and a policy by which to access them.
A Service in Kubernetes is a REST object, similar to a Pod.
Service tags the pods, logically grouping them using selectors.
Pods have a lifecycle, they are born, they live, process and then they die. The pods (desired state) can be managed using deployment.
A label is an identifying attribute of an object, and can be defined at the creation time, and subsequently modified and added at any time.
Label selector is a grouping primitive in Kubernetes and allows client/user to identify a set of objects. There are two sets of selectors
- equality-based and
- set-based.

An abstract way to expose an application running on a set of Pods as a network service.

The set of Pods targeted by a Service is usually determined by a selector.

ConfigMaps: K8S - 2

Fri, 21 Aug 2020 12:24:01 +0000

We looked at using the environment variables using the configmap in the last blog.

The config map remains the same but now in YAML.

apiVersion: v1
kind: ConfigMap
metadata:
 name: happy-faces-mnt
data:
 data.values: |-
 my.name=samarthya
 my.profession=engineer

I will use this to mount it to a container and then see the contents of the file.

K8S

Mon, 17 Aug 2020 07:56:44 +0000

What is K8S?
#

To manage distributed containerised workloads, you use Kubernetes.

It is resilient declarative configuration.
It is open source platform.
It allows you to bundle and run your application.

From the official Kubernetes documentation.

Kubernetes objects
#

K8S objects are the persistent entities in the K8S system and thee entities are used to represent the state of the cluster.

PV, PVC

Wed, 15 Jul 2020 00:00:00 +0000

PV and PVC: PersistentVolume and PersistentVolumeClaim
#

PV - Defines the structure for provisioning storage for use. They have a defined lifecycle, independent of any individual Pod that uses the PV.

Kubectl - A revisit

Tue, 07 Apr 2020 16:58:30 +0000

It is always fun learning and trying out some new tech. In this blog, I will try and capture my experience of running my first pod using kubectl.

If you wish to learn how to install a typical master, worker cluster of K8S I recommend going through this blog.

KubeLab

Thu, 10 Oct 2019 00:00:00 +0000

Kubernetes

cloud_user@ip-10-0-1-101:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.11", GitCommit:"25074a190ef2a07d8b0ed38734f2cb373edfb868", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

cloud_user@ip-10-0-1-101:~$ kubectl api-resources
NAME SHORTNAMES APIGROUP NAMESPACED KIND
bindings true Binding
componentstatuses cs false ComponentStatus
configmaps cm true ConfigMap
endpoints ep true Endpoints
events ev true Event
limitranges limits true LimitRange
namespaces ns false Namespace
nodes no false Node
persistentvolumeclaims pvc true PersistentVolumeClaim
persistentvolumes pv false PersistentVolume
pods po true Pod
podtemplates true PodTemplate
replicationcontrollers rc true ReplicationController
resourcequotas quota true ResourceQuota
secrets true Secret
serviceaccounts sa true ServiceAccount
services svc true Service
mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration
customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition
apiservices apiregistration.k8s.io false APIService
controllerrevisions apps true ControllerRevision
daemonsets ds apps true DaemonSet
deployments deploy apps true Deployment
replicasets rs apps true ReplicaSet
statefulsets sts apps true StatefulSet
tokenreviews authentication.k8s.io false TokenReview
localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview
selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview
selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview
subjectaccessreviews authorization.k8s.io false SubjectAccessReview
horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler
cronjobs cj batch true CronJob
jobs batch true Job
certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest
leases coordination.k8s.io true Lease
events ev events.k8s.io true Event
daemonsets ds extensions true DaemonSet
deployments deploy extensions true Deployment
ingresses ing extensions true Ingress
networkpolicies netpol extensions true NetworkPolicy
podsecuritypolicies psp extensions false PodSecurityPolicy
replicasets rs extensions true ReplicaSet
networkpolicies netpol networking.k8s.io true NetworkPolicy
poddisruptionbudgets pdb policy true PodDisruptionBudget
podsecuritypolicies psp policy false PodSecurityPolicy
clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding
clusterroles rbac.authorization.k8s.io false ClusterRole
rolebindings rbac.authorization.k8s.io true RoleBinding
roles rbac.authorization.k8s.io true Role
priorityclasses pc scheduling.k8s.io false PriorityClass
storageclasses sc storage.k8s.io false StorageClass
volumeattachments storage.k8s.io false VolumeAttachment

cloud_user@ip-10-0-1-101:~$ kubectl get cm
NAME DATA AGE
fruit-service-ambassador-config 1 5m49s

Basic Commands (Beginner):
 create Create a resource from a file or from stdin.
 expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
 run Run a particular image on the cluster
 set Set specific features on objects

Basic Commands (Intermediate):
 explain Documentation of resources
 get Display one or many resources
 edit Edit a resource on the server
 delete Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
 rollout Manage the rollout of a resource
 scale Set a new size for a Deployment, ReplicaSet, Replication Controller, or Job
 autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
 certificate Modify certificate resources.
 cluster-info Display cluster info
 top Display Resource (CPU/Memory/Storage) usage.
 cordon Mark node as unschedulable
 uncordon Mark node as schedulable
 drain Drain node in preparation for maintenance
 taint Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
 describe Show details of a specific resource or group of resources
 logs Print the logs for a container in a pod
 attach Attach to a running container
 exec Execute a command in a container
 port-forward Forward one or more local ports to a pod
 proxy Run a proxy to the Kubernetes API server
 cp Copy files and directories to and from containers.
 auth Inspect authorization

Advanced Commands:
 diff Diff live version against would-be applied version
 apply Apply a configuration to a resource by filename or stdin
 patch Update field(s) of a resource using strategic merge patch
 replace Replace a resource by filename or stdin
 wait Experimental: Wait for a specific condition on one or many resources.
 convert Convert config files between different API versions

Settings Commands:
 label Update the labels on a resource
 annotate Update the annotations on a resource
 completion Output shell completion code for the specified shell (bash or zsh)

Other Commands:
 api-resources Print the supported API resources on the server
 api-versions Print the supported API versions on the server, in the form of "group/version"
 config Modify kubeconfig files
 plugin Provides utilities for interacting with plugins.
 version Print the client and server version information

Usage:
 kubectl [flags] [options]