Kubernetes
Contents |
Learn
Online learn deploy https://kubernetes.io/docs/tutorials/kubernetes-basics/ (it uses https://www.katacoda.com/ ) Another online playground https://labs.play-with-k8s.com/
Install Kubernetes locally
Kind, k3d, minikube or microk8s
Kind
https://kind.sigs.k8s.io/
K3d
https://k3d.io/
Example 3 nodes
k3d cluster create gaia --servers 1 --agents 2 --port 80:80@loadbalancer --port 443:443@loadbalancer --api-port 6443 --k3s-arg '--no-deploy=traefik@server:*'
Minikube
Install minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
minicube itself is running inside docker
minikube start
minikube ip
# to stop
docker ps
minikube stop
Microk8s
For ubuntu you can use https://microk8s.io/docs
sudo snap install microk8s --classic --channel=1.21
sudo usermod -a -G microk8s $USER
newgrp microk8s
microk8s status --wait-ready
# microk8s is running
# to stop
microk8s stop
You can use alias
alias k=microk8s.kubectl
or you can copy config
microk8s config > ~/.kube/config
It is running as /snap/microk8s/2694/kubelite
proccess.
To stop run
microk8s stop
Kubectl
Install kubectl https://kubernetes.io/docs/tasks/tools/
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
Check if kubectl can access
export KUBECONFIG=~/.kube/kubernetes-rails-example-kubeconfig.yaml
kubectl version
# this will create Deployment, ReplicaSet and Pod
kubectl run my-nginx --image nginx
kubectl create deployment httpenv --image=bretfisher/httpenv
kubectl delete pod/httpenv-6fdc8554fb-f568c
kubectl scale deployment/httpenv --replicas 5
# create ClusterIP
kubectl expose deployment/httpenv --port 8888
kubectl get nodes -o wide # find node ip address
kubectl get pods -o wide # pods are running on nodes, always replaced
kubectl get pods -w # watch
kubectl get pod my-pod -o jsonpath="{.spec.containers[0].ports[0].containerPort}"
kubectl get services
kubectl get svc hello
kubectl get events
kubectl get deployments
kubectl get all
kubectl describe service my-load-balancer
kubectl describe pods
kubectl config view
kubectl logs -l app=rails-app -f # print logs from rails-app pods
kubectl logs my-pod -p # log for specific pod of previous crashed CrashLoopBackOff instance
kubectl apply -f config/kube/terminal.yml
# live edit kubernetes
kubectl edit service/my-service
kubectl edit replicaset/app-nginx-deployment-56845f96cc
kubectl exec terminal -- date # run command in first container inside pod
kubectl exec terminal -it -- bash # run bash inside first container
kubectl exec terminal -c ruby-container -it bash # run ssh inside specific con
kubectl exec terminal -it -- bundle exec rails console
# to trigger refresh deploy you can simply remove
kubectl delete pods -l app=rails-app
kubectl create -f config/kube/migrate.yml
kubectl wait --for=condition=complete --timeout=600s job/migrate
kubectl delete job migrate
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=application-1,app.kubernetes.io/instance=chart-1" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
# $POD_NAME is chart-1-application-1-6bcd5f68bd-kjmqr $CONTAINER_PORT is 80
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
To list all kinds that we can create
kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
nodes no v1 false Node
deployments deploy apps/v1 true Deployment
To list all versions that are supported
kubectl api-versions
Find docs for specific API object
kubectl explain services
kubectl explain services --recursive
kubectl explain services.spec.type
# note that this is a documentation of the client (not from the server, so maybe
server does not support that version)
Full references is on https://kubernetes.io/docs/reference/
To see what generated configuration is used from cli arguments
kubectl create deployment sample --image nginx --dry-run -o yaml > deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: sample
name: sample
spec:
replicas: 1
selector:
matchLabels:
app: sample
...
kubectl expose deployment/test --port 80 --dry-run -o yaml
You can see changes on server that differs from yaml
kubectl diff -f app.yml
Declarative is more what
needs to be created, but impperative is more how
.
Declerative is using apply command and we do not care how it will be performed
to the desired state.
Docs
Learn Kubernetes with videos https://www.youtube.com/hashtag/kubernetesessentialsfromgooglecloud and documentation for example Specs for pod https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
Control plane is a set of containers that manage the cluster, includes: API server, scheduler, controller manager, etcd, CoreDNS On nodes we have: kubelet (agent running on nodes) and kube-proxy.
Yaml
Four root keys are: apiVersion
, kind
, metadata
and spec
.
Inside metadata
you can use labels
(for example: app: api
, env: prod
)
and for large objects use annotations
Pods
Pod is one or more containers running together on one Node. Pod is basic unit of deployment. Containers are always in pods. Controller is used for creating and updating pods, many types of controlles: Deployment, ReplicaSet, DaemonSet, Job
Pod can be in Pending Running Succeded Failed Unknown phase https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase Container state (old has Pending state - insufficient resource)
- Waiting - Pod can’t run (there are enough resource but maybe image can not be downloaded yet)
- Running - you can debug using exec or logs
- Terminated - either ran to completion or failed for some reason
Use probe to check container status https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
Example
# pod.yml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.17.3
ports:
- containerPort: 80
but that does not create replica set so better is to create Deployment
# config/kube/deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubernetes-rails-example-deployment
spec:
replicas: 3
selector:
matchLabels:
app: rails-app
template:
metadata:
labels:
app: rails-app
spec:
containers:
- name: rails-app
image: duleorlovic/kubernetes-rails-example:latest
ports:
- containerPort: 3000
Services
Services route traffic to pods matching app: MyApp
and redirects traffic from
80 to 9376 only on internal cluster ip address.
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: ClusterIP
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
https://www.youtube.com/watch?v=uGm_A9qRCsk
https://kubernetes.io/docs/concepts/services-networking/service/
https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
Service types:
ClusterIP
(default) is reachable only inside cluster.
NodePort
is opening a specific (high) port on a specific machine
LoadBalancer
is used for using cloud-provided load balancer to route traffic
between services pods but since you pay for every exposed service it is cheaper
to use Ingress
which can run on one port and route based on domain or path
ExternalName
adds CNAME DNS record to CoreDNS
# create ClusterIP
kubectl expose deployment/httpenv --port 8888
# create terminal
kubectl run tmp-shell --rm -it --image bretfisher/netshoot -- bash
curl httpenv:8888
# find ip from kubectl get services
curl 10.152.183.129:8888
# create NodePort
kubectl expose deployment/httpenv --port 8888 --name httpenv-np --type NodePort
kubectl get services # this returns ports 8888:31471
# from host
curl localhost:31471
curl 10.152.183.129:8888
curl <hostname>.<namespace>.svc.cluster.local
3rd party proxy: Traefik, HAProxy, Nginx
Namespaces
Namespaces is used to filter group of object in cluster https://www.youtube.com/watch?v=plB3kyZLHe8 Debug services and dns with exec https://www.youtube.com/watch?v=CSKRy7Ldqis
Namespace Authentication and cluster is defined inside ~/.kube/config
You can see all clusters with
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* microk8s microk8s-cluster admin
ConfigMap
kubectl create configmap nginx-frontend.conf --from-file nginx/frontend.conf
You can create using yml
# config/kube/env.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: env
data:
RAILS_ENV: production
RAILS_LOG_TO_STDOUT: enabled
RAILS_SERVE_STATIC_FILES: enabled
DATABASE_URL: postgresql://example.com/mydb
REDIS_URL: redis://redis.default.svc.cluster.local:6379/
and use in your container
envFrom:
- configMapRef:
name: env
You can create from text
kubectl create secret generic rails-secrets --from-literal=rails_master_key='example'
and use in container
env:
- name: RAILS_MASTER_KEY
valueFrom:
secretKeyRef:
name: rails-secrets
key: rails_master_key
Secrets
kubectl create secret generic tls-certs --from-file tls/
StatefulSets
Use db-as-a-service whenever you can.
Volumes
tied to lifecycle of a Pod, all containers in a single Pod can share
them.
PersistentVolumes
created at cluster level, outlives a Pod, multiple Pods can
share them.
Templating YAML
https://kustomize.io/ is template free, using overrides
docker app
and compose-on-kubernetes https://kompose.io/
Helm
It is similar to liquid shopify template language.
sudo snap install helm --classic
microk8s.enable helm3
Example
heml install chart-1 .
heml list
helm uninstall chart-1
You can see generated template or apply them
helm template .
helm template . | kubectl apply -f -
https://helm.sh/docs/chart_template_guide/getting_started/ Template
It is using sprig library https://helm.sh/docs/howto/charts_tips_and_tricks/
{ {/*
and*/}}
comment{ {- define "labels" -}}
and use with{ { include "application-1" . | nindent 4 }}
{ {- if eq .Values.container1.enabled true -}}
or short{ {- if .Values.container1.enabled -}}
flow control https://helm.sh/docs/chart_template_guide/control_structures/
Chart repository https://helm.sh/docs/topics/chart_repository/ you can run your own chartmuseum https://chartmuseum.com/docs/#helm-chart For grafana we use
helm repo add grafana https://grafana.github.io/helm-charts
helm search repo grafana
helm install my-grafana grafana/grafana
# if you want to customize
helm pull grafana/grafana
tar -xf grafana-6.20.3.tgz
# remove chart
helm uninstall my-grafana
Dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml
kubectl proxy
Starting to serve on 127.0.0.1:8001
Context
https://blog.mikesir87.io/2019/08/using-ssh-connections-in-docker-contexts/ You can access docker using ssh (no need to expose docker port, only ssh with keys authentication is enough)
# create
docker context create ssh-box --docker "host=ssh://user@my-box"
# Set the context for a single command
docker --context=ssh-box ps
# OR set the context globally
docker context use ssh-box
docker ps
# OR use the DOCKER_CONTEXT env var
DOCKER_CONTEXT=ssh-box docker ps
Security
https://docs.docker.com/engine/security/ https://github.com/BretFisher/ama/issues/17
- namespace: access to own cpu, ram… (by default they can use all, so you need to limit) and view only its own filesystem (not from other containers)
- use non root user
USER node
andCOPY --chown=node
Ruby
https://kubernetes.io/docs/reference/using-api/client-libraries/#community-maintained-client-libraries
Rails
Deploy rails to kubernetes https://kubernetes-rails.com/
# Dockerfile
FROM ruby:2.6.3
# update yarn repo
RUN curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list
# update node
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash -
RUN apt-get update && apt-get install -y nodejs yarn postgresql-client
RUN mkdir /app
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN gem install bundler
RUN bundle install
COPY . ./
RUN bundle exec rake assets:precompile
EXPOSE 3000
CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
Create, test and push image to duleorlovic/kubernetes-rails-example:latest
docker build -t duleorlovic/kubernetes-rails-example .
docker run -p 3000:3000 duleorlovic/kubernetes-rails-example
docker push duleorlovic/kubernetes-rails-example
Enable kubernetes to download docker from private repo
kubectl create secret docker-registry my-docker-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
kubectl edit serviceaccounts default
TODO http://blog.jedrychowski.org/2018/rails-kubernetes-basic-deployment/ rancher https://www.youtube.com/watch?v=LK6KbAlQRIg kubernetes https://www.youtube.com/watch?v=X48VuDVv0do
https://evilmartians.com/chronicles/kubing-rails-stressless-kubernetes-deployments-with-kuby
Digital ocean
Digital ocean install
sudo snap install doctl
doctl auth init
doctl kubernetes cluster kubeconfig save 901f6f80-2728-4ac7-bc5e-77365d839e39
Check if we can access
export KUBECONFIG=~/.kube/kubernetes-rails-example-kubeconfig.yaml
kubectl version
kubectl get nodes
Setup Kubernetes using Vagrant Virtuarbox https://medium.com/swlh/setup-own-kubernetes-cluster-via-virtualbox-99a82605bfcc https://github.com/mbaykara/k8s-cluster
Vim
Using coc-yaml plugin for coc https://github.com/neoclide/coc-yaml so we have code completion and documentation similar to VSCode https://octetz.com/docs/2020/2020-01-06-vim-k8s-yaml-support/ https://www.youtube.com/watch?v=eSAzGx34gUE
Install coc https://github.com/neoclide/coc.nvim/wiki/Install-coc.nvim using native package manager vim8
mkdir -p ~/.vim/pack/coc/start
cd ~/.vim/pack/coc/start
git clone --branch release https://github.com/neoclide/coc.nvim.git --depth=1
inside vim run
:CocInfo
:CocInstall coc-yaml
It is installing https://github.com/redhat-developer/vscode-yaml so run
:CocConfig
to edit ~/.vim/coc-settings.json
{
"yaml.schemas": {
"kubernetes": ["/*.yaml"]
}
}
Copy paste keys https://github.com/neoclide/coc.nvim#example-vim-configuration
Control-space or tab is used to code complete. Use <c-p>
and <c-n>
to go to
next or prev item, or just type letters.
Sidecar pattern
Book Designed Distributed Systems” by Brendan Burns https://azure.microsoft.com/en-us/resources/designing-distributed-systems/