Wednesday, November 20, 2019

Kubernetes on Deepthot

Kubernetes on Deepthot v2

Installing control plane

The current version of kubernetes is 1.16.3. However it has deleted a bunch of deprecated configuration options which unfortunately are being used by pretty much every helm chart I’ve run across. Since I’m just learning I’m going to stick with version 1.15.6.

Install the packages apt install {kubeadm,kubelet,kubectl}=1.15.6-00

Sadly this makes the configuration a bit more difficult. In 1.16.0 you can do everything with the command line. However we need to create a config file:

key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.6
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF

And build the cluster controller with kubeadm init --config=kubeadm-config.yaml --upload-certs
After it runs copy the config file where you want to use it from:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Finally set up the network. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" I chose weave because that’s what they mentioned in the document.

And that worked, came right up. As did the second control plane.

Worker nodes

Upgraded them to 4/6 cpus and 16G of core a piece.

apt full-upgrade  {kubectl,kubelet,kubeadm}=1.15.6-00
kubeadm join blah blah printed out on the control plane when it comes up

Easy peasy. Looks beautiful

Older Version

Today kids we’re going to be trying to get kubernetes up on deepthot. I’ve been playing with microk8s at work, however gitlab seemed to be very unhappy with it. I don’t know if it’s microk8s that is the issue or not, but I’m going to try this the long way.

I’m working from https://kubernetes.io/docs/setup/ There’s a lot to learn. Anyway, at this point I’m ready to give it a go

First thing is installing kubeadm. Simple enough sudo snap install kubeadm Now before I even run it it mentions but first choose which network plugin you want to use. Joy, there’s like 8 or so of them. I’m gonna go with ‘flannel’ since it’s simple and mature. I mean we’re talking about a home network here yeah? Anyway, skipping ahead I see that flannel requires --pod-network-cidr=10.244.0.0/16 to be added to the kubeadm init command.

But but first, it wants me to run sudo kubeadm config images pull to make sure everything is working. Turns out it’s not, it wants docker. Because I work with it all the time, I think I’m going to go with docker’s docker-ce version instead of ubuntu’s.

I’ve not done this on ubuntu before, so I’m going to go with https://www.linode.com/docs/applications/containers/install-docker-ce-ubuntu-1804/ Looks really normal. So, here’s the commands:

sudo apt remove docker docker-engine docker.io
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update

Ok, now the guide I’m following wants me to install docker-ce and then add my user to the docker group. I’m thinking I’m going to be doing a lot of this, and I’m lazy, so instead I think I want to create a docker group in LDAP and add myself to it. The intent here is so I don’t have to keep creating the docker group, it will be done once and have the same value on all of my machines. So, let me see how to do that

denebeim@deepthot.aa@k8s-control:~/kubernetes$ kinit denebeim
Password for denebeim@DEEPTHOT.AA: 
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add docker
--------------------
Added group "docker"
--------------------
  Group name: docker
  GID: 1847800013
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add-member docker  --groups=admins
  Group name: docker
  GID: 1847800013
  Member groups: admins
  Indirect Member users: admin, denebeim
-------------------------
Number of members added 1
-------------------------

OK, now going to install docker-ce and see what happens

denebeim@deepthot.aa@k8s-control:~/kubernetes$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ll /var/run/docker.sock 
srw-rw---- 1 root docker@deepthot.aa 0 Oct 25 04:41 /var/run/docker.sock=

Boo… not working… Log out, log back in, working yay. I guess the group membership in core isn’t refreshed. I suspected as much. Anyway, we’re good.

Back to kubernetes now.

denebeim@deepthot.aa@k8s-control:~$ sudo kubeadm config images pull
cannot open path of the current working directory: Permission denied
denebeim@deepthot.aa@k8s-control:~$ sudo -i
root@k8s-control:~# kubeadm config images pull

Well… something to keep in mind. You gotta be in a root shell, not just sudoing a command to do some of this stuff.

root@k8s-control:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
[init] Using Kubernetes version: v1.16.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING FileExisting-socat]: socat not found in system path
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': executable file not found in $PATH
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

OK, not working. Looks like I need to read https://kubernetes.io/docs/setup/cri/ and kubelet hasn’t been installed yet. OK, there’s some commands in that web site, this is pretty much what I did:

# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  } 
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

However, I’m using btrfs, so I took out this line "storage-driver": "overlay2" (which I also did in the paste above) use docker’s btrfs storage driver is a better idea.

This is having be downgrade docker. So I did. However, current docker is using containerd, I wonder why it’s not supported here. Just to be sure, I’m sticking with the old one for the time being.

Anyway, installing the snap for kubelet and trying again. Not working. I guess I was too smart for my own good. It’s insisting on the overlay2 storage driver… So…

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
systemctl daemon-reload
systemctl restart docker

I ended up having issues with btrfs subvolumes, I needed to delete them, which I did, but it would be easier if I never did it. So listen to the docs next time and don’t try to do the btrfs storage driver

Well, kubelet won’t start. It’s getting too late. Try again tomorrow. What I should do is:

  1. ipa-client-install --uninstall k8s-control
  2. delete k8s-control (bet I didn’t take a snapshot, so so after doing the base install tomorrow)
  3. poke around ipa and look for stuff to delete, there’s probably some dns stuff
  4. recreate k8s-control
  5. install docker-ce
  6. make sure to do it with the overlay config and systemd
  7. only now snap install kubeadm and kubelet
  8. try for another run of kubeadm init

Second attempt

Well, maybe anyway, I’m going to try a couple more things. See if I can get away with not having to install from scratch. Poked around removed everything I could find that was kubernetes or docker related.

# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common -y

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu -y && apt-mark hold docker-ce

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

Something annoying: see above we installed a specific version of docker. That package version was never locked in so docker gets upgraded every time you do a apt upgrade. The way to fix that is apt-mark hold <package-name> I’ve edited the commands above.

I found this on the kubernetes setup production-environment container-runtimes I’m gonna go with this first.

Same issue, kubelet doesn’t want to start. I’ve been digging around and it appears that people have been installing the kubernetes xenial repositories (there’s no bionic ones) and using those to install kubelet and kubeadm. So, let’s try that

cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
apt update
apt install kubectl kubelet kubeadm kubernetes-cni -y

I found this in a post “@civik it has been resolved, you need to add --cgroup-driver=system to kubelet start up parameters in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf”

So, let’s try that. It was actually in /var/lib/kubelet/kubeadm-flags.env. Editing it the way the post said didn’t help anything. I rebooted the server and now I’ve got kubernetes containers running. Dunno how healthy it is.

I used this command kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml to get the flannel cni (network) running. Seems to have worked. Nope flannel keeps restarting.

I’m going to try uninstalling kubernetes kubeadm reset, and installing it again. This is what I did:

kubectl drain k8s-control --delete-local-data --force --ignore-daemonsets
kubectl delete node k8s-control
kubeadm reset

Which seems to have worked because…

kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
    --discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993 \
    --control-plane 	  

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
    --discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993 

So it looks like I have a node up. Yay. What next? Flannel again, I suppose.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml Seems to be stable this time.

Setting up the dashboard

I’m following this: web-ui-dashboard and this creating sample user

So, at this point I can operate kubernetes from my desktop mostly. Here’s how to set it up

scp -r k8s-control:.kube .
sudo snap install kubectl --classic
# that's it, however if you want to get to the web interface stuff
kubectl proxy &

Installing the dashboard is similarly easy, install the dashboard, create an admin user, bind it to the cluster admin role, get the token.

denebeim@frankiemouse:~/kubernetes$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
cat <<EOF >dashboard-adminuser.yaml
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: admin-user
>   namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
denebeim@frankiemouse:~/kubernetes$ cat <<EOF >dashboard-adminbind.yaml
> apiVersion: rbac.authorization.k8s.io/v1
> kind: ClusterRoleBinding
> metadata:
>   name: admin-user
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: cluster-admin
> subjects:
> - kind: ServiceAccount
>   name: admin-user
>   namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminbind.yaml 
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')|awk '/token:/{print $2}'

That’s really all there is to it. I guess this makes us done.

This is better helm upgrade --install dashboard stable/kubernetes-dashboard --namespace dashboard --set enableSkipLogin=true --set ingress.enabled=true --set enableInsecureLogin=true but it’s running a way older version of the software

Written with StackEdit.

No comments:

Post a Comment

How Simple Does It Have to Be Before It’s Mine?

 Hi, Joy here, I'm tired.  I'm too tired to polish this.  However bare with me, the text below is *trying* to be me.  It's not l...