Kubernetes on Deepthot v2
Installing control plane
The current version of kubernetes is 1.16.3. However it has deleted a bunch of deprecated configuration options which unfortunately are being used by pretty much every helm chart I’ve run across. Since I’m just learning I’m going to stick with version 1.15.6.
Install the packages apt install {kubeadm,kubelet,kubectl}=1.15.6-00
Sadly this makes the configuration a bit more difficult. In 1.16.0 you can do everything with the command line. However we need to create a config file:
key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.6
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF
And build the cluster controller with kubeadm init --config=kubeadm-config.yaml --upload-certs
After it runs copy the config file where you want to use it from:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Finally set up the network. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" I chose weave because that’s what they mentioned in the document.
And that worked, came right up. As did the second control plane.
Worker nodes
Upgraded them to 4/6 cpus and 16G of core a piece.
apt full-upgrade {kubectl,kubelet,kubeadm}=1.15.6-00
kubeadm join blah blah printed out on the control plane when it comes up
Easy peasy. Looks beautiful
Older Version
Today kids we’re going to be trying to get kubernetes up on deepthot. I’ve been playing with microk8s at work, however gitlab seemed to be very unhappy with it. I don’t know if it’s microk8s that is the issue or not, but I’m going to try this the long way.
I’m working from https://kubernetes.io/docs/setup/ There’s a lot to learn. Anyway, at this point I’m ready to give it a go
First thing is installing kubeadm. Simple enough sudo snap install kubeadm Now before I even run it it mentions but first choose which network plugin you want to use. Joy, there’s like 8 or so of them. I’m gonna go with ‘flannel’ since it’s simple and mature. I mean we’re talking about a home network here yeah? Anyway, skipping ahead I see that flannel requires --pod-network-cidr=10.244.0.0/16 to be added to the kubeadm init command.
But but first, it wants me to run sudo kubeadm config images pull to make sure everything is working. Turns out it’s not, it wants docker. Because I work with it all the time, I think I’m going to go with docker’s docker-ce version instead of ubuntu’s.
I’ve not done this on ubuntu before, so I’m going to go with https://www.linode.com/docs/applications/containers/install-docker-ce-ubuntu-1804/ Looks really normal. So, here’s the commands:
sudo apt remove docker docker-engine docker.io
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
Ok, now the guide I’m following wants me to install docker-ce and then add my user to the docker group. I’m thinking I’m going to be doing a lot of this, and I’m lazy, so instead I think I want to create a docker group in LDAP and add myself to it. The intent here is so I don’t have to keep creating the docker group, it will be done once and have the same value on all of my machines. So, let me see how to do that
denebeim@deepthot.aa@k8s-control:~/kubernetes$ kinit denebeim
Password for denebeim@DEEPTHOT.AA:
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add docker
--------------------
Added group "docker"
--------------------
Group name: docker
GID: 1847800013
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add-member docker --groups=admins
Group name: docker
GID: 1847800013
Member groups: admins
Indirect Member users: admin, denebeim
-------------------------
Number of members added 1
-------------------------
OK, now going to install docker-ce and see what happens
denebeim@deepthot.aa@k8s-control:~/kubernetes$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ll /var/run/docker.sock
srw-rw---- 1 root docker@deepthot.aa 0 Oct 25 04:41 /var/run/docker.sock=
Boo… not working… Log out, log back in, working yay. I guess the group membership in core isn’t refreshed. I suspected as much. Anyway, we’re good.
Back to kubernetes now.
denebeim@deepthot.aa@k8s-control:~$ sudo kubeadm config images pull
cannot open path of the current working directory: Permission denied
denebeim@deepthot.aa@k8s-control:~$ sudo -i
root@k8s-control:~# kubeadm config images pull
Well… something to keep in mind. You gotta be in a root shell, not just sudoing a command to do some of this stuff.
root@k8s-control:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
[init] Using Kubernetes version: v1.16.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING FileExisting-socat]: socat not found in system path
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': executable file not found in $PATH
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
OK, not working. Looks like I need to read https://kubernetes.io/docs/setup/cri/ and kubelet hasn’t been installed yet. OK, there’s some commands in that web site, this is pretty much what I did:
# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common
### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
### Add Docker apt repository.
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu
# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
}
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart docker.
systemctl daemon-reload
systemctl restart docker
However, I’m using btrfs, so I took out this line "storage-driver": "overlay2" (which I also did in the paste above) use docker’s btrfs storage driver is a better idea.
This is having be downgrade docker. So I did. However, current docker is using containerd, I wonder why it’s not supported here. Just to be sure, I’m sticking with the old one for the time being.
Anyway, installing the snap for kubelet and trying again. Not working. I guess I was too smart for my own good. It’s insisting on the overlay2 storage driver… So…
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
systemctl daemon-reload
systemctl restart docker
I ended up having issues with btrfs subvolumes, I needed to delete them, which I did, but it would be easier if I never did it. So listen to the docs next time and don’t try to do the btrfs storage driver
Well, kubelet won’t start. It’s getting too late. Try again tomorrow. What I should do is:
- ipa-client-install --uninstall k8s-control
- delete k8s-control (bet I didn’t take a snapshot, so so after doing the base install tomorrow)
- poke around ipa and look for stuff to delete, there’s probably some dns stuff
- recreate k8s-control
- install docker-ce
- make sure to do it with the overlay config and systemd
- only now snap install kubeadm and kubelet
- try for another run of
kubeadm init
Second attempt
Well, maybe anyway, I’m going to try a couple more things. See if I can get away with not having to install from scratch. Poked around removed everything I could find that was kubernetes or docker related.
# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common -y
### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
### Add Docker apt repository.
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu -y && apt-mark hold docker-ce
# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart docker.
systemctl daemon-reload
systemctl restart docker
Something annoying: see above we installed a specific version of docker. That package version was never locked in so docker gets upgraded every time you do a apt upgrade. The way to fix that is apt-mark hold <package-name> I’ve edited the commands above.
I found this on the kubernetes setup production-environment container-runtimes I’m gonna go with this first.
Same issue, kubelet doesn’t want to start. I’ve been digging around and it appears that people have been installing the kubernetes xenial repositories (there’s no bionic ones) and using those to install kubelet and kubeadm. So, let’s try that
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
apt update
apt install kubectl kubelet kubeadm kubernetes-cni -y
I found this in a post “@civik it has been resolved, you need to add --cgroup-driver=system to kubelet start up parameters in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf”
So, let’s try that. It was actually in /var/lib/kubelet/kubeadm-flags.env. Editing it the way the post said didn’t help anything. I rebooted the server and now I’ve got kubernetes containers running. Dunno how healthy it is.
I used this command kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml to get the flannel cni (network) running. Seems to have worked. Nope flannel keeps restarting.
I’m going to try uninstalling kubernetes kubeadm reset, and installing it again. This is what I did:
kubectl drain k8s-control --delete-local-data --force --ignore-daemonsets
kubectl delete node k8s-control
kubeadm reset
Which seems to have worked because…
kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
--discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
--discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993
So it looks like I have a node up. Yay. What next? Flannel again, I suppose.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml Seems to be stable this time.
Setting up the dashboard
I’m following this: web-ui-dashboard and this creating sample user
So, at this point I can operate kubernetes from my desktop mostly. Here’s how to set it up
scp -r k8s-control:.kube .
sudo snap install kubectl --classic
# that's it, however if you want to get to the web interface stuff
kubectl proxy &
Installing the dashboard is similarly easy, install the dashboard, create an admin user, bind it to the cluster admin role, get the token.
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
cat <<EOF >dashboard-adminuser.yaml
> apiVersion: v1
> kind: ServiceAccount
> metadata:
> name: admin-user
> namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
denebeim@frankiemouse:~/kubernetes$ cat <<EOF >dashboard-adminbind.yaml
> apiVersion: rbac.authorization.k8s.io/v1
> kind: ClusterRoleBinding
> metadata:
> name: admin-user
> roleRef:
> apiGroup: rbac.authorization.k8s.io
> kind: ClusterRole
> name: cluster-admin
> subjects:
> - kind: ServiceAccount
> name: admin-user
> namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminbind.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')|awk '/token:/{print $2}'
That’s really all there is to it. I guess this makes us done.
This is better helm upgrade --install dashboard stable/kubernetes-dashboard --namespace dashboard --set enableSkipLogin=true --set ingress.enabled=true --set enableInsecureLogin=true but it’s running a way older version of the software
Written with StackEdit.
No comments:
Post a Comment