Kubernetes Dashboard Helm Chart
I’ve written a half assed kubernetes dashboard helm chart.
Deploying the chart
I’ve written a half assed kubernetes dashboard helm chart.
Deploying the chart
I really want a good dashboard configuration at deepthot. I had it pretty nice at one point, but it’s crap now. Actually I haven’t installed it at all on the new version. However the issue is this, the helm chart version is pretty old. The pages are totally static, I want pages that auto update. With graphics. The graphics, I haven’t quite figured out yet, heapster is supposed to be all it takes to make it work, but I’ve not gotten it to work yet.
Googling around I found kubernetes-dashboard-v2.0.0 I’m going to try this.
Well, got it working, there’s some permissions issues at this point I need to iron out. Still don’t have the helm working. The permissions issues are an issue. I’ve got it working fine other than the graphics with a custom ‘recommended.yaml’ like from the kubernetes.io dashboard install page.
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.0.0-beta6
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
- --enable-skip-login=true
- --enable-insecure-login=true
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
"beta.kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.1
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
"beta.kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: kubernetes-dashboard
name: dashboard
namespace: kubernetes-dashboard
annotations:
cert-manager.io/cluster-issuer: ca-issuer
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.provider: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
tls:
- secretName: dashboard-tls
hosts:
- dashboard.deepthot.aa
- dashboard
rules:
- http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 443
path: /
host:
dashboard.deepthot.aa
- http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 443
path: /
host:
dashboard
Heck with this. I’ve got the recommended working, I’m going to make my own helm chart from it.
Note: Helm 2.16.0 is broken don’t use it. That was the snap as of 11/21/19 emphasized text
This is a total win. I’ve got the cert manager working great. Internal and external. This works with the helm chart as shipped to deploy/upgrade simply:
helm repo add gitlab https://charts.gitlab.io/
helm repo update
helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab
Here’s the current configure:
# values-minikube.yaml
# This example intended as baseline to use Minikube for the deployment of GitLab
# - Services that are not compatible with how Minikube runs are disabled
# - Configured to use 192.168.99.100, and nip.io for the domain
# Minimal settings
global:
ingress:
configureCertmanager: false
class: "nginx"
tls:
secretName: {}
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer : "ca-issuer"
hosts:
domain: deepthot.aa
externalIP: 192.168.42.64
https: false
gitlab:
name: gitlab.deepthot.aa
https: true
registry:
name: registry.deepthot.aa
https: true
minio:
name: minio.deepthot.aa
https: true
# Don't use certmanager, we'll use our own
certmanager:
install: false
nginx-ingress:
enabled: false
gitlab-runner:
install: false
certsSecretName: deepthot-aa-crt
gitlab:
unicorn:
ingress:
tls:
secretName: gitlab-gitlab-tls
registry:
ingress:
tls:
secretName: gitlab-registry-tls
minio:
ingress:
tls:
secretName: gitlab-minio-tls
Well, yet again I was making things too hard, ish. Where I am right now is I’ve set up a values.yaml file
# values-minikube.yaml
# This example intended as baseline to use Minikube for the deployment of GitLab
# - Services that are not compatible with how Minikube runs are disabled
# - Configured to use 192.168.99.100, and nip.io for the domain
# Minimal settings
global:
ingress:
configureCertmanager: false
class: "nginx"
hosts:
domain: deepthot.aa
externalIP: 192.168.42.64
https: false
gitlab:
name: gitlab.deepthot.aa
https: false
registry:
name: registry.deepthot.aa
https: false
minio:
name: minio.deepthot.aa
https: false
hosts:
# Don't use certmanager, we'll self-sign
certmanager:
install: false
# Use the `ingress` addon, not our Ingress (can't map 22/80/443)
nginx-ingress:
enabled: true
# GitLab Runner isn't a big fan of self-signed certificates
gitlab-runner:
install: false
Then I type helm upgrade --install -f values-deepthot.yaml --namespace gitlab gitlab . However I had to muck with the helm chart to work with the kubernetes I’m running here 1.16.2. The thing is they deleted several deprecated things and all of the helm charts I’ve played with have not been updated. I’m going to downgrade kubernetes and then get back to here.
Right now it is configured for a self-signed cert. That makes gitlab runner not function. So I need to generate a cert and get the deepthot.aa cert into it. I’ll do that after kubernetes downgrading though.
Downgrading was really easy. Just follow kubeadm-upgrade/ I downgraded the worker first, dunno if it was necessary, but all went clean. Now to try the helmchart from the repo again.
I think it’s working as well as it ever did. Now though it needs to have a real cert.
Wildcard_certificates that’s very problematical. OK. New plan, move my lan onto a delegated subdomain of deepthot.org. That’s a big production. I was planning on doing this anyway, but for the time being I really don’t need all that many certs. So, let’s do them by hand.
#!/bin/bash
dn=$(hostname -d)
me=$(hostname -f)
ns="gitlab"
ip="192.168.42.64"
kubectl create namespace "${ns}"
for host in gitlab registry minio; do
ipa dnsrecord-add "${dn}" $host --a-rec "${ip}"
ipa host-add $host."${dn}"
ipa service-add HTTP/$host."${dn}"
ipa service-add-host --hosts="${me}" HTTP/$host."${dn}"
sudo ipa-getcert request -r -f $host."${dn}".crt -k $host."${dn}".key -N CN=$host."${dn}" -D $host."${dn}" -K HTTP/$host."${dn}" -w
sudo chown denebeim: $host."${dn}".crt $host."${dn}".key
kubectl create secret tls gitlab-$host-tls --cert=./$host."${dn}".crt --key=./$host."${dn}".key --namespace="${ns}"
cat /etc/ipa/ca.crt >>"${host}"."${dn}".crt
done
kubectl --namespace ${ns} create secret generic deepthot-aa-crt --from-file={,{gitlab,minio,registry}.}deepthot.aa.crt
helm upgrade --install -f values.yaml --namespace "${ns}" "${ns}" gitlab/gitlab
Next step is to get gitlab runner up. It is not liking the custom CA that we have here.
gitlab-runner/issues/3968 describes the problem. There were a few workarounds, I tried figuring out the root cause, then I tried the solutions. The documentation says I should be concatenating the chain of certs.
I did this, replaced all the certs, didn’t help. Finally I looked at the stored cert in kubernetes. It was encrypted, but you know, it looked like the length was about the length of the key before it was concatenated. Also, I noticed the logs that came out when it was sucking the key in was saying ‘Updating local copy of SSL certificate “gitlab/gitlab-gitlab-tls” with missing intermediate CA certs’ Interesting. So I reread the bug report and one of the comments mentioned working around the problem by creating a generic secret. As you can see above the secrets we were building were tls secrets. So, I tried making a generic secret from the gitlab one we used to make the tls secret earlier. That worked. This is what you do:
cat /etc/ipa/ca.crt >>gitlab.deepthot.aa.crt
kubectl \
--namespace gitlab \
create secret generic gitlab-deepthot-aa-crt \
--from-file=gitlab.deepthot.aa.crt
helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab \
--set gitlab-runner.install=true \
--set gitlab-runner.certsSecretName=gitlab-deepthot-aa-crt
The first script has been modified to include concatenating /etc/ipa/ca.crt onto all of the certs that it is making. However
there’s still issues. Basically reading on the internet someone mentioned how problematical it is to use your own CA. Every container on the planet has it’s own copy of SSL stuff. You can’t put your CA into all of them. I ran into the same sort of trouble at work. There I just rebuilt everything and put our CA into the base container. The solution is to get us on a public DNS we control and get let’s encrypt working. But that’s a pretty big thing, and I guess all this stuff can wait until then.
I guess that means I’m done with gitlab for the moment. I’ll call this a win.
This is a really short and easy one. I already know metalLB orks. I just want to change the configuration. Let’s give it a real cidr.
cat <<EOF >values.yaml
configInline:
address-pools:
- name: deepthot.int
protocol: layer2
addresses:
- 192.168.42.64/26
EOF
helm upgrade --install --namespace metallb metallb stable/metallb -f values.yaml
Now, this is going to overlap with dhcpd so that needs to be updated as well.
range 192.168.42.128 192.168.42.254;
The range needed to be changed from 100 to 128
Written with StackEdit.
Well, this is odd, I’ve not run into this before. When I was trying to reinstall cert-manager I got
helm install \
> --name cert-manager \
> --namespace cert-manager \
> --version v0.11.0 \
> jetstack/cert-manager
Error: release cert-manager failed: namespaces "cert-manager" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "cert-manager"
Googling I found Helm Example Service account with cluster admin role Which shows:
cat <<EOF >rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
EOF
kubectl create -f rbac-config.yaml
helm init --service-account tiller
I’ll get this in a gitlab repo once I have gitlab up, but for now, it’ll just go in this doc, hence the cat stuff. Looks like that’s all that’s needed.
The current version of kubernetes is 1.16.3. However it has deleted a bunch of deprecated configuration options which unfortunately are being used by pretty much every helm chart I’ve run across. Since I’m just learning I’m going to stick with version 1.15.6.
Install the packages apt install {kubeadm,kubelet,kubectl}=1.15.6-00
Sadly this makes the configuration a bit more difficult. In 1.16.0 you can do everything with the command line. However we need to create a config file:
key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.6
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF
And build the cluster controller with kubeadm init --config=kubeadm-config.yaml --upload-certs
After it runs copy the config file where you want to use it from:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Finally set up the network. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" I chose weave because that’s what they mentioned in the document.
And that worked, came right up. As did the second control plane.
Upgraded them to 4/6 cpus and 16G of core a piece.
apt full-upgrade {kubectl,kubelet,kubeadm}=1.15.6-00
kubeadm join blah blah printed out on the control plane when it comes up
Easy peasy. Looks beautiful
Today kids we’re going to be trying to get kubernetes up on deepthot. I’ve been playing with microk8s at work, however gitlab seemed to be very unhappy with it. I don’t know if it’s microk8s that is the issue or not, but I’m going to try this the long way.
I’m working from https://kubernetes.io/docs/setup/ There’s a lot to learn. Anyway, at this point I’m ready to give it a go
First thing is installing kubeadm. Simple enough sudo snap install kubeadm Now before I even run it it mentions but first choose which network plugin you want to use. Joy, there’s like 8 or so of them. I’m gonna go with ‘flannel’ since it’s simple and mature. I mean we’re talking about a home network here yeah? Anyway, skipping ahead I see that flannel requires --pod-network-cidr=10.244.0.0/16 to be added to the kubeadm init command.
But but first, it wants me to run sudo kubeadm config images pull to make sure everything is working. Turns out it’s not, it wants docker. Because I work with it all the time, I think I’m going to go with docker’s docker-ce version instead of ubuntu’s.
I’ve not done this on ubuntu before, so I’m going to go with https://www.linode.com/docs/applications/containers/install-docker-ce-ubuntu-1804/ Looks really normal. So, here’s the commands:
sudo apt remove docker docker-engine docker.io
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
Ok, now the guide I’m following wants me to install docker-ce and then add my user to the docker group. I’m thinking I’m going to be doing a lot of this, and I’m lazy, so instead I think I want to create a docker group in LDAP and add myself to it. The intent here is so I don’t have to keep creating the docker group, it will be done once and have the same value on all of my machines. So, let me see how to do that
denebeim@deepthot.aa@k8s-control:~/kubernetes$ kinit denebeim
Password for denebeim@DEEPTHOT.AA:
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add docker
--------------------
Added group "docker"
--------------------
Group name: docker
GID: 1847800013
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add-member docker --groups=admins
Group name: docker
GID: 1847800013
Member groups: admins
Indirect Member users: admin, denebeim
-------------------------
Number of members added 1
-------------------------
OK, now going to install docker-ce and see what happens
denebeim@deepthot.aa@k8s-control:~/kubernetes$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ll /var/run/docker.sock
srw-rw---- 1 root docker@deepthot.aa 0 Oct 25 04:41 /var/run/docker.sock=
Boo… not working… Log out, log back in, working yay. I guess the group membership in core isn’t refreshed. I suspected as much. Anyway, we’re good.
Back to kubernetes now.
denebeim@deepthot.aa@k8s-control:~$ sudo kubeadm config images pull
cannot open path of the current working directory: Permission denied
denebeim@deepthot.aa@k8s-control:~$ sudo -i
root@k8s-control:~# kubeadm config images pull
Well… something to keep in mind. You gotta be in a root shell, not just sudoing a command to do some of this stuff.
root@k8s-control:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
[init] Using Kubernetes version: v1.16.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING FileExisting-socat]: socat not found in system path
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': executable file not found in $PATH
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
OK, not working. Looks like I need to read https://kubernetes.io/docs/setup/cri/ and kubelet hasn’t been installed yet. OK, there’s some commands in that web site, this is pretty much what I did:
# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common
### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
### Add Docker apt repository.
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu
# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
}
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart docker.
systemctl daemon-reload
systemctl restart docker
However, I’m using btrfs, so I took out this line "storage-driver": "overlay2" (which I also did in the paste above) use docker’s btrfs storage driver is a better idea.
This is having be downgrade docker. So I did. However, current docker is using containerd, I wonder why it’s not supported here. Just to be sure, I’m sticking with the old one for the time being.
Anyway, installing the snap for kubelet and trying again. Not working. I guess I was too smart for my own good. It’s insisting on the overlay2 storage driver… So…
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
systemctl daemon-reload
systemctl restart docker
I ended up having issues with btrfs subvolumes, I needed to delete them, which I did, but it would be easier if I never did it. So listen to the docs next time and don’t try to do the btrfs storage driver
Well, kubelet won’t start. It’s getting too late. Try again tomorrow. What I should do is:
kubeadm initWell, maybe anyway, I’m going to try a couple more things. See if I can get away with not having to install from scratch. Poked around removed everything I could find that was kubernetes or docker related.
# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common -y
### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
### Add Docker apt repository.
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu -y && apt-mark hold docker-ce
# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart docker.
systemctl daemon-reload
systemctl restart docker
Something annoying: see above we installed a specific version of docker. That package version was never locked in so docker gets upgraded every time you do a apt upgrade. The way to fix that is apt-mark hold <package-name> I’ve edited the commands above.
I found this on the kubernetes setup production-environment container-runtimes I’m gonna go with this first.
Same issue, kubelet doesn’t want to start. I’ve been digging around and it appears that people have been installing the kubernetes xenial repositories (there’s no bionic ones) and using those to install kubelet and kubeadm. So, let’s try that
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
apt update
apt install kubectl kubelet kubeadm kubernetes-cni -y
I found this in a post “@civik it has been resolved, you need to add --cgroup-driver=system to kubelet start up parameters in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf”
So, let’s try that. It was actually in /var/lib/kubelet/kubeadm-flags.env. Editing it the way the post said didn’t help anything. I rebooted the server and now I’ve got kubernetes containers running. Dunno how healthy it is.
I used this command kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml to get the flannel cni (network) running. Seems to have worked. Nope flannel keeps restarting.
I’m going to try uninstalling kubernetes kubeadm reset, and installing it again. This is what I did:
kubectl drain k8s-control --delete-local-data --force --ignore-daemonsets
kubectl delete node k8s-control
kubeadm reset
Which seems to have worked because…
kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
--discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
--discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993
So it looks like I have a node up. Yay. What next? Flannel again, I suppose.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml Seems to be stable this time.
I’m following this: web-ui-dashboard and this creating sample user
So, at this point I can operate kubernetes from my desktop mostly. Here’s how to set it up
scp -r k8s-control:.kube .
sudo snap install kubectl --classic
# that's it, however if you want to get to the web interface stuff
kubectl proxy &
Installing the dashboard is similarly easy, install the dashboard, create an admin user, bind it to the cluster admin role, get the token.
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
cat <<EOF >dashboard-adminuser.yaml
> apiVersion: v1
> kind: ServiceAccount
> metadata:
> name: admin-user
> namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
denebeim@frankiemouse:~/kubernetes$ cat <<EOF >dashboard-adminbind.yaml
> apiVersion: rbac.authorization.k8s.io/v1
> kind: ClusterRoleBinding
> metadata:
> name: admin-user
> roleRef:
> apiGroup: rbac.authorization.k8s.io
> kind: ClusterRole
> name: cluster-admin
> subjects:
> - kind: ServiceAccount
> name: admin-user
> namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminbind.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')|awk '/token:/{print $2}'
That’s really all there is to it. I guess this makes us done.
This is better helm upgrade --install dashboard stable/kubernetes-dashboard --namespace dashboard --set enableSkipLogin=true --set ingress.enabled=true --set enableInsecureLogin=true but it’s running a way older version of the software
Written with StackEdit.
I’ve not tried mailu yet. It looks precisely what I want, however it doesn’t appear to have helm yet, or at least it didn’t. Just in the last few weeks there was a new helm chart that has been added and is in active development. So I suspect I’m going to have to put in some changes, or at least go crazy trying to keep up with it.
The helm chart looks to be going pretty well. I got it installing sorta at work. I don’t have any internet facing infrastructure there so I couldn’t actually deploy a mail server for real.
Anyway here is the documentation for the values.yaml file:
| Parameter | Description | Default |
|---|---|---|
mailuVersion |
Version/tag of mailu images | master |
logLevel |
Level of logging | WARNING |
nameOverride |
Override the resource name prefix | mailu |
fullnameOverride |
Override the full resource names | mailu-{release-name} (or mailu if release-name is mailu) |
hostnames |
List of hostnames to generate certificates and ingresses for | not set |
domain |
Mail domain name, see https://github.com/Mailu/Mailu/blob/master/docs/faq.rst#what-is-the-difference-between-domain-and-hostnames | not set |
passwordScheme |
Scheme used to hash passwords | PBKDF2 |
secretKey |
Session encryption key for admin and webmail | not set |
mail.messageSizeLimitInMegabytes |
Message size limit in Megabytes | 50 |
mail.authRatelimit |
Rate limit for authentication requests | 10/minute;1000/hour |
initialAccount.username |
Local part (part after @) for initial admin account | not set |
initialAccount.domain |
Domain part (part before @) for initial admin account | not set |
initialAccount.password |
Password for initial admin account | not set |
certmanager.issuerType |
Issuer type for cert manager | ClusterIssuer |
certmanager.issuerName |
Name of a preconfigured cert issuer | letsencrypt |
The secretKey is required, I’m gonna go with lastpass for now.
So, this is all I’m supposed to have to do to get it up:
helm repo add mailu https://mailu.github.io/helm-charts/
helm install --values values.yaml mailu/mailu
Looks like I need persistence.type looking at the code this variable just needs to not be ‘hostPath’. It will take storageClass, however here at deepthot I’ve already got nfs-client set up as the default so I don’t have to do anything.
I’m setting persistence.type=nfs-client although that’s not really a thing.
Some of the resources were set too low causing a really unhelpful error message. I bumped them quite a bit:
front:
resources:
requests:
memory: 200Mi
cpu: 200m
limits:
memory: 1000Mi
cpu: 1000m
I’ve been spending a bunch of time installing the non helmchart version and I think I’ve pretty much figured it out.
The port mappings are all in the nginx configuration repository git@gitlab.deepthot.aa:denebeim/nginxconfig.git
| Port | Container | Purpose |
|---|---|---|
| 25 | mailu/mailu-smtp:25 | SMTP |
| 465 | mailu/mailu-smtp:465 | DEPRECATED |
| 587 | mailu/mailu-smtp:587 | SMTP-TLS |
| 2525 | mailu/mailu-smtp:2525 | SMTP |
| 10025 | mailu/mailu-smtp:10025 | SMTP-AUTH |
| 143 | mailu/mailu-imap:143 | IMAP |
| 993 | mailu/mailu-imap:993 | IMAPS |
| 110 | mailu/mailu-imap:110 | POP |
| 995 | mailu/mailu-imap:995 | POPS |
Currently mailu is actually in the mailu-mailserver namespace since that’s what the non helmified server is shipped with. So what will go into the nginxconfig is
tcp:
25: "mailu-mailserver/mailu-smtp:25"
587: "mailu-mailserver/mailu-smtp:587"
2525: "mailu-mailserver/mailu-smtp:2525"
10025: "mailu-mailserver/mailu-smtp:10025"
143: "mailu-mailserver/mailu-imap:143"
993: "mailu-mailserver/mailu-imap:993"
110: "mailu-mailserver/mailu-pop:110"
995: "mailu-mailserver/mailu-pop:995"
I’m using the kubernetes main documentation for this. I’ve got a couple of deployments that have installed their own nginx. I’m thinking I just want one for everything so here we go.
This is the basic install
helm install stable/nginx-ingress --name deepthot-nginx --namespace ingress-nginx --set rbac.create=true
Let’s get it working for the dashboard. I want it at deepthot.aa/dashboard. First set up the dashboard
helm upgrade --install dashboard stable/kubernetes-dashboard \
--namespace dashboard \
--set enableSkipLogin=true \
--set enableInsecureLogin=true \
--set ingress.enabled=true
And there it is. This created a dashboard at http(s)://192.168.42.65 Not quite what we wanted, but progress. Let’s see what it takes to move it to /dashboard.
{
"kind": "Ingress",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "dashboard-kubernetes-dashboard",
"namespace": "dashboard",
"selfLink": "/apis/extensions/v1beta1/namespaces/dashboard/ingresses/dashboard-kubernetes-dashboard",
"uid": "69dbf23d-e2c5-44e5-9519-b2f3cc48a0d2",
"resourceVersion": "2241749",
"generation": 6,
"creationTimestamp": "2019-11-09T15:12:35Z",
"labels": {
"app": "kubernetes-dashboard",
"chart": "kubernetes-dashboard-1.10.0",
"heritage": "Tiller",
"release": "dashboard"
}
},
"spec": {
"rules": [
{
"http": {
"paths": [
{
"path": "/",
"backend": {
"serviceName": "dashboard-kubernetes-dashboard",
"servicePort": 443
}
}
]
}
}
]
},
"status": {
"loadBalancer": {
"ingress": [
{}
]
}
}
}
I was trying to pass everything on the command line, but the array of annotations escaped my abilities. Anyway, here is the command: helm upgrade --install dashboard stable/kubernetes-dashboard --namespace dashboard -f kubernetes-dashboard/values.yaml and the values file:
ingress:
enabled: true
paths:
- '/dashboard(.*)'
annotations:
nginx.ingress.kubernetes.io/rewrite-target: $1
enableSkipLogin: true
enableInsecureLogin: true
However after going through all it took to get here (yes, I’m a foolish girl). I’m thinking wtf do I want it at deepthot.aa/dashboard? dashboard.deepthot.aa will let me just enter dashboard on the url in the web browser.
Here’s the revised yaml
ingress:
enabled: true
hosts:
- dashboard.deepthot.aa
- dashboard
enableSkipLogin: true
enableInsecureLogin: true
Notice I can list the hosts. For instance I could have one ingress make both the public and local DNS work.
Speaking of local DNS, I also needed to ipa dnsrecord-add deepthot.aa dashboard --cname-hostname @ Note that I’m referring to ‘@’, I also made a record for the site itself ipa dnsrecord-add --a-ip-address=192.168.42.50 deepthot.aa @
So, after all that effort to get gitlab running I’m going to undo all of the ingress stuff in gitlab and put it into my site. Turns out this is really easy.
Basically it’s just this: helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab --set nginx-ingress.enabled=false Don’t forget to edit the values.yaml file to reflect this.
ipa dnsrecord-del deepthot.aa gitlab --a-rec 192.168.42.64
ipa dnsrecord-del deepthot.aa minio --a-rec 192.168.42.64
ipa dnsrecord-del deepthot.aa registry --a-rec 192.168.42.64
ipa dnsrecord-add deepthot.aa gitlab --cname-hostname @
ipa dnsrecord-add deepthot.aa minio --cname-hostname @
ipa dnsrecord-add deepthot.aa registry --cname-hostname @
Just remove the old records and replace with the new. This will allow me to easily switch all of the services by just changing the record for deepthot.aa (the @ above)
helm upgrade --install deepthot-nginx stable/nginx-ingress --namespace ingress-nginx --set rbac.create=true --set tcp.22="gitlab/gitlab-gitlab-shell:22"
I don’t have a values.yaml, I probably should even though there’s not much to these commands. Going to do that right now.
Hi, Joy here, I'm tired. I'm too tired to polish this. However bare with me, the text below is *trying* to be me. It's not l...