Tuesday, November 26, 2019

Kubernetes Dashboard

Kubernetes Dashboard Helm Chart

I’ve written a half assed kubernetes dashboard helm chart.
Deploying the chart

Old Kubernetes Dashboard

Dashboard Configuration for Deepthot

I really want a good dashboard configuration at deepthot. I had it pretty nice at one point, but it’s crap now. Actually I haven’t installed it at all on the new version. However the issue is this, the helm chart version is pretty old. The pages are totally static, I want pages that auto update. With graphics. The graphics, I haven’t quite figured out yet, heapster is supposed to be all it takes to make it work, but I’ve not gotten it to work yet.

  1. Look at the current version of the helm chart.
  2. Try again to get the latest version running.

Googling around I found kubernetes-dashboard-v2.0.0 I’m going to try this.

Well, got it working, there’s some permissions issues at this point I need to iron out. Still don’t have the helm working. The permissions issues are an issue. I’ve got it working fine other than the graphics with a custom ‘recommended.yaml’ like from the kubernetes.io dashboard install page.

# Copyright 2017 The Kubernetes Authors.  
#  
# Licensed under the Apache License, Version 2.0 (the "License");  
# you may not use this file except in compliance with the License.  
# You may obtain a copy of the License at  
#  
#     http://www.apache.org/licenses/LICENSE-2.0  
#  
# Unless required by applicable law or agreed to in writing, software  
# distributed under the License is distributed on an "AS IS" BASIS,  
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
# See the License for the specific language governing permissions and  
# limitations under the License.  
  
apiVersion: v1  
kind: Namespace  
metadata:  
  name: kubernetes-dashboard  
  
---  
  
apiVersion: v1  
kind: ServiceAccount  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard  
  namespace: kubernetes-dashboard  
  
---  
  
kind: Service  
apiVersion: v1  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard  
  namespace: kubernetes-dashboard  
spec:  
  ports:  
    - port: 443  
      targetPort: 8443  
  selector:  
    k8s-app: kubernetes-dashboard  
  
---  
  
apiVersion: v1  
kind: Secret  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard-certs  
  namespace: kubernetes-dashboard  
type: Opaque  
  
---  
  
apiVersion: v1  
kind: Secret  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard-csrf  
  namespace: kubernetes-dashboard  
type: Opaque  
data:  
  csrf: ""  
  
---  
  
apiVersion: v1  
kind: Secret  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard-key-holder  
  namespace: kubernetes-dashboard  
type: Opaque  
  
---  
  
kind: ConfigMap  
apiVersion: v1  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard-settings  
  namespace: kubernetes-dashboard  
  
---  
  
kind: Role  
apiVersion: rbac.authorization.k8s.io/v1  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard  
  namespace: kubernetes-dashboard  
rules:  
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.  
  - apiGroups: [""]  
    resources: ["secrets"]  
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]  
    verbs: ["get", "update", "delete"]  
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.  
  - apiGroups: [""]  
    resources: ["configmaps"]  
    resourceNames: ["kubernetes-dashboard-settings"]  
    verbs: ["get", "update"]  
    # Allow Dashboard to get metrics.  
  - apiGroups: [""]  
    resources: ["services"]  
    resourceNames: ["heapster", "dashboard-metrics-scraper"]  
    verbs: ["proxy"]  
  - apiGroups: [""]  
    resources: ["services/proxy"]  
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]  
    verbs: ["get"]  
  
---  
  
kind: ClusterRole  
apiVersion: rbac.authorization.k8s.io/v1  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard  
rules:  
  # Allow Metrics Scraper to get metrics from the Metrics server  
  - apiGroups: ["metrics.k8s.io"]  
    resources: ["pods", "nodes"]  
    verbs: ["get", "list", "watch"]  
  
---  
  
apiVersion: rbac.authorization.k8s.io/v1  
kind: RoleBinding  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard  
  namespace: kubernetes-dashboard  
roleRef:  
  apiGroup: rbac.authorization.k8s.io  
  kind: Role  
  name: kubernetes-dashboard  
subjects:  
  - kind: ServiceAccount  
    name: kubernetes-dashboard  
    namespace: kubernetes-dashboard  
  
---  
  
apiVersion: rbac.authorization.k8s.io/v1  
kind: ClusterRoleBinding  
metadata:  
  name: kubernetes-dashboard  
roleRef:  
  apiGroup: rbac.authorization.k8s.io  
  kind: ClusterRole  
  name: cluster-admin  
subjects:  
  - kind: ServiceAccount  
    name: kubernetes-dashboard  
    namespace: kubernetes-dashboard  
  
---  
  
kind: Deployment  
apiVersion: apps/v1  
metadata:  
  labels:  
    k8s-app: kubernetes-dashboard  
  name: kubernetes-dashboard  
  namespace: kubernetes-dashboard  
spec:  
  replicas: 1  
  revisionHistoryLimit: 10  
  selector:  
    matchLabels:  
      k8s-app: kubernetes-dashboard  
  template:  
    metadata:  
      labels:  
        k8s-app: kubernetes-dashboard  
    spec:  
      containers:  
        - name: kubernetes-dashboard  
          image: kubernetesui/dashboard:v2.0.0-beta6  
          imagePullPolicy: Always  
          ports:  
            - containerPort: 8443  
              protocol: TCP  
          args:  
            - --auto-generate-certificates  
            - --namespace=kubernetes-dashboard  
            - --enable-skip-login=true  
            - --enable-insecure-login=true  
            # Uncomment the following line to manually specify Kubernetes API server Host  
 # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port  volumeMounts:  
            - name: kubernetes-dashboard-certs  
              mountPath: /certs  
              # Create on-disk volume to store exec logs  
  - mountPath: /tmp  
              name: tmp-volume  
          livenessProbe:  
            httpGet:  
              scheme: HTTPS  
              path: /  
              port: 8443  
            initialDelaySeconds: 30  
            timeoutSeconds: 30  
          securityContext:  
            allowPrivilegeEscalation: false  
            readOnlyRootFilesystem: true  
            runAsUser: 1001  
            runAsGroup: 2001  
      volumes:  
        - name: kubernetes-dashboard-certs  
          secret:  
            secretName: kubernetes-dashboard-certs  
        - name: tmp-volume  
          emptyDir: {}  
      serviceAccountName: kubernetes-dashboard  
      nodeSelector:  
        "beta.kubernetes.io/os": linux  
      # Comment the following tolerations if Dashboard must not be deployed on master  
  tolerations:  
        - key: node-role.kubernetes.io/master  
          effect: NoSchedule  
  
---  
  
kind: Service  
apiVersion: v1  
metadata:  
  labels:  
    k8s-app: dashboard-metrics-scraper  
  name: dashboard-metrics-scraper  
  namespace: kubernetes-dashboard  
spec:  
  ports:  
    - port: 8000  
      targetPort: 8000  
  selector:  
    k8s-app: dashboard-metrics-scraper  
  
---  
  
kind: Deployment  
apiVersion: apps/v1  
metadata:  
  labels:  
    k8s-app: dashboard-metrics-scraper  
  name: dashboard-metrics-scraper  
  namespace: kubernetes-dashboard  
spec:  
  replicas: 1  
  revisionHistoryLimit: 10  
  selector:  
    matchLabels:  
      k8s-app: dashboard-metrics-scraper  
  template:  
    metadata:  
      labels:  
        k8s-app: dashboard-metrics-scraper  
      annotations:  
        seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'  
  spec:  
      containers:  
        - name: dashboard-metrics-scraper  
          image: kubernetesui/metrics-scraper:v1.0.1  
          ports:  
            - containerPort: 8000  
              protocol: TCP  
          livenessProbe:  
            httpGet:  
              scheme: HTTP  
              path: /  
              port: 8000  
            initialDelaySeconds: 30  
            timeoutSeconds: 30  
          volumeMounts:  
          - mountPath: /tmp  
            name: tmp-volume  
          securityContext:  
            allowPrivilegeEscalation: false  
            readOnlyRootFilesystem: true  
            runAsUser: 1001  
            runAsGroup: 2001  
      serviceAccountName: kubernetes-dashboard  
      nodeSelector:  
        "beta.kubernetes.io/os": linux  
      # Comment the following tolerations if Dashboard must not be deployed on master  
  tolerations:  
        - key: node-role.kubernetes.io/master  
          effect: NoSchedule  
      volumes:  
        - name: tmp-volume  
          emptyDir: {}  
  
---  
  
apiVersion: extensions/v1beta1  
kind: Ingress  
metadata:  
  labels:  
    app: kubernetes-dashboard  
  name: dashboard  
  namespace: kubernetes-dashboard  
  annotations:  
    cert-manager.io/cluster-issuer: ca-issuer  
    kubernetes.io/ingress.class: nginx  
    kubernetes.io/ingress.provider: nginx  
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"  
spec:  
  tls:  
    - secretName: dashboard-tls  
      hosts:  
        - dashboard.deepthot.aa  
        - dashboard  
  rules:  
    - http:  
        paths:  
        - backend:  
            serviceName: kubernetes-dashboard  
            servicePort: 443  
          path: /  
      host:  
        dashboard.deepthot.aa  
    - http:  
       paths:  
          - backend:  
              serviceName: kubernetes-dashboard  
              servicePort: 443  
            path: /  
      host:  
        dashboard

Heck with this. I’ve got the recommended working, I’m going to make my own helm chart from it.

Thursday, November 21, 2019

Gitlab Kubernetes

Current Configuration

Note: Helm 2.16.0 is broken don’t use it. That was the snap as of 11/21/19 emphasized text
This is a total win. I’ve got the cert manager working great. Internal and external. This works with the helm chart as shipped to deploy/upgrade simply:

helm repo add gitlab https://charts.gitlab.io/
helm repo update
helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab

Here’s the current configure:

# values-minikube.yaml  
# This example intended as baseline to use Minikube for the deployment of GitLab  
# - Services that are not compatible with how Minikube runs are disabled  
# - Configured to use 192.168.99.100, and nip.io for the domain  
  
# Minimal settings  
global:  
  ingress:  
    configureCertmanager: false  
    class: "nginx"  
  tls:  
      secretName: {}  
  annotations:  
    kubernetes.io/ingress.class: "nginx"  
    cert-manager.io/cluster-issuer : "ca-issuer"  
  
  hosts:  
    domain: deepthot.aa  
    externalIP: 192.168.42.64  
    https: false  
    gitlab:  
      name: gitlab.deepthot.aa  
      https: true  
    registry:  
      name: registry.deepthot.aa  
      https: true  
    minio:  
      name: minio.deepthot.aa  
      https: true  
  
# Don't use certmanager, we'll use our own  
certmanager:  
  install: false  
  
nginx-ingress:  
  enabled: false  
  
gitlab-runner:  
  install: false  
  certsSecretName: deepthot-aa-crt  
  
gitlab:  
  unicorn:  
    ingress:  
      tls:  
        secretName: gitlab-gitlab-tls  
  
registry:  
  ingress:  
    tls:  
      secretName: gitlab-registry-tls  
  
minio:  
  ingress:  
    tls:  
      secretName: gitlab-minio-tls

Old configuration

Well, yet again I was making things too hard, ish. Where I am right now is I’ve set up a values.yaml file

# values-minikube.yaml  
# This example intended as baseline to use Minikube for the deployment of GitLab  
# - Services that are not compatible with how Minikube runs are disabled  
# - Configured to use 192.168.99.100, and nip.io for the domain  
  
# Minimal settings  
global:  
  ingress:  
    configureCertmanager: false  
    class: "nginx"  
  hosts:  
    domain: deepthot.aa  
    externalIP: 192.168.42.64  
    https: false  
    gitlab:  
      name: gitlab.deepthot.aa  
      https: false  
    registry:  
      name: registry.deepthot.aa  
      https: false  
    minio:  
      name: minio.deepthot.aa  
      https: false  
    hosts:  
# Don't use certmanager, we'll self-sign  
certmanager:  
  install: false  
# Use the `ingress` addon, not our Ingress (can't map 22/80/443)  
nginx-ingress:  
  enabled: true  
# GitLab Runner isn't a big fan of self-signed certificates  
gitlab-runner:  
  install: false  

Then I type helm upgrade --install -f values-deepthot.yaml --namespace gitlab gitlab . However I had to muck with the helm chart to work with the kubernetes I’m running here 1.16.2. The thing is they deleted several deprecated things and all of the helm charts I’ve played with have not been updated. I’m going to downgrade kubernetes and then get back to here.

Right now it is configured for a self-signed cert. That makes gitlab runner not function. So I need to generate a cert and get the deepthot.aa cert into it. I’ll do that after kubernetes downgrading though.

Downgrading was really easy. Just follow kubeadm-upgrade/ I downgraded the worker first, dunno if it was necessary, but all went clean. Now to try the helmchart from the repo again.

I think it’s working as well as it ever did. Now though it needs to have a real cert.
Wildcard_certificates that’s very problematical. OK. New plan, move my lan onto a delegated subdomain of deepthot.org. That’s a big production. I was planning on doing this anyway, but for the time being I really don’t need all that many certs. So, let’s do them by hand.

#!/bin/bash

dn=$(hostname -d)
me=$(hostname -f)
ns="gitlab"
ip="192.168.42.64"

kubectl create namespace "${ns}"
for host in gitlab registry minio; do
 ipa dnsrecord-add "${dn}" $host --a-rec "${ip}"
 ipa host-add $host."${dn}"
 ipa service-add HTTP/$host."${dn}"
 ipa service-add-host --hosts="${me}" HTTP/$host."${dn}"
 sudo ipa-getcert request -r -f $host."${dn}".crt -k $host."${dn}".key -N CN=$host."${dn}" -D $host."${dn}" -K HTTP/$host."${dn}" -w
 sudo chown denebeim: $host."${dn}".crt $host."${dn}".key
 kubectl create secret tls gitlab-$host-tls --cert=./$host."${dn}".crt --key=./$host."${dn}".key --namespace="${ns}"
 cat /etc/ipa/ca.crt >>"${host}"."${dn}".crt
done
kubectl --namespace ${ns} create secret generic deepthot-aa-crt --from-file={,{gitlab,minio,registry}.}deepthot.aa.crt
helm upgrade --install -f values.yaml --namespace "${ns}" "${ns}" gitlab/gitlab 

Next step is to get gitlab runner up. It is not liking the custom CA that we have here.
gitlab-runner/issues/3968 describes the problem. There were a few workarounds, I tried figuring out the root cause, then I tried the solutions. The documentation says I should be concatenating the chain of certs.

I did this, replaced all the certs, didn’t help. Finally I looked at the stored cert in kubernetes. It was encrypted, but you know, it looked like the length was about the length of the key before it was concatenated. Also, I noticed the logs that came out when it was sucking the key in was saying ‘Updating local copy of SSL certificate “gitlab/gitlab-gitlab-tls” with missing intermediate CA certs’ Interesting. So I reread the bug report and one of the comments mentioned working around the problem by creating a generic secret. As you can see above the secrets we were building were tls secrets. So, I tried making a generic secret from the gitlab one we used to make the tls secret earlier. That worked. This is what you do:

cat /etc/ipa/ca.crt >>gitlab.deepthot.aa.crt
kubectl \
  --namespace gitlab \
  create secret generic gitlab-deepthot-aa-crt \
  --from-file=gitlab.deepthot.aa.crt
helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab \
	--set gitlab-runner.install=true \
	--set gitlab-runner.certsSecretName=gitlab-deepthot-aa-crt

The first script has been modified to include concatenating /etc/ipa/ca.crt onto all of the certs that it is making. However
there’s still issues. Basically reading on the internet someone mentioned how problematical it is to use your own CA. Every container on the planet has it’s own copy of SSL stuff. You can’t put your CA into all of them. I ran into the same sort of trouble at work. There I just rebuilt everything and put our CA into the base container. The solution is to get us on a public DNS we control and get let’s encrypt working. But that’s a pretty big thing, and I guess all this stuff can wait until then.

I guess that means I’m done with gitlab for the moment. I’ll call this a win.

Bare Metal Load Balancer (metallb) on Deepthot

MetalLB

This is a really short and easy one. I already know metalLB orks. I just want to change the configuration. Let’s give it a real cidr.

cat <<EOF >values.yaml
configInline:
  address-pools:
  - name: deepthot.int
    protocol: layer2
    addresses:
      - 192.168.42.64/26
EOF
helm upgrade --install --namespace metallb metallb stable/metallb -f values.yaml

Now, this is going to overlap with dhcpd so that needs to be updated as well.

   range 192.168.42.128 192.168.42.254;

The range needed to be changed from 100 to 128

Written with StackEdit.

Helm on Deepthot

Installing Helm

Well, this is odd, I’ve not run into this before. When I was trying to reinstall cert-manager I got

helm install \
> --name cert-manager \
> --namespace cert-manager \
> --version v0.11.0 \
> jetstack/cert-manager
Error: release cert-manager failed: namespaces "cert-manager" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "cert-manager"

Googling I found Helm Example Service account with cluster admin role Which shows:

cat <<EOF >rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF
kubectl create -f rbac-config.yaml
helm init --service-account tiller

I’ll get this in a gitlab repo once I have gitlab up, but for now, it’ll just go in this doc, hence the cat stuff. Looks like that’s all that’s needed.

Wednesday, November 20, 2019

Kubernetes on Deepthot

Kubernetes on Deepthot v2

Installing control plane

The current version of kubernetes is 1.16.3. However it has deleted a bunch of deprecated configuration options which unfortunately are being used by pretty much every helm chart I’ve run across. Since I’m just learning I’m going to stick with version 1.15.6.

Install the packages apt install {kubeadm,kubelet,kubectl}=1.15.6-00

Sadly this makes the configuration a bit more difficult. In 1.16.0 you can do everything with the command line. However we need to create a config file:

key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.6
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF

And build the cluster controller with kubeadm init --config=kubeadm-config.yaml --upload-certs
After it runs copy the config file where you want to use it from:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Finally set up the network. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" I chose weave because that’s what they mentioned in the document.

And that worked, came right up. As did the second control plane.

Worker nodes

Upgraded them to 4/6 cpus and 16G of core a piece.

apt full-upgrade  {kubectl,kubelet,kubeadm}=1.15.6-00
kubeadm join blah blah printed out on the control plane when it comes up

Easy peasy. Looks beautiful

Older Version

Today kids we’re going to be trying to get kubernetes up on deepthot. I’ve been playing with microk8s at work, however gitlab seemed to be very unhappy with it. I don’t know if it’s microk8s that is the issue or not, but I’m going to try this the long way.

I’m working from https://kubernetes.io/docs/setup/ There’s a lot to learn. Anyway, at this point I’m ready to give it a go

First thing is installing kubeadm. Simple enough sudo snap install kubeadm Now before I even run it it mentions but first choose which network plugin you want to use. Joy, there’s like 8 or so of them. I’m gonna go with ‘flannel’ since it’s simple and mature. I mean we’re talking about a home network here yeah? Anyway, skipping ahead I see that flannel requires --pod-network-cidr=10.244.0.0/16 to be added to the kubeadm init command.

But but first, it wants me to run sudo kubeadm config images pull to make sure everything is working. Turns out it’s not, it wants docker. Because I work with it all the time, I think I’m going to go with docker’s docker-ce version instead of ubuntu’s.

I’ve not done this on ubuntu before, so I’m going to go with https://www.linode.com/docs/applications/containers/install-docker-ce-ubuntu-1804/ Looks really normal. So, here’s the commands:

sudo apt remove docker docker-engine docker.io
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update

Ok, now the guide I’m following wants me to install docker-ce and then add my user to the docker group. I’m thinking I’m going to be doing a lot of this, and I’m lazy, so instead I think I want to create a docker group in LDAP and add myself to it. The intent here is so I don’t have to keep creating the docker group, it will be done once and have the same value on all of my machines. So, let me see how to do that

denebeim@deepthot.aa@k8s-control:~/kubernetes$ kinit denebeim
Password for denebeim@DEEPTHOT.AA: 
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add docker
--------------------
Added group "docker"
--------------------
  Group name: docker
  GID: 1847800013
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ipa group-add-member docker  --groups=admins
  Group name: docker
  GID: 1847800013
  Member groups: admins
  Indirect Member users: admin, denebeim
-------------------------
Number of members added 1
-------------------------

OK, now going to install docker-ce and see what happens

denebeim@deepthot.aa@k8s-control:~/kubernetes$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied
denebeim@deepthot.aa@k8s-control:~/kubernetes$ ll /var/run/docker.sock 
srw-rw---- 1 root docker@deepthot.aa 0 Oct 25 04:41 /var/run/docker.sock=

Boo… not working… Log out, log back in, working yay. I guess the group membership in core isn’t refreshed. I suspected as much. Anyway, we’re good.

Back to kubernetes now.

denebeim@deepthot.aa@k8s-control:~$ sudo kubeadm config images pull
cannot open path of the current working directory: Permission denied
denebeim@deepthot.aa@k8s-control:~$ sudo -i
root@k8s-control:~# kubeadm config images pull

Well… something to keep in mind. You gotta be in a root shell, not just sudoing a command to do some of this stuff.

root@k8s-control:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
[init] Using Kubernetes version: v1.16.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING FileExisting-socat]: socat not found in system path
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': executable file not found in $PATH
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

OK, not working. Looks like I need to read https://kubernetes.io/docs/setup/cri/ and kubelet hasn’t been installed yet. OK, there’s some commands in that web site, this is pretty much what I did:

# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  } 
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

However, I’m using btrfs, so I took out this line "storage-driver": "overlay2" (which I also did in the paste above) use docker’s btrfs storage driver is a better idea.

This is having be downgrade docker. So I did. However, current docker is using containerd, I wonder why it’s not supported here. Just to be sure, I’m sticking with the old one for the time being.

Anyway, installing the snap for kubelet and trying again. Not working. I guess I was too smart for my own good. It’s insisting on the overlay2 storage driver… So…

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
systemctl daemon-reload
systemctl restart docker

I ended up having issues with btrfs subvolumes, I needed to delete them, which I did, but it would be easier if I never did it. So listen to the docs next time and don’t try to do the btrfs storage driver

Well, kubelet won’t start. It’s getting too late. Try again tomorrow. What I should do is:

  1. ipa-client-install --uninstall k8s-control
  2. delete k8s-control (bet I didn’t take a snapshot, so so after doing the base install tomorrow)
  3. poke around ipa and look for stuff to delete, there’s probably some dns stuff
  4. recreate k8s-control
  5. install docker-ce
  6. make sure to do it with the overlay config and systemd
  7. only now snap install kubeadm and kubelet
  8. try for another run of kubeadm init

Second attempt

Well, maybe anyway, I’m going to try a couple more things. See if I can get away with not having to install from scratch. Poked around removed everything I could find that was kubernetes or docker related.

# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common -y

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

## Install Docker CE.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu -y && apt-mark hold docker-ce

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

Something annoying: see above we installed a specific version of docker. That package version was never locked in so docker gets upgraded every time you do a apt upgrade. The way to fix that is apt-mark hold <package-name> I’ve edited the commands above.

I found this on the kubernetes setup production-environment container-runtimes I’m gonna go with this first.

Same issue, kubelet doesn’t want to start. I’ve been digging around and it appears that people have been installing the kubernetes xenial repositories (there’s no bionic ones) and using those to install kubelet and kubeadm. So, let’s try that

cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
apt update
apt install kubectl kubelet kubeadm kubernetes-cni -y

I found this in a post “@civik it has been resolved, you need to add --cgroup-driver=system to kubelet start up parameters in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf”

So, let’s try that. It was actually in /var/lib/kubelet/kubeadm-flags.env. Editing it the way the post said didn’t help anything. I rebooted the server and now I’ve got kubernetes containers running. Dunno how healthy it is.

I used this command kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml to get the flannel cni (network) running. Seems to have worked. Nope flannel keeps restarting.

I’m going to try uninstalling kubernetes kubeadm reset, and installing it again. This is what I did:

kubectl drain k8s-control --delete-local-data --force --ignore-daemonsets
kubectl delete node k8s-control
kubeadm reset

Which seems to have worked because…

kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=cluster-endpoint
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
    --discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993 \
    --control-plane 	  

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token fzgjty.cxk9bjk271a077y8 \
    --discovery-token-ca-cert-hash sha256:ee6e28702bba0ff646404f21c4bee584ee524ce1a734f579ed4b2f55db817993 

So it looks like I have a node up. Yay. What next? Flannel again, I suppose.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml Seems to be stable this time.

Setting up the dashboard

I’m following this: web-ui-dashboard and this creating sample user

So, at this point I can operate kubernetes from my desktop mostly. Here’s how to set it up

scp -r k8s-control:.kube .
sudo snap install kubectl --classic
# that's it, however if you want to get to the web interface stuff
kubectl proxy &

Installing the dashboard is similarly easy, install the dashboard, create an admin user, bind it to the cluster admin role, get the token.

denebeim@frankiemouse:~/kubernetes$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
cat <<EOF >dashboard-adminuser.yaml
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: admin-user
>   namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
denebeim@frankiemouse:~/kubernetes$ cat <<EOF >dashboard-adminbind.yaml
> apiVersion: rbac.authorization.k8s.io/v1
> kind: ClusterRoleBinding
> metadata:
>   name: admin-user
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: cluster-admin
> subjects:
> - kind: ServiceAccount
>   name: admin-user
>   namespace: kubernetes-dashboard
> EOF
denebeim@frankiemouse:~/kubernetes$ kubectl apply -f dashboard-adminbind.yaml 
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')|awk '/token:/{print $2}'

That’s really all there is to it. I guess this makes us done.

This is better helm upgrade --install dashboard stable/kubernetes-dashboard --namespace dashboard --set enableSkipLogin=true --set ingress.enabled=true --set enableInsecureLogin=true but it’s running a way older version of the software

Written with StackEdit.

Sunday, November 17, 2019

Mailu configuration

Mailu discussion

I’ve not tried mailu yet. It looks precisely what I want, however it doesn’t appear to have helm yet, or at least it didn’t. Just in the last few weeks there was a new helm chart that has been added and is in active development. So I suspect I’m going to have to put in some changes, or at least go crazy trying to keep up with it.

First pass at configuration

The helm chart looks to be going pretty well. I got it installing sorta at work. I don’t have any internet facing infrastructure there so I couldn’t actually deploy a mail server for real.
Anyway here is the documentation for the values.yaml file:

Parameter Description Default
mailuVersion Version/tag of mailu images master
logLevel Level of logging WARNING
nameOverride Override the resource name prefix mailu
fullnameOverride Override the full resource names mailu-{release-name} (or mailu if release-name is mailu)
hostnames List of hostnames to generate certificates and ingresses for not set
domain Mail domain name, see https://github.com/Mailu/Mailu/blob/master/docs/faq.rst#what-is-the-difference-between-domain-and-hostnames not set
passwordScheme Scheme used to hash passwords PBKDF2
secretKey Session encryption key for admin and webmail not set
mail.messageSizeLimitInMegabytes Message size limit in Megabytes 50
mail.authRatelimit Rate limit for authentication requests 10/minute;1000/hour
initialAccount.username Local part (part after @) for initial admin account not set
initialAccount.domain Domain part (part before @) for initial admin account not set
initialAccount.password Password for initial admin account not set
certmanager.issuerType Issuer type for cert manager ClusterIssuer
certmanager.issuerName Name of a preconfigured cert issuer letsencrypt

The secretKey is required, I’m gonna go with lastpass for now.
So, this is all I’m supposed to have to do to get it up:

helm repo add mailu https://mailu.github.io/helm-charts/
helm install --values values.yaml mailu/mailu

Looks like I need persistence.type looking at the code this variable just needs to not be ‘hostPath’. It will take storageClass, however here at deepthot I’ve already got nfs-client set up as the default so I don’t have to do anything.
I’m setting persistence.type=nfs-client although that’s not really a thing.

Some of the resources were set too low causing a really unhelpful error message. I bumped them quite a bit:

front:
  resources:
    requests:
      memory: 200Mi
      cpu: 200m
    limits:
      memory: 1000Mi
      cpu: 1000m

Taking it apart and working on the pieces

I’ve been spending a bunch of time installing the non helmchart version and I think I’ve pretty much figured it out.

Setting up ingresses for mail

The port mappings are all in the nginx configuration repository git@gitlab.deepthot.aa:denebeim/nginxconfig.git

Port Container Purpose
25 mailu/mailu-smtp:25 SMTP
465 mailu/mailu-smtp:465 DEPRECATED
587 mailu/mailu-smtp:587 SMTP-TLS
2525 mailu/mailu-smtp:2525 SMTP
10025 mailu/mailu-smtp:10025 SMTP-AUTH
143 mailu/mailu-imap:143 IMAP
993 mailu/mailu-imap:993 IMAPS
110 mailu/mailu-imap:110 POP
995 mailu/mailu-imap:995 POPS

Currently mailu is actually in the mailu-mailserver namespace since that’s what the non helmified server is shipped with. So what will go into the nginxconfig is

tcp:
  25: "mailu-mailserver/mailu-smtp:25"
  587: "mailu-mailserver/mailu-smtp:587"
  2525: "mailu-mailserver/mailu-smtp:2525"
  10025: "mailu-mailserver/mailu-smtp:10025"
  143: "mailu-mailserver/mailu-imap:143"
  993: "mailu-mailserver/mailu-imap:993"
  110: "mailu-mailserver/mailu-pop:110"
  995: "mailu-mailserver/mailu-pop:995"

Sunday, November 10, 2019

NGINX Ingress Controller

I’m using the kubernetes main documentation for this. I’ve got a couple of deployments that have installed their own nginx. I’m thinking I just want one for everything so here we go.

This is the basic install

helm install stable/nginx-ingress --name deepthot-nginx --namespace ingress-nginx --set rbac.create=true

Configuration

Dashboard

Let’s get it working for the dashboard. I want it at deepthot.aa/dashboard. First set up the dashboard

 helm upgrade  --install dashboard stable/kubernetes-dashboard \
 --namespace dashboard  \
 --set enableSkipLogin=true \
 --set enableInsecureLogin=true \
 --set ingress.enabled=true

And there it is. This created a dashboard at http(s)://192.168.42.65 Not quite what we wanted, but progress. Let’s see what it takes to move it to /dashboard.

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "dashboard-kubernetes-dashboard",
    "namespace": "dashboard",
    "selfLink": "/apis/extensions/v1beta1/namespaces/dashboard/ingresses/dashboard-kubernetes-dashboard",
    "uid": "69dbf23d-e2c5-44e5-9519-b2f3cc48a0d2",
    "resourceVersion": "2241749",
    "generation": 6,
    "creationTimestamp": "2019-11-09T15:12:35Z",
    "labels": {
      "app": "kubernetes-dashboard",
      "chart": "kubernetes-dashboard-1.10.0",
      "heritage": "Tiller",
      "release": "dashboard"
    }
  },
  "spec": {
    "rules": [
      {
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "dashboard-kubernetes-dashboard",
                "servicePort": 443
              }
            }
          ]
        }
      }
    ]
  },
  "status": {
    "loadBalancer": {
      "ingress": [
        {}
      ]
    }
  }
}

I was trying to pass everything on the command line, but the array of annotations escaped my abilities. Anyway, here is the command: helm upgrade --install dashboard stable/kubernetes-dashboard --namespace dashboard -f kubernetes-dashboard/values.yaml and the values file:

ingress:
  enabled: true
  paths:
    - '/dashboard(.*)'
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: $1
enableSkipLogin: true
enableInsecureLogin: true

However after going through all it took to get here (yes, I’m a foolish girl). I’m thinking wtf do I want it at deepthot.aa/dashboard? dashboard.deepthot.aa will let me just enter dashboard on the url in the web browser.
Here’s the revised yaml

ingress:  
  enabled: true  
  hosts:  
    - dashboard.deepthot.aa  
    - dashboard  
enableSkipLogin: true  
enableInsecureLogin: true

Notice I can list the hosts. For instance I could have one ingress make both the public and local DNS work.
Speaking of local DNS, I also needed to ipa dnsrecord-add deepthot.aa dashboard --cname-hostname @ Note that I’m referring to ‘@’, I also made a record for the site itself ipa dnsrecord-add --a-ip-address=192.168.42.50 deepthot.aa @

gitlab

So, after all that effort to get gitlab running I’m going to undo all of the ingress stuff in gitlab and put it into my site. Turns out this is really easy.

  1. Aim the site at gitlab turning off the old ingress
  2. Update DNS
  3. Set up the mapping for the ssh interface

Aim the site at gitlab turning off the old ingress

Basically it’s just this: helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab --set nginx-ingress.enabled=false Don’t forget to edit the values.yaml file to reflect this.

Update the DNS

ipa dnsrecord-del  deepthot.aa gitlab --a-rec 192.168.42.64
ipa dnsrecord-del  deepthot.aa minio --a-rec 192.168.42.64
ipa dnsrecord-del  deepthot.aa registry --a-rec 192.168.42.64
ipa dnsrecord-add deepthot.aa gitlab --cname-hostname @
ipa dnsrecord-add deepthot.aa minio --cname-hostname @
ipa dnsrecord-add deepthot.aa registry --cname-hostname @

Just remove the old records and replace with the new. This will allow me to easily switch all of the services by just changing the record for deepthot.aa (the @ above)

Set up the mapping for the ssh interface

helm upgrade --install deepthot-nginx stable/nginx-ingress --namespace ingress-nginx --set rbac.create=true --set tcp.22="gitlab/gitlab-gitlab-shell:22"
I don’t have a values.yaml, I probably should even though there’s not much to these commands. Going to do that right now.

How Simple Does It Have to Be Before It’s Mine?

 Hi, Joy here, I'm tired.  I'm too tired to polish this.  However bare with me, the text below is *trying* to be me.  It's not l...