Saturday, December 21, 2019

01 Deepthot Configuration

Deepthot Configuration

This is a detailed document discussing the current configuration

Ubuntu 18.04 Installs

Here are the steps for a basic install of an ubuntu VM. Every VM on deepthot has this exact configuration.

  1. Create the VM, use the disk image on the SAN
  2. Install the VM
    1. btrfs
    2. package repository http://nexus:8082/repository/ubuntu/
    3. install ssh
    4. lookup on gihub denebeim
    5. allow non ssh key logins
  3. log in from the console
  4. sudo apt update;sudo apt dist-upgrade -y (ssh is broken until you upgrade)
  5. reboot
  6. ssh in as the device specific user
  7. sudo apt install emacs-nox freeipa-client autofs nfs-kernel-server docker.io -y
  8. echo automount: files sss|sudo tee -a /etc/nsswitch.conf
  9. sudo hostnamectl set-hostname $(hostname -s).deepthot.aa
  10. sudo ipa-client-install
  11. sudo kinit
  12. sudo ipa service-add nfs/$(hostname -f)
  13. sudo ipa-getkeytab -s ipa.deepthot.aa -p nfs/$(hostname -f) -k /etc/krb5.keytab
  14. sudo klist -k /etc/krb5.keytab (to check and make sure, should have host and nfs keys)
  15. sudo ipa-client-automount
  16. sudo gpasswd -a k8s docker
  17. sudo gpasswd -a denebeim@deepthot.aa docker
  18. reboot machine and ssh in. Make sure nfs and docker are working
    NOTE: the main gotcha here is the docker group must be local. That means you need to install docker before freeipa.

Docker configuration

docker is configured to use btrfs at this point. That’s how I want it to be, however the kubernetes version we’re installing at the moment does not support btrfs docker layers. So, the next instructions will end up destroying the docker things that are running and switching it to overlay2. So don’t run anything if you’re setting up a kubernetes cluster.

sudo tee /etc/docker/daemon.json <<EOF
{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
	"max-size": "100m"
    },
    "storage-driver": "btrfs",
    "insecure-registries" : ["artifactory.deepthot.aa:8081","artifactory:8081"],
    "registry-mirrors": ["http://artifactory.deepthot.aa:8081/"]
}
EOF
sudo systemctl enable docker
sudo systemctl restart docker

I’ve been having some grief with docker.io, here is how to set up the repository for docker-ce and used it

curl -s http://nexus:8082/repository/apt-docker/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository 'deb http://nexus:8082/repository/apt-docker/linux/ubuntu bionic stable'
sudo apt install -y docker-ce

Kubernetes Cluster

Currently: 1 control node, 1 worker node

Type CPU Core Disk
Control 8 4 16
Worker 4 16 32
Worker-2 4 16 32

Control Node Configuration

Create the DNS record:

kinit denebeim
ipa dnsrecord-add deepthot.aa cluster-endpoint --a-ip-address=$(hostname -i)

We’re currently using dns as a load balancer. I’m not thinking this is working too good though. This may change to either
a non redundant control, or put an actual vm here to be the load balancer

Install the software:

sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s http://nexus:8082/repository/apt-kubernetes/doc/apt-key.gpg | sudo apt-key add -
add-apt-repository \
	'deb http://nexus:8082/repository/apt-kubernetes/ kubernetes-xenial main'
apt install {kubeadm,kubelet,kubectl}=1.15.7-00 -y --allow-downgrades -y
apt-mark hold kubectl kubelet kubeadm

Create configuration

The current kubernetes installer version needs configuration.

key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.7
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF

Configure docker daemon

sudo systemctl stop docker
sudo sed -i "/storage-driver/s/btrfs/overlay2/" /etc/docker/daemon.json
sudo rm -rf /var/lib/docker
sudo systemctl start docker

NOTE: After you stop docker, make sure there isn’t any mounts or anything left around. The rm -rf might fail.

And build the cluster controller

kubeadm init --config=kubeadm-config.yaml --upload-certs

copy the config file where you want to use it from:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Finally set up the network.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

I chose weave because that’s what they mentioned in the kubernetes.io install document.

Worker Node Configuration

Install the Software

apt-get update && apt-get install -y apt-transport-https curl
curl -s http://nexus:8082/repository/apt-kubernetes/doc/apt-key.gpg | sudo apt-key add -
add-apt-repository \
	'deb http://nexus:8082/repository/apt-kubernetes/ kubernetes-xenial main'
apt install {kubeadm,kubelet,kubectl}=1.15.7-00 -y --allow-downgrades -y
apt-mark hold kubectl kubelet kubeadm

Configure docker daemon

sudo systemctl stop docker
sudo sed -i "/storage-driver/s/btrfs/overlay2/" /etc/docker/daemon.json
sudo rm -rf /var/lib/docker
sudo systemctl start docker
sudo systemctl enable docker

NOTE: After you stop docker, make sure there isn’t any mounts or anything left around. The rm -rf might fail.

Start up the worker

sudo $(ssh k8s-control kubeadm token create --print-join-command)

Infrastructure

Now we get the infrastructure up that is:

  • helm
  • metallb
  • certmanager
  • dynamic storage provisioning
  • ingress

helm

We’re using helm 3, so no configuration needed. (other than getting helm 3)

metallb

cat <<EOF >values.yaml
configInline:
  address-pools:
  - name: deepthot.aa
    protocol: layer2
    addresses:
      - 192.168.42.64/26
EOF
kubectl create namespace deepthot-system
helm upgrade --install  metallb stable/metallb -f values.yaml

certmanager

Here’s a script to create an apiKey for an artifactory user

cat <<'EOF' >apikey.sh
#!/bin/bash  
  
# Fetch or create apikey for a user  
  
cred=$1  
if [ "$cred" == "-d" ]; then  
  delete=1  
  cred=$2  
fi  
art="http://artifactory:8081/artifactory"  
  
if [ "$delete" != "" ]; then  
  curl -u "$cred" -X DELETE $art/api/security/apiKey  
  exit  
fi  
apiKey=$(curl -u "$cred" -s -X GET $art/api/security/apiKey | jq -r .apiKey)  
if [ "$apiKey" == "null" ]; then  
  apiKey=$(curl -u "$cred" -s -X POST $art/api/security/apiKey | jq -r .apiKey);  
fi  
echo "$apiKey"
EOF
chmod +x apikey.sh

Get the helm repository for cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

(I was trying to get an artifactory virtual repository for helm working here, but no go.)

kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
helm upgrade \
--install cert-manager \
--version v0.12.0 \
jetstack/cert-manager

There’s several bits about getting the deepthot.aa signing keys in cert-manager we won’t go into that here since I’ve got the keys on my disk. There’s also some stuff about let’s encrypt. I’m just going to do the last part here since everything is already set up.

cd ~/kubenetes/certconfig
kubectl create secret tls ca-key-pair --cert=ca.crt --key=ca.key
kubectl apply -f ca-issuer.yaml
kubectl apply -f letsencrypt-staging.yaml
kubectl apply -f letsencrypt-production.yaml

(no ingress controller at this point so we’re limited as to what we can do.)

ingress

cat <<EOF >values.yaml
rbac:
  create: true

tcp:
# gitlab ssh ingress.
#  22: "gitlab/gitlab-gitlab-shell:22"

# email ingress
#  25: "mailu/mailu-front:25"
#  587: "mailu/mailu-front:587"
#  2525: "mailu/mailu-dovecot:2525"
#  10025: "mailu/mailu-front:10025"
#  143: "mailu/mailu-front:143"
#  993: "mailu/mailu-front:993"
#  110: "mailu/mailu-front:110"
#  995: "mailu/mailu-front:995"
EOF
helm upgrade --install deepthot-nginx stable/nginx-ingress -f values.yaml

Somehow my dnsrecord got missing. Unclear how that happened, but…

ipa dnsrecord-add deepthot.aa @ --a-ip-address=192.168.42.64 --a-create-reverse

If you want to get logging from the useful container in this:

kubectl -n default logs -l 'app=nginx-ingress,component=controller' -f

testing

I looked at how-to-set-up-an-nginx-ingress and ran through it, everything works up to, but not including letsencrypt and I’m not really ready to do external access yet.

dashboard

cat <<EOF >dashboard-config.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: ca-issuer
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/secure-backends: "true"
  name: dashboard
spec:
  rules:
  - host: dashboard.deepthot.aa
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 443
        path: /
  - host: dashboard
    http:
      paths:
        - backend:
            serviceName: kubernetes-dashboard
            servicePort: 443
          path: /
  tls:
  - hosts:
    - dashboard.deepthot.aa
    - dashboard
    secretName: dashboard-tls
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
kubectl apply -n kubernetes-dashboard -f dashboard-config.yaml
kubectl -n kubernetes-dashboard get secret -o json|jq '.items[] | select(.metadata.name | contains("admin-user")) | .data.token' -r| base64 --decode

Dynamic Storage Provider

dynamic storage config has more detail, we don’t need it here. Basically much of the synology configuration is outside of the realm I’m intending to cover in this document.

helm upgrade --install nfs-client-provisioner stable/nfs-client-provisioner \
--set nfs.server=$(getent hosts h2gt2g|awk '{print $1}') \
--set nfs.path=/volume1/k8s \
--set storageClass.defaultClass=true

gitlab

finally we get to something that does real work ;-) and another week month goes by…

I’ve been waffling, however really, namespaces are a good thing for keeping things straight, even if kubernetes thinks you should be a big group to have them. Anyway, lets configure a gitlab default namespace and context:

kubectl config set-context gitlab --namespace=gitlab --cluster=default --user=kubernetes-admin  
kubectl config use-context gitlab 2831  kubectl config view
kubectl create namespace gitlab

If you’re restoring from backup hopefully you’ll have the secrets, if not the database will not spark you young lady. If you have it, before you start creating stuff do this kubectl create secret generic gitlab-rails-secret --from-file=secrets.yml=secrets.yaml the actual restore you do after the rest of the setup, but the keys have to be there from the beginning.

Create static persistent volumes

cat <<EOF >pv.yaml
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: gitlab-minio  
spec:  
  capacity:  
    storage: 10Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/minio"  
  storageClassName: standard  
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: gitlab-postgresql  
spec:  
  capacity:  
    storage: 8Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/postgresql"  
  storageClassName: standard  
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: gitlab-redis  
spec:  
  capacity:  
    storage: 5Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/redis"  
  storageClassName: standard  
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: repo-data-gitlab-gitaly-0  
spec:  
  capacity:  
    storage: 50Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/gitaly"  
  storageClassName: standard
EOF

Configuration for deepthot gitlab

cat <<EOF >values.yaml
global:  
  ingress:  
    configureCertmanager: false  
    class: "nginx"  
  tls:  
      secretName: {}  
    annotations:  
      kubernetes.io/ingress.class: "nginx"  
  cert-manager.io/cluster-issuer : "ca-issuer"  
  hosts:  
    domain: deepthot.aa  
    externalIP: 192.168.42.64  
    https: false  
    gitlab:  
      name: gitlab.deepthot.aa  
      https: true  
    registry:  
      name: registry.deepthot.aa  
      https: true  
    minio:  
      name: minio.deepthot.aa  
      https: true  
  
  certificates:  
    customCAs:  
      - secret: deepthot-ca  
  appConfig:  
    ldap:  
      preventSignin: false  
      servers:  
        main:  
          ca_file: /etc/ssl/certs/ca-cert-deepthot-ca.pem  
          label: 'LDAP'  
  host: 'ipa.deepthot.aa'  
  port: 389  
          uid: 'uid'  
  bind_dn: 'uid=ldap,cn=sysaccounts,cn=etc,dc=deepthot,dc=aa'  
  password:  
            secret: deepthot-ldap-cred  
            key: password  
          base: 'cn=users,cn=accounts,dc=deepthot,dc=aa'  
  attributes:  
            username: ['uid', 'userid', 'sAMAccountName']  
            email:    ['mail', 'email', 'userPrincipalName']  
            name: 'cn'  
  first_name: 'givenName'  
  last_name:  'sn'  
  group_base: 'ou=groups,cn=accounts,dc=deepthot,dc=aa'  
  admin_group: 'admins'  
  cron_jobs:  
      stuck_ci_jobs_worker:  
        cron: "0 * * * *"  
  pipeline_schedule_worker:  
        cron: "19 * * * *"  
  expire_build_artifacts_worker:  
        cron: "50 * * * *"  
  
# Don't use certmanager, we'll use our own  
certmanager:  
  install: false  
  
nginx-ingress:  
  enabled: false  
  
gitlab-runner:  
  install: false  
  certsSecretName: deepthot-aa-crt  
  
gitlab:  
  unicorn:  
    ingress:  
      tls:  
        secretName: gitlab-gitlab-tls  
  gitaly:  
    persistence:  
      storageClass: standard  
      volumeName: repo-data-gitlab-gitaly-0  
  task-runner:  
    backups:  
      cron:  
        schedule: '@daily'  
  enabled: true  
  
registry:  
  ingress:  
    tls:  
      secretName: gitlab-registry-tls  
  
minio:  
  ingress:  
    tls:  
      secretName: gitlab-minio-tls  
  persistence:  
    storageClass: standard  
    volumeName: gitlab-minio  
  
postgresql:  
  persistence:  
    storageClass: standard  
    volumeName: gitlab-postgresql  
  
redis:  
  persistence:  
    storageClass: standard  
    volumeName: gitlab-redisEOF

LDAP configuration

kubectl create secret generic deepthot-ldap-cred --from-literal=password=youknow
kubectl create secret generic deepthot-ca --from-file=deepthot-ca.pem=/etc/ipa/ca.crt
helm repo add gitlab https://charts.gitlab.io/
helm repo update
kubectl apply -f pv.yaml
helm upgrade --install gitlab gitlab/gitlab -f values.yaml
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo

I’ve tested the above configuration, it is able to be delete and recreated with the data preserved in a well-known location. Removing the chart also requires removing the gitaly claim kubectl delete pvc repo-data-gitlab-gitaly-0 and removing the pvs kubectl delete -f pv.yaml. This leaves it in a clean state. The data is still present.

Configure ingress to work with port 22

helm upgrade --install deepthot-nginx stable/nginx-ingress \
--set tcp.22=gitlab/gitlab-gitlab-shell:22

backup gitlab

The cron is part of the values.yaml file, so that’s all set up. Plus the deepthot cert was populated for ldap, so it all just kinda works now. I’m leaving this section because some of the commands are kinda interesting.

Retrieve keys to get into minio

kubectl get secrets gitlab-minio-secret -o yaml|\
awk '/accesskey|secretkey/{cmd="echo "$2" | base64 -di"; cmd|getline x;print $1,x}'

Create the backup every day. And backup the secrets manually whenever.

kubectl exec `kubectl get pods -l'release=gitlab,app=task-runner' \
-ojsonpath='{.items[0].metadata.name}'` \
-it backup-utility
kubectl get secrets gitlab-rails-secret \
-o jsonpath="{.data['secrets\.yml']}" | base64 --decode > secrets.yaml
```

Restore
```bash
kubectl delete secret gitlab-rails-secret
kubectl create secret generic gitlab-rails-secret --from-file=secrets.yml=secrets.yaml
kubectl delete pods -lapp=sidekiq,release=gitlab
kubectl delete pods -lapp=unicorn,release=gitlab
kubectl delete pods -lapp=task-runner,release=gitlab
kubectl exec $(kubectl get pods -l'release=gitlab,app=task-runner' \
-ojsonpath='{.items[0].metadata.name}') -it -- \
backup-utility --restore -t 1576802423_2019_12_20_12.5.4-ee
```
# Done!!!
OK, not quite done, but this is now where it was and kept crashing.  I've got ubuntu, kubernetes, cert manager, nginx ingress, metal LB, default permanent storage, and source and binary control.   Next project is the mail server and since that has a way to go we'll do a development blog until it's running then reflect it here.

No comments:

Post a Comment

How Simple Does It Have to Be Before It’s Mine?

 Hi, Joy here, I'm tired.  I'm too tired to polish this.  However bare with me, the text below is *trying* to be me.  It's not l...