Deepthot Configuration
This is a detailed document discussing the current configuration
Ubuntu 18.04 Installs
Here are the steps for a basic install of an ubuntu VM. Every VM on deepthot has this exact configuration.
- Create the VM, use the disk image on the SAN
- Install the VM
- btrfs
- package repository http://nexus:8082/repository/ubuntu/
- install ssh
- lookup on gihub denebeim
- allow non ssh key logins
- log in from the console
- sudo apt update;sudo apt dist-upgrade -y (ssh is broken until you upgrade)
- reboot
- ssh in as the device specific user
- sudo apt install emacs-nox freeipa-client autofs nfs-kernel-server docker.io -y
echo automount: files sss|sudo tee -a /etc/nsswitch.conf- sudo hostnamectl set-hostname $(hostname -s).deepthot.aa
- sudo ipa-client-install
- sudo kinit
- sudo ipa service-add nfs/$(hostname -f)
- sudo ipa-getkeytab -s ipa.deepthot.aa -p nfs/$(hostname -f) -k /etc/krb5.keytab
- sudo klist -k /etc/krb5.keytab (to check and make sure, should have host and nfs keys)
- sudo ipa-client-automount
- sudo gpasswd -a k8s docker
- sudo gpasswd -a denebeim@deepthot.aa docker
- reboot machine and ssh in. Make sure nfs and docker are working
NOTE: the main gotcha here is the docker group must be local. That means you need to install docker before freeipa.
Docker configuration
docker is configured to use btrfs at this point. That’s how I want it to be, however the kubernetes version we’re installing at the moment does not support btrfs docker layers. So, the next instructions will end up destroying the docker things that are running and switching it to overlay2. So don’t run anything if you’re setting up a kubernetes cluster.
sudo tee /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "btrfs",
"insecure-registries" : ["artifactory.deepthot.aa:8081","artifactory:8081"],
"registry-mirrors": ["http://artifactory.deepthot.aa:8081/"]
}
EOF
sudo systemctl enable docker
sudo systemctl restart docker
I’ve been having some grief with docker.io, here is how to set up the repository for docker-ce and used it
curl -s http://nexus:8082/repository/apt-docker/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository 'deb http://nexus:8082/repository/apt-docker/linux/ubuntu bionic stable'
sudo apt install -y docker-ce
Kubernetes Cluster
Currently: 1 control node, 1 worker node
| Type | CPU | Core | Disk |
|---|---|---|---|
| Control | 8 | 4 | 16 |
| Worker | 4 | 16 | 32 |
| Worker-2 | 4 | 16 | 32 |
Control Node Configuration
Create the DNS record:
kinit denebeim
ipa dnsrecord-add deepthot.aa cluster-endpoint --a-ip-address=$(hostname -i)
We’re currently using dns as a load balancer. I’m not thinking this is working too good though. This may change to either
a non redundant control, or put an actual vm here to be the load balancer
Install the software:
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s http://nexus:8082/repository/apt-kubernetes/doc/apt-key.gpg | sudo apt-key add -
add-apt-repository \
'deb http://nexus:8082/repository/apt-kubernetes/ kubernetes-xenial main'
apt install {kubeadm,kubelet,kubectl}=1.15.7-00 -y --allow-downgrades -y
apt-mark hold kubectl kubelet kubeadm
Create configuration
The current kubernetes installer version needs configuration.
key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.7
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF
Configure docker daemon
sudo systemctl stop docker
sudo sed -i "/storage-driver/s/btrfs/overlay2/" /etc/docker/daemon.json
sudo rm -rf /var/lib/docker
sudo systemctl start docker
NOTE: After you stop docker, make sure there isn’t any mounts or anything left around. The rm -rf might fail.
And build the cluster controller
kubeadm init --config=kubeadm-config.yaml --upload-certs
copy the config file where you want to use it from:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Finally set up the network.
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
I chose weave because that’s what they mentioned in the kubernetes.io install document.
Worker Node Configuration
Install the Software
apt-get update && apt-get install -y apt-transport-https curl
curl -s http://nexus:8082/repository/apt-kubernetes/doc/apt-key.gpg | sudo apt-key add -
add-apt-repository \
'deb http://nexus:8082/repository/apt-kubernetes/ kubernetes-xenial main'
apt install {kubeadm,kubelet,kubectl}=1.15.7-00 -y --allow-downgrades -y
apt-mark hold kubectl kubelet kubeadm
Configure docker daemon
sudo systemctl stop docker
sudo sed -i "/storage-driver/s/btrfs/overlay2/" /etc/docker/daemon.json
sudo rm -rf /var/lib/docker
sudo systemctl start docker
sudo systemctl enable docker
NOTE: After you stop docker, make sure there isn’t any mounts or anything left around. The rm -rf might fail.
Start up the worker
sudo $(ssh k8s-control kubeadm token create --print-join-command)
Infrastructure
Now we get the infrastructure up that is:
- helm
- metallb
- certmanager
- dynamic storage provisioning
- ingress
helm
We’re using helm 3, so no configuration needed. (other than getting helm 3)
metallb
cat <<EOF >values.yaml
configInline:
address-pools:
- name: deepthot.aa
protocol: layer2
addresses:
- 192.168.42.64/26
EOF
kubectl create namespace deepthot-system
helm upgrade --install metallb stable/metallb -f values.yaml
certmanager
Here’s a script to create an apiKey for an artifactory user
cat <<'EOF' >apikey.sh
#!/bin/bash
# Fetch or create apikey for a user
cred=$1
if [ "$cred" == "-d" ]; then
delete=1
cred=$2
fi
art="http://artifactory:8081/artifactory"
if [ "$delete" != "" ]; then
curl -u "$cred" -X DELETE $art/api/security/apiKey
exit
fi
apiKey=$(curl -u "$cred" -s -X GET $art/api/security/apiKey | jq -r .apiKey)
if [ "$apiKey" == "null" ]; then
apiKey=$(curl -u "$cred" -s -X POST $art/api/security/apiKey | jq -r .apiKey);
fi
echo "$apiKey"
EOF
chmod +x apikey.sh
Get the helm repository for cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
(I was trying to get an artifactory virtual repository for helm working here, but no go.)
kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
helm upgrade \
--install cert-manager \
--version v0.12.0 \
jetstack/cert-manager
There’s several bits about getting the deepthot.aa signing keys in cert-manager we won’t go into that here since I’ve got the keys on my disk. There’s also some stuff about let’s encrypt. I’m just going to do the last part here since everything is already set up.
cd ~/kubenetes/certconfig
kubectl create secret tls ca-key-pair --cert=ca.crt --key=ca.key
kubectl apply -f ca-issuer.yaml
kubectl apply -f letsencrypt-staging.yaml
kubectl apply -f letsencrypt-production.yaml
(no ingress controller at this point so we’re limited as to what we can do.)
ingress
cat <<EOF >values.yaml
rbac:
create: true
tcp:
# gitlab ssh ingress.
# 22: "gitlab/gitlab-gitlab-shell:22"
# email ingress
# 25: "mailu/mailu-front:25"
# 587: "mailu/mailu-front:587"
# 2525: "mailu/mailu-dovecot:2525"
# 10025: "mailu/mailu-front:10025"
# 143: "mailu/mailu-front:143"
# 993: "mailu/mailu-front:993"
# 110: "mailu/mailu-front:110"
# 995: "mailu/mailu-front:995"
EOF
helm upgrade --install deepthot-nginx stable/nginx-ingress -f values.yaml
Somehow my dnsrecord got missing. Unclear how that happened, but…
ipa dnsrecord-add deepthot.aa @ --a-ip-address=192.168.42.64 --a-create-reverse
If you want to get logging from the useful container in this:
kubectl -n default logs -l 'app=nginx-ingress,component=controller' -f
testing
I looked at how-to-set-up-an-nginx-ingress and ran through it, everything works up to, but not including letsencrypt and I’m not really ready to do external access yet.
dashboard
cat <<EOF >dashboard-config.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: ca-issuer
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/secure-backends: "true"
name: dashboard
spec:
rules:
- host: dashboard.deepthot.aa
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 443
path: /
- host: dashboard
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 443
path: /
tls:
- hosts:
- dashboard.deepthot.aa
- dashboard
secretName: dashboard-tls
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
kubectl apply -n kubernetes-dashboard -f dashboard-config.yaml
kubectl -n kubernetes-dashboard get secret -o json|jq '.items[] | select(.metadata.name | contains("admin-user")) | .data.token' -r| base64 --decode
Dynamic Storage Provider
dynamic storage config has more detail, we don’t need it here. Basically much of the synology configuration is outside of the realm I’m intending to cover in this document.
helm upgrade --install nfs-client-provisioner stable/nfs-client-provisioner \
--set nfs.server=$(getent hosts h2gt2g|awk '{print $1}') \
--set nfs.path=/volume1/k8s \
--set storageClass.defaultClass=true
gitlab
finally we get to something that does real work ;-) and another week month goes by…
I’ve been waffling, however really, namespaces are a good thing for keeping things straight, even if kubernetes thinks you should be a big group to have them. Anyway, lets configure a gitlab default namespace and context:
kubectl config set-context gitlab --namespace=gitlab --cluster=default --user=kubernetes-admin
kubectl config use-context gitlab 2831 kubectl config view
kubectl create namespace gitlab
If you’re restoring from backup hopefully you’ll have the secrets, if not the database will not spark you young lady. If you have it, before you start creating stuff do this kubectl create secret generic gitlab-rails-secret --from-file=secrets.yml=secrets.yaml the actual restore you do after the rest of the setup, but the keys have to be there from the beginning.
Create static persistent volumes
cat <<EOF >pv.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: gitlab-minio
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
server: h2gt2g
path: "/volume1/k8s/gitlab/minio"
storageClassName: standard
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: gitlab-postgresql
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
nfs:
server: h2gt2g
path: "/volume1/k8s/gitlab/postgresql"
storageClassName: standard
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: gitlab-redis
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: h2gt2g
path: "/volume1/k8s/gitlab/redis"
storageClassName: standard
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: repo-data-gitlab-gitaly-0
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
nfs:
server: h2gt2g
path: "/volume1/k8s/gitlab/gitaly"
storageClassName: standard
EOF
Configuration for deepthot gitlab
cat <<EOF >values.yaml
global:
ingress:
configureCertmanager: false
class: "nginx"
tls:
secretName: {}
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer : "ca-issuer"
hosts:
domain: deepthot.aa
externalIP: 192.168.42.64
https: false
gitlab:
name: gitlab.deepthot.aa
https: true
registry:
name: registry.deepthot.aa
https: true
minio:
name: minio.deepthot.aa
https: true
certificates:
customCAs:
- secret: deepthot-ca
appConfig:
ldap:
preventSignin: false
servers:
main:
ca_file: /etc/ssl/certs/ca-cert-deepthot-ca.pem
label: 'LDAP'
host: 'ipa.deepthot.aa'
port: 389
uid: 'uid'
bind_dn: 'uid=ldap,cn=sysaccounts,cn=etc,dc=deepthot,dc=aa'
password:
secret: deepthot-ldap-cred
key: password
base: 'cn=users,cn=accounts,dc=deepthot,dc=aa'
attributes:
username: ['uid', 'userid', 'sAMAccountName']
email: ['mail', 'email', 'userPrincipalName']
name: 'cn'
first_name: 'givenName'
last_name: 'sn'
group_base: 'ou=groups,cn=accounts,dc=deepthot,dc=aa'
admin_group: 'admins'
cron_jobs:
stuck_ci_jobs_worker:
cron: "0 * * * *"
pipeline_schedule_worker:
cron: "19 * * * *"
expire_build_artifacts_worker:
cron: "50 * * * *"
# Don't use certmanager, we'll use our own
certmanager:
install: false
nginx-ingress:
enabled: false
gitlab-runner:
install: false
certsSecretName: deepthot-aa-crt
gitlab:
unicorn:
ingress:
tls:
secretName: gitlab-gitlab-tls
gitaly:
persistence:
storageClass: standard
volumeName: repo-data-gitlab-gitaly-0
task-runner:
backups:
cron:
schedule: '@daily'
enabled: true
registry:
ingress:
tls:
secretName: gitlab-registry-tls
minio:
ingress:
tls:
secretName: gitlab-minio-tls
persistence:
storageClass: standard
volumeName: gitlab-minio
postgresql:
persistence:
storageClass: standard
volumeName: gitlab-postgresql
redis:
persistence:
storageClass: standard
volumeName: gitlab-redisEOF
LDAP configuration
kubectl create secret generic deepthot-ldap-cred --from-literal=password=youknow
kubectl create secret generic deepthot-ca --from-file=deepthot-ca.pem=/etc/ipa/ca.crt
helm repo add gitlab https://charts.gitlab.io/
helm repo update
kubectl apply -f pv.yaml
helm upgrade --install gitlab gitlab/gitlab -f values.yaml
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
I’ve tested the above configuration, it is able to be delete and recreated with the data preserved in a well-known location. Removing the chart also requires removing the gitaly claim kubectl delete pvc repo-data-gitlab-gitaly-0 and removing the pvs kubectl delete -f pv.yaml. This leaves it in a clean state. The data is still present.
Configure ingress to work with port 22
helm upgrade --install deepthot-nginx stable/nginx-ingress \
--set tcp.22=gitlab/gitlab-gitlab-shell:22
backup gitlab
The cron is part of the values.yaml file, so that’s all set up. Plus the deepthot cert was populated for ldap, so it all just kinda works now. I’m leaving this section because some of the commands are kinda interesting.
Retrieve keys to get into minio
kubectl get secrets gitlab-minio-secret -o yaml|\
awk '/accesskey|secretkey/{cmd="echo "$2" | base64 -di"; cmd|getline x;print $1,x}'
Create the backup every day. And backup the secrets manually whenever.
kubectl exec `kubectl get pods -l'release=gitlab,app=task-runner' \
-ojsonpath='{.items[0].metadata.name}'` \
-it backup-utility
kubectl get secrets gitlab-rails-secret \
-o jsonpath="{.data['secrets\.yml']}" | base64 --decode > secrets.yaml
```
Restore
```bash
kubectl delete secret gitlab-rails-secret
kubectl create secret generic gitlab-rails-secret --from-file=secrets.yml=secrets.yaml
kubectl delete pods -lapp=sidekiq,release=gitlab
kubectl delete pods -lapp=unicorn,release=gitlab
kubectl delete pods -lapp=task-runner,release=gitlab
kubectl exec $(kubectl get pods -l'release=gitlab,app=task-runner' \
-ojsonpath='{.items[0].metadata.name}') -it -- \
backup-utility --restore -t 1576802423_2019_12_20_12.5.4-ee
```
# Done!!!
OK, not quite done, but this is now where it was and kept crashing. I've got ubuntu, kubernetes, cert manager, nginx ingress, metal LB, default permanent storage, and source and binary control. Next project is the mail server and since that has a way to go we'll do a development blog until it's running then reflect it here.
No comments:
Post a Comment