Saturday, December 21, 2019

01 Deepthot Configuration

Deepthot Configuration

This is a detailed document discussing the current configuration

Ubuntu 18.04 Installs

Here are the steps for a basic install of an ubuntu VM. Every VM on deepthot has this exact configuration.

  1. Create the VM, use the disk image on the SAN
  2. Install the VM
    1. btrfs
    2. package repository http://nexus:8082/repository/ubuntu/
    3. install ssh
    4. lookup on gihub denebeim
    5. allow non ssh key logins
  3. log in from the console
  4. sudo apt update;sudo apt dist-upgrade -y (ssh is broken until you upgrade)
  5. reboot
  6. ssh in as the device specific user
  7. sudo apt install emacs-nox freeipa-client autofs nfs-kernel-server docker.io -y
  8. echo automount: files sss|sudo tee -a /etc/nsswitch.conf
  9. sudo hostnamectl set-hostname $(hostname -s).deepthot.aa
  10. sudo ipa-client-install
  11. sudo kinit
  12. sudo ipa service-add nfs/$(hostname -f)
  13. sudo ipa-getkeytab -s ipa.deepthot.aa -p nfs/$(hostname -f) -k /etc/krb5.keytab
  14. sudo klist -k /etc/krb5.keytab (to check and make sure, should have host and nfs keys)
  15. sudo ipa-client-automount
  16. sudo gpasswd -a k8s docker
  17. sudo gpasswd -a denebeim@deepthot.aa docker
  18. reboot machine and ssh in. Make sure nfs and docker are working
    NOTE: the main gotcha here is the docker group must be local. That means you need to install docker before freeipa.

Docker configuration

docker is configured to use btrfs at this point. That’s how I want it to be, however the kubernetes version we’re installing at the moment does not support btrfs docker layers. So, the next instructions will end up destroying the docker things that are running and switching it to overlay2. So don’t run anything if you’re setting up a kubernetes cluster.

sudo tee /etc/docker/daemon.json <<EOF
{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
	"max-size": "100m"
    },
    "storage-driver": "btrfs",
    "insecure-registries" : ["artifactory.deepthot.aa:8081","artifactory:8081"],
    "registry-mirrors": ["http://artifactory.deepthot.aa:8081/"]
}
EOF
sudo systemctl enable docker
sudo systemctl restart docker

I’ve been having some grief with docker.io, here is how to set up the repository for docker-ce and used it

curl -s http://nexus:8082/repository/apt-docker/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository 'deb http://nexus:8082/repository/apt-docker/linux/ubuntu bionic stable'
sudo apt install -y docker-ce

Kubernetes Cluster

Currently: 1 control node, 1 worker node

Type CPU Core Disk
Control 8 4 16
Worker 4 16 32
Worker-2 4 16 32

Control Node Configuration

Create the DNS record:

kinit denebeim
ipa dnsrecord-add deepthot.aa cluster-endpoint --a-ip-address=$(hostname -i)

We’re currently using dns as a load balancer. I’m not thinking this is working too good though. This may change to either
a non redundant control, or put an actual vm here to be the load balancer

Install the software:

sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s http://nexus:8082/repository/apt-kubernetes/doc/apt-key.gpg | sudo apt-key add -
add-apt-repository \
	'deb http://nexus:8082/repository/apt-kubernetes/ kubernetes-xenial main'
apt install {kubeadm,kubelet,kubectl}=1.15.7-00 -y --allow-downgrades -y
apt-mark hold kubectl kubelet kubeadm

Create configuration

The current kubernetes installer version needs configuration.

key=$(kubeadm alpha certs certificate-key)
cat <<EOF >kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
certificateKey: "$key"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.7
controlPlaneEndpoint: "cluster-endpoint:6443"
EOF

Configure docker daemon

sudo systemctl stop docker
sudo sed -i "/storage-driver/s/btrfs/overlay2/" /etc/docker/daemon.json
sudo rm -rf /var/lib/docker
sudo systemctl start docker

NOTE: After you stop docker, make sure there isn’t any mounts or anything left around. The rm -rf might fail.

And build the cluster controller

kubeadm init --config=kubeadm-config.yaml --upload-certs

copy the config file where you want to use it from:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Finally set up the network.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

I chose weave because that’s what they mentioned in the kubernetes.io install document.

Worker Node Configuration

Install the Software

apt-get update && apt-get install -y apt-transport-https curl
curl -s http://nexus:8082/repository/apt-kubernetes/doc/apt-key.gpg | sudo apt-key add -
add-apt-repository \
	'deb http://nexus:8082/repository/apt-kubernetes/ kubernetes-xenial main'
apt install {kubeadm,kubelet,kubectl}=1.15.7-00 -y --allow-downgrades -y
apt-mark hold kubectl kubelet kubeadm

Configure docker daemon

sudo systemctl stop docker
sudo sed -i "/storage-driver/s/btrfs/overlay2/" /etc/docker/daemon.json
sudo rm -rf /var/lib/docker
sudo systemctl start docker
sudo systemctl enable docker

NOTE: After you stop docker, make sure there isn’t any mounts or anything left around. The rm -rf might fail.

Start up the worker

sudo $(ssh k8s-control kubeadm token create --print-join-command)

Infrastructure

Now we get the infrastructure up that is:

  • helm
  • metallb
  • certmanager
  • dynamic storage provisioning
  • ingress

helm

We’re using helm 3, so no configuration needed. (other than getting helm 3)

metallb

cat <<EOF >values.yaml
configInline:
  address-pools:
  - name: deepthot.aa
    protocol: layer2
    addresses:
      - 192.168.42.64/26
EOF
kubectl create namespace deepthot-system
helm upgrade --install  metallb stable/metallb -f values.yaml

certmanager

Here’s a script to create an apiKey for an artifactory user

cat <<'EOF' >apikey.sh
#!/bin/bash  
  
# Fetch or create apikey for a user  
  
cred=$1  
if [ "$cred" == "-d" ]; then  
  delete=1  
  cred=$2  
fi  
art="http://artifactory:8081/artifactory"  
  
if [ "$delete" != "" ]; then  
  curl -u "$cred" -X DELETE $art/api/security/apiKey  
  exit  
fi  
apiKey=$(curl -u "$cred" -s -X GET $art/api/security/apiKey | jq -r .apiKey)  
if [ "$apiKey" == "null" ]; then  
  apiKey=$(curl -u "$cred" -s -X POST $art/api/security/apiKey | jq -r .apiKey);  
fi  
echo "$apiKey"
EOF
chmod +x apikey.sh

Get the helm repository for cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

(I was trying to get an artifactory virtual repository for helm working here, but no go.)

kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
helm upgrade \
--install cert-manager \
--version v0.12.0 \
jetstack/cert-manager

There’s several bits about getting the deepthot.aa signing keys in cert-manager we won’t go into that here since I’ve got the keys on my disk. There’s also some stuff about let’s encrypt. I’m just going to do the last part here since everything is already set up.

cd ~/kubenetes/certconfig
kubectl create secret tls ca-key-pair --cert=ca.crt --key=ca.key
kubectl apply -f ca-issuer.yaml
kubectl apply -f letsencrypt-staging.yaml
kubectl apply -f letsencrypt-production.yaml

(no ingress controller at this point so we’re limited as to what we can do.)

ingress

cat <<EOF >values.yaml
rbac:
  create: true

tcp:
# gitlab ssh ingress.
#  22: "gitlab/gitlab-gitlab-shell:22"

# email ingress
#  25: "mailu/mailu-front:25"
#  587: "mailu/mailu-front:587"
#  2525: "mailu/mailu-dovecot:2525"
#  10025: "mailu/mailu-front:10025"
#  143: "mailu/mailu-front:143"
#  993: "mailu/mailu-front:993"
#  110: "mailu/mailu-front:110"
#  995: "mailu/mailu-front:995"
EOF
helm upgrade --install deepthot-nginx stable/nginx-ingress -f values.yaml

Somehow my dnsrecord got missing. Unclear how that happened, but…

ipa dnsrecord-add deepthot.aa @ --a-ip-address=192.168.42.64 --a-create-reverse

If you want to get logging from the useful container in this:

kubectl -n default logs -l 'app=nginx-ingress,component=controller' -f

testing

I looked at how-to-set-up-an-nginx-ingress and ran through it, everything works up to, but not including letsencrypt and I’m not really ready to do external access yet.

dashboard

cat <<EOF >dashboard-config.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: ca-issuer
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/secure-backends: "true"
  name: dashboard
spec:
  rules:
  - host: dashboard.deepthot.aa
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 443
        path: /
  - host: dashboard
    http:
      paths:
        - backend:
            serviceName: kubernetes-dashboard
            servicePort: 443
          path: /
  tls:
  - hosts:
    - dashboard.deepthot.aa
    - dashboard
    secretName: dashboard-tls
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
kubectl apply -n kubernetes-dashboard -f dashboard-config.yaml
kubectl -n kubernetes-dashboard get secret -o json|jq '.items[] | select(.metadata.name | contains("admin-user")) | .data.token' -r| base64 --decode

Dynamic Storage Provider

dynamic storage config has more detail, we don’t need it here. Basically much of the synology configuration is outside of the realm I’m intending to cover in this document.

helm upgrade --install nfs-client-provisioner stable/nfs-client-provisioner \
--set nfs.server=$(getent hosts h2gt2g|awk '{print $1}') \
--set nfs.path=/volume1/k8s \
--set storageClass.defaultClass=true

gitlab

finally we get to something that does real work ;-) and another week month goes by…

I’ve been waffling, however really, namespaces are a good thing for keeping things straight, even if kubernetes thinks you should be a big group to have them. Anyway, lets configure a gitlab default namespace and context:

kubectl config set-context gitlab --namespace=gitlab --cluster=default --user=kubernetes-admin  
kubectl config use-context gitlab 2831  kubectl config view
kubectl create namespace gitlab

If you’re restoring from backup hopefully you’ll have the secrets, if not the database will not spark you young lady. If you have it, before you start creating stuff do this kubectl create secret generic gitlab-rails-secret --from-file=secrets.yml=secrets.yaml the actual restore you do after the rest of the setup, but the keys have to be there from the beginning.

Create static persistent volumes

cat <<EOF >pv.yaml
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: gitlab-minio  
spec:  
  capacity:  
    storage: 10Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/minio"  
  storageClassName: standard  
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: gitlab-postgresql  
spec:  
  capacity:  
    storage: 8Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/postgresql"  
  storageClassName: standard  
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: gitlab-redis  
spec:  
  capacity:  
    storage: 5Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/redis"  
  storageClassName: standard  
---  
apiVersion: v1  
kind: PersistentVolume  
metadata:  
  name: repo-data-gitlab-gitaly-0  
spec:  
  capacity:  
    storage: 50Gi  
  accessModes:  
    - ReadWriteOnce  
  nfs:  
    server: h2gt2g  
    path: "/volume1/k8s/gitlab/gitaly"  
  storageClassName: standard
EOF

Configuration for deepthot gitlab

cat <<EOF >values.yaml
global:  
  ingress:  
    configureCertmanager: false  
    class: "nginx"  
  tls:  
      secretName: {}  
    annotations:  
      kubernetes.io/ingress.class: "nginx"  
  cert-manager.io/cluster-issuer : "ca-issuer"  
  hosts:  
    domain: deepthot.aa  
    externalIP: 192.168.42.64  
    https: false  
    gitlab:  
      name: gitlab.deepthot.aa  
      https: true  
    registry:  
      name: registry.deepthot.aa  
      https: true  
    minio:  
      name: minio.deepthot.aa  
      https: true  
  
  certificates:  
    customCAs:  
      - secret: deepthot-ca  
  appConfig:  
    ldap:  
      preventSignin: false  
      servers:  
        main:  
          ca_file: /etc/ssl/certs/ca-cert-deepthot-ca.pem  
          label: 'LDAP'  
  host: 'ipa.deepthot.aa'  
  port: 389  
          uid: 'uid'  
  bind_dn: 'uid=ldap,cn=sysaccounts,cn=etc,dc=deepthot,dc=aa'  
  password:  
            secret: deepthot-ldap-cred  
            key: password  
          base: 'cn=users,cn=accounts,dc=deepthot,dc=aa'  
  attributes:  
            username: ['uid', 'userid', 'sAMAccountName']  
            email:    ['mail', 'email', 'userPrincipalName']  
            name: 'cn'  
  first_name: 'givenName'  
  last_name:  'sn'  
  group_base: 'ou=groups,cn=accounts,dc=deepthot,dc=aa'  
  admin_group: 'admins'  
  cron_jobs:  
      stuck_ci_jobs_worker:  
        cron: "0 * * * *"  
  pipeline_schedule_worker:  
        cron: "19 * * * *"  
  expire_build_artifacts_worker:  
        cron: "50 * * * *"  
  
# Don't use certmanager, we'll use our own  
certmanager:  
  install: false  
  
nginx-ingress:  
  enabled: false  
  
gitlab-runner:  
  install: false  
  certsSecretName: deepthot-aa-crt  
  
gitlab:  
  unicorn:  
    ingress:  
      tls:  
        secretName: gitlab-gitlab-tls  
  gitaly:  
    persistence:  
      storageClass: standard  
      volumeName: repo-data-gitlab-gitaly-0  
  task-runner:  
    backups:  
      cron:  
        schedule: '@daily'  
  enabled: true  
  
registry:  
  ingress:  
    tls:  
      secretName: gitlab-registry-tls  
  
minio:  
  ingress:  
    tls:  
      secretName: gitlab-minio-tls  
  persistence:  
    storageClass: standard  
    volumeName: gitlab-minio  
  
postgresql:  
  persistence:  
    storageClass: standard  
    volumeName: gitlab-postgresql  
  
redis:  
  persistence:  
    storageClass: standard  
    volumeName: gitlab-redisEOF

LDAP configuration

kubectl create secret generic deepthot-ldap-cred --from-literal=password=youknow
kubectl create secret generic deepthot-ca --from-file=deepthot-ca.pem=/etc/ipa/ca.crt
helm repo add gitlab https://charts.gitlab.io/
helm repo update
kubectl apply -f pv.yaml
helm upgrade --install gitlab gitlab/gitlab -f values.yaml
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo

I’ve tested the above configuration, it is able to be delete and recreated with the data preserved in a well-known location. Removing the chart also requires removing the gitaly claim kubectl delete pvc repo-data-gitlab-gitaly-0 and removing the pvs kubectl delete -f pv.yaml. This leaves it in a clean state. The data is still present.

Configure ingress to work with port 22

helm upgrade --install deepthot-nginx stable/nginx-ingress \
--set tcp.22=gitlab/gitlab-gitlab-shell:22

backup gitlab

The cron is part of the values.yaml file, so that’s all set up. Plus the deepthot cert was populated for ldap, so it all just kinda works now. I’m leaving this section because some of the commands are kinda interesting.

Retrieve keys to get into minio

kubectl get secrets gitlab-minio-secret -o yaml|\
awk '/accesskey|secretkey/{cmd="echo "$2" | base64 -di"; cmd|getline x;print $1,x}'

Create the backup every day. And backup the secrets manually whenever.

kubectl exec `kubectl get pods -l'release=gitlab,app=task-runner' \
-ojsonpath='{.items[0].metadata.name}'` \
-it backup-utility
kubectl get secrets gitlab-rails-secret \
-o jsonpath="{.data['secrets\.yml']}" | base64 --decode > secrets.yaml
```

Restore
```bash
kubectl delete secret gitlab-rails-secret
kubectl create secret generic gitlab-rails-secret --from-file=secrets.yml=secrets.yaml
kubectl delete pods -lapp=sidekiq,release=gitlab
kubectl delete pods -lapp=unicorn,release=gitlab
kubectl delete pods -lapp=task-runner,release=gitlab
kubectl exec $(kubectl get pods -l'release=gitlab,app=task-runner' \
-ojsonpath='{.items[0].metadata.name}') -it -- \
backup-utility --restore -t 1576802423_2019_12_20_12.5.4-ee
```
# Done!!!
OK, not quite done, but this is now where it was and kept crashing.  I've got ubuntu, kubernetes, cert manager, nginx ingress, metal LB, default permanent storage, and source and binary control.   Next project is the mail server and since that has a way to go we'll do a development blog until it's running then reflect it here.

2019-12-21 yay

Everything kinda came together today! Right in time for my vacation.

  • LDAP on gitlab working
  • Backup/Restore on gitlab working (sorta, it’s storing to the s3 bucket that is part of the gitlab install)
  • Everything is (or will be) documented
  • Oh, extra bonus, I have the router set up so it will take over if deepthot goes down. It is just doing dhcp and passing the dns resolution on to google. That’s enough for the system to continue working if heartogold goes down.

Sunday, December 15, 2019

2019-12-14 infrastructure

Today I’m planning on getting all of the infrastructure up that is:

  • helm
  • metallb
  • certmanager
  • dynamic storage provisioning
  • ingress
    I think the plan is to figure out how to use known mounts as well. I want to not lose everything. So while I’m setting it up I either have to figure out how to make repeatable requests with the dynamic storage or give it a fixed name with autofs.

I was just reading in the kubernetes doc that there really isn’t a need for namespaces in a single user cluster. So, I’m gonna stick everything in default for now, if that becomes an issue I’ll revisit it then.

I’m thinking I want to add helm repositories to my artifactory. I’m also thinking I want to automate repository creation, assuming I can with this artifactory version. Well, I can’t fetch the repository to update it without a bunch of grief, so I’ll make the remote repos and then let the virtual repository be configured by hand.

And, sadly the virtual repository doesn’t seem to work. That’s really annoying.

Friday, December 13, 2019

2019-12-13

Today my plans are to get the kubernetes cluster up. Get metallb, dynamic storage, certmanager, ingress, and hopefully gitlab.
I’m planning on doing this and creating 01 Deepthot Configuration with the current state without a bunch of notes and tangents in it. This is intended to be a document I’ll be able to go to at any time to get the server back to it’s current state. As such it will have the steps in it to create the current state.

I got distracted by organizing my terry pratchett books on google play music. So, I didn’t get as far as I’d like. What I do have though is bare metal up to kubernetes deepthot configuration written and the vms are there as well. So I’m gonna call it good.

Thursday, December 12, 2019

Creating a ubuntu 18.04 vm on deepthot

Steps to install ubuntu on deepthot

Creating an ubuntu Bionic Beaver (18.04) vm here at deepthot is really easy. I wanted to write down all of the
steps here so I’ve got them.

  1. Create the machine on the esx server
  2. Remember to point the cd drive to the iso image on the san
  3. Right click on the VM and select console->launch console
  4. Default through everything until you get to the disk
  5. On the disk, select manual
  6. Create a partition on / formatted as btrfs
  7. Select install openssh server
  8. import ssh identity from github, denebeim
  9. allow password authentication over ssh
  10. Create a user that’s specific to the machine
  11. Finish install/reboot
  12. log in as that specific user
  13. sudo apt update;sudo apt dist-upgrade -y (ssh is broken until you upgrade)
  14. yawn, this is slow I wish I had artifactory
  15. reboot
  16. sudo systemd-resolve --flush-caches (on the laptop)
  17. ssh in as the device specific user
  18. sudo apt install emacs-nox freeipa-client autofs nfs-kernel-server docker.io -y
  19. echo automount: files sss|sudo tee -a /etc/nsswitch.conf
  20. sudo hostnamectl set-hostname $(hostname -s).deepthot.aa
  21. sudo ipa-client-install
  22. sudo kinit
  23. sudo ipa service-add nfs/$(hostname -f)
  24. sudo ipa-getkeytab -s ipa.deepthot.aa -p nfs/$(hostname -f) -k /etc/krb5.keytab
  25. sudo klist -k /etc/krb5.keytab (to check and make sure, should have host and nfs keys)
  26. sudo ipa-client-automount
  27. ls -lA /nethome/denebeim (to make sure automount is doing its bit)
  28. sudo gpasswd -a k8s docker
  29. sudo gpasswd -a denebeim@deepthot.aa docker
  30. exit back and ssh in again as myself. Now I’m sparked.

notes: the sudos after the kinit are because /etc/krb5.keytab is only root writable. kinit gives you root for kerberos/ipa stuff. Or rather it gives you whatever role you’re initing as. Cool really.

Ok, you now have a client configured. It should really take you about 5 minutes or so if you’re not making a blog post.

Written with StackEdit.

Sunday, December 8, 2019

Create System Account with FreeIPA

Creating a system account with FreeIPA

This would be suitable for like an ldap verification. It has almost 0 privilege. LDAP should die in fire IMO.

ldapmodify -x -D 'cn=Directory Manager' -y password <<EOF
dn: uid=ldap,cn=sysaccounts,cn=etc,dc=deepthot,dc=aa
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: ldap
userPassword: $PASSWORD
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

EOF

The -y password is referring to a file that contains the entire password (don’t put a lineend in the file). $PASSWORD is the literal password.
If you wanted to give it more permissions you could add a memberOf to a group that has more permissions or whatever.

Create SSL certificates with FreeIPA

Creating SSL Certs with FreeIPA

This is how you create a host based SSL cert in FreeIPA. You’d use it for something like a web server. This will be
automatically renewed before it expires. NOTE: make sure hostname returns the fqdn, before doing this. If it doesn’t use hostnamectl to set it that way.

sudo -i
hostnamectl set-hostname $(hostname -f)
kinit denebeim
ipa service-add HTTP/$(hostname -f)
ipa service-add-host --hosts=$(hostname -f) HTTP/$(hostname -f)
mkdir -p /etc/pki/tls/{certs,private}
ipa-getcert request -r -w \
-f "/etc/pki/tls/certs/$(hostname -f)".crt \
-k "/etc/pki/tls/private/$(hostname -f)".key \
-N CN="$(hostname -f)" -D "$(hostname -f)" -K HTTP/"$(hostname -f)"

I chose to put it in /etc/pki/tls because the other place you might put it /etc/ssl is full of symbolic links and stuff and mostly CA certs in ubuntu.

2019-12-08 2 k3s

Play with k3s

OK, why do I keep getting sucked into rancher? Oy… Well, I really do like it, it makes things so theoretically easy. Maybe if I just get good at it it will become second nature. Anyway, here is their light weight kubernetes implementation. Let’s see what it does.
I guess i should run it on the rancher node.
Nope, not a good idea, rancher is dead. And of course it was made before I figured out how to keep docker from dying on reboot. Grr… fix that.
OK, I’ve lost rancher, oh well. This time I want to have rancher signed by the ipa cert that I’m using everywhere. How do I do this?

sudo -i
hostnamectl set-hostname $(hostname -f)
kinit denebeim
ipa service-add HTTP/$(hostname -f)
ipa service-add-host --hosts=$(hostname -f) HTTP/$(hostname -f)
mkdir -p /etc/pki/tls/{certs,private}
ipa-getcert request -r -w \
-f "/etc/pki/tls/certs/$(hostname -f)".crt \
-k "/etc/pki/tls/private/$(hostname -f)".key \
-N CN="$(hostname -f)" -D "$(hostname -f)" -K HTTP/"$(hostname -f)"

I cannot believe how many hours that took. the hostnamectl did not stick or I forgot to do it when I did the ipa-client-install. Normally that doesn’t do anything, however it screwed me over with the above. So kids, make sure the default returned from hostname is the fqdn, it matters.
Oh, this was the clue that showed me what to fix kinit -kt /etc/krb5.keytab

So, we finally start rancher with this:

docker run -d --restart=unless-stopped \
	-p 80:80 -p 443:443 \
	-v /etc/pki/tls/certs/$(hostname -f).crt:/etc/rancher/ssl/cert.pem \
	-v /etc/pki/tls/private/$(hostname -f).key:/etc/rancher/ssl/key.pem \
	rancher/rancher:latest \
  --no-cacerts

I want to use freeipa for logging in. I’ve been poking around, I got it working last time, but of course I didn’t write it down… huh, maybe I did. nope. this time, I’m going to write it down.

ldapmodify -x -D 'cn=Directory Manager' -y password <<EOF
dn: uid=ldap,cn=sysaccounts,cn=etc,dc=deepthot,dc=aa
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: ldap
userPassword: $PASSWORD
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

EOF

$PASSWORD doesn’t do what it should do, I thought <<EOF expanded, but it doesn’t seem to be working for me today.

This is something that works: ldapdelete -D 'uid=admin,cn=users,cn=accounts,dc=deepthot,dc=aa' -y password 'uid=ldap,cn=sysaccounts,cn=etc,dc=deepthot,dc=aa' that admin account is the root account.

This is what I set up in rancher:

Users General Group
Server: ipa.deepthot.aa:636 Object Class: person Search Base: dc=deepthot,dc=aa
TLS: Yes Login Attribute: uid Object Class: groupofnames
Server Connection Timeout: 5000 Username Attribute: displayName Name Attribute: cn
Search Attribute: uid|sn|display Search Attribute: cn
Status Attribute:
Disabled BitMask: 0

Hah, so I didn’t even get back to k3s at all, silly girl. I will write these up as blog entries though

2019-12-08 Charmed Kubernetes

Charmed Kubernetes

I’m investigating Charmed Kubernetes. Canonical’s version. Maybe a short detour.
juju is an ubuntu specific bare metal install system. I’ve never tried it before so I guess I’ll do it now. sudo snap install juju --classic Create a new intellij project charmed. Make it into a git repo. This is the README.md, or would be if stackedit supported raw git repos. Gitlab isn’t up.
According to ubuntu’s kubernetes install manual the first thing I need to do is create an override.yaml file with a bunch of snippets starting with the network snippet. I chose this one
Since I’m not using anyone’s cloud, that appears to be the only thing I need to do except juju deploy charmed-kubernetes --overlay deepthot-overlay.yaml nope:

Since Juju 2 is being run for the first time, downloading latest cloud information.
Fetching latest public cloud list...
This client's list of public clouds is up to date, see `juju clouds --client-only`.
ERROR No controllers registered.

Please either create a new controller using "juju bootstrap" or connect to
another controller that you have been given access to using "juju register".
Clouds
aws
aws-china
aws-gov
azure
azure-china
cloudsigma
google
joyent
localhost
oracle
oracle-classic
rackspace

Select a cloud [localhost]:     

Enter a name for the Controller [localhost-localhost]: 

ERROR Permission denied, are you in the lxd group?

Please configure LXD by running:
	$ newgrp lxd
	$ lxd init
sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:    
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, ceph, dir, lvm, zfs) [default=btrfs]: 
Would you like to create a new btrfs subvolume under /var/snap/lxd/common/lxd? (yes/no) [default=yes]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like LXD to be available over the network? (yes/no) [default=no]: yes
Address to bind LXD to (not including port) [default=all]: 
Port to bind LXD to [default=8443]: 
Trust password for new clients: 
Again: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]
denebeim@frankiemouse:~$ sudo gpasswd -a denebeim lxd
Adding user denebeim to group lxd
denebeim@frankiemouse:~$ sudo gpasswd -a denebeim@deepthot.aa lxd
Adding user denebeim@deepthot.aa to group lxd

Heck with it, don’t waste your time Joy. The free esx hypervisor has a read only rest api. Which means NOTHING can be used to create VMs. You must do it yourself. So, no shortcuts. None… never

Friday, December 6, 2019

Certmanager

Certmanager Configuration for Deep Thought

I’m working from the official docs for cert-manager

Installing

Here’s the basic install from the web site:

# Install the CustomResourceDefinition resources separately
kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml

# Create the namespace for cert-manager
kubectl create namespace cert-manager

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm upgrade \
--install cert-manager \
--namespace cert-manager \
--version v0.12.0 \
jetstack/cert-manager

Looks good:

denebeim@frankiemouse:~/kubernetes$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-55fff7f85f-r8q4s 1/1 Running 0 3m27s
cert-manager-cainjector-54c4796c5d-85cq9 1/1 Running 0 3m27s
cert-manager-webhook-77ccf5c8b4-m2v99 1/1 Running 1 3m27s

And let’s try a basic self signed cert:

# Create a ClusterIssuer to test the webhook works okay
cat <<EOF > test-resources.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager-test
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: test-selfsigned
namespace: cert-manager-test
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: selfsigned-cert
namespace: cert-manager-test
spec:
commonName: example.com
secretName: selfsigned-cert-tls
issuerRef:
name: test-selfsigned
EOF

# Create the test resources
kubectl apply -f test-resources.yaml

# Check the status of the newly created certificate
# You may need to wait a few seconds before cert-manager processes the
# certificate request
kubectl describe certificate -n cert-manager-test
...
Spec:
Common Name: example.com
Issuer Ref:
Name: test-selfsigned
Secret Name: selfsigned-cert-tls
Status:
Conditions:
Last Transition Time: 2019-01-29T17:34:30Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2019-04-29T17:34:29Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CertIssued 4s cert-manager Certificate issued successfully

# Clean up the test resources
kubectl delete -f test-resources.yaml

I love it when things just work.

Configuring for Deepthot

I’m wanting to set this up to generate let’s encrypt for deepthot.org domains and it’s own CA for deepthot.aa domains. Since I don’t have a hole through the firewall to the internet yet for http we’ll do the local CA first

deepthot.aa CA signed certs

First step is to create a CA pair.

ssh over to the ipa server and retrieve the ipa server’s signing keys.

  1. sudo grep "internal=" /var/lib/pki/pki-tomcat/conf/password.conf | awk -F= '{print $2}' >pass.txt
  2. sudo pki-server ca-clone-prepare --pkcs12-file ca-certs.p12 --pkcs12-password-file pass.txt
  3. pki pkcs12-cert-find --pkcs12-file ca-certs.p12 --pkcs12-password-file pass.txt
  4. openssl pkcs12 -in ca-certs.p12 -out keys.txt -password file:pass.txt
  5. openssl rsa -in ca-enc.key -out ca.key -passin file:pass.txt

This is not quite right sadly. As far as I can tell you still can’t get the key trivially. You can get the cert after step 3 by typing sudo pki pkcs12-cert-export "caSigningCert cert-pki-ca" --pkcs12-file ca-certs.p12 --pkcs12-password $(cat pass.txt) --cert-file ca.crt But I haven’t found a key that way. Instead you have to edit the keys.txt file from step 4 and find the CA cert and key. Save those in ca.crt and ca-enc.key respectively. Finally you use step 5 to decode the key.
About that step 4, it will ask you for a password for each key twice. I just catted out pass.txt and pasted it in over, and over, and…

Once you have the ca crt and key you need to import that into the key that cert manager wants. Switch from the ipa machine back to something with kubectl installed and:

kubectl create secret tls ca-key-pair --cert=ca.crt --key=ca.key --namespace=cert-manager

Creating an ingress controller. This is an issuer:

apiVersion: cert-manager.io/v1alpha2  
kind: ClusterIssuer  
metadata:  
  name: ca-issuer  
spec:  
  ca:  
    secretName: ca-key-pair

Which you apply with kubectl apply -f ca-issuer.yaml

To use it you need to specify cert-manager.io/cluster-issuer : "ca-issuer" in your ingress definition, like this:

apiVersion: networking.k8s.io/v1beta1  
kind: Ingress  
metadata:  
  name: kuard  
  annotations:  
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer : "ca-issuer"  
spec:  
  tls:  
  - hosts:  
    - example.deepthot.aa  
    secretName: quickstart-example-tls  
  rules:  
  - host: example.deepthot.aa  
    http:  
      paths:  
      - path: /  
        backend:  
          serviceName: kuard  
          servicePort: 80

lets encrypt

First step is to open ports to the ingress box. ssh into the router and type the following commands. If it acts funky type ‘^V’ to disable the autocomplete.

/ip firewall nat add chain=dstnat action=dst-nat to-addresses=192.168.42.64 to-ports=80 protocol=tcp dst-address=174.70.12.249 in-interface=wan dst-port=80
/ip firewall nat add chain=dstnat action=dst-nat to-addresses=192.168.42.64 to-ports=443 protocol=tcp dst-address=174.70.12.249 in-interface=wan dst-port=443

So… my gitlab fucked up the repository. I can’t delete it either. Very annoying. Not sure what to do about it. Well, onward. Setting up let’s encrypt.

OK, here’s the deal, the problem, and it is an old friend, is that trying to route to my external IP doesn’t really work. I’ve always had issues with this, it has to do with the kernel being unable to send a modified packet back to the same interface it came in on. Or something like that. Regardless, the easiest way to fix that is, since I own my dns server, make it authoritative for deepthot.org and give deepthot.org the address of where my ingress is.

ipa dnszone-add --skip-overlap-check deepthot.org
ipa dnsrecord-add --a-ip-address=192.168.42.64 deepthot.org @
sudo systemd-resolve --flush-caches

That last bit was for good measure.

We’ve defined two let’s encrypt cert managers, one for staging and another for production. There’s two servers out there, the production one is very limited bandwidth wise, you can only do like 5 a week for a given address, if you exceed that you’re screwed. So when you’re trying to debug it you very much want to use the staging one.

First the staging:

apiVersion: cert-manager.io/v1alpha2  
kind: Issuer  
metadata:  
  name: letsencrypt-staging  
spec:  
  acme:  
    # The ACME server URL  
  server: https://acme-staging-v02.api.letsencrypt.org/directory  
    # Email address used for ACME registration  
  email: denebeim@gmail.com  
    # Name of a secret used to store the ACME account private key  
  privateKeySecretRef:  
      name: letsencrypt-staging  
    # Enable the HTTP-01 challenge provider  
  solvers:  
      - http01:  
          ingress:  
            class:  nginx

And now the production:

apiVersion: cert-manager.io/v1alpha2  
kind: Issuer  
metadata:  
  name: letsencrypt-prod  
spec:  
  acme:  
    # The ACME server URL  
  server: https://acme-v02.api.letsencrypt.org/directory  
    # Email address used for ACME registration  
  email: denebeim@gmail.com  
    # Name of a secret used to store the ACME account private key  
  privateKeySecretRef:  
      name: letsencrypt-prod  
    # Enable the HTTP-01 challenge provider  
  solvers:  
      - http01:  
          ingress:  
            class: nginx

Wednesday, December 4, 2019

Deepthot Artifactory Configuration

JFrog Container Registry Step 1

https://jfrog.com/container-registry/
I barely got it up before, so there’s nothing really lost. Eventually I want to tie it into freeipa and have it running in kubernetes and stuff. Right now though I’m going to install it on the only docker VM I have working right now, the plex machine. And I want all of the data stored on the san. I guess in the k8s share.

ssh -XA plex
sudo -i
docker pull docker.bintray.io/jfrog/artifactory-jcr:latest
ssh admin@h2gt2g
sudo -i
cd /volume1
mkdir artifactory
chmod oug+rwx artifactory
chown 1030:1030 artifactory
# the nginx user ID is 104; the group ID is 107 was mentioned in the jfrog configuration document for the nginx 
# However if you run it without nginx it's 1030:1030 instead of what they said
exit
# back on plex
echo homes -fstype=nfs4,rw,sec=sys,soft,rsize=8192,wsize=8192,no_root_squash h2gt2g.deepthot.aa:/volume1/k8s/& >/etc/auto.k8s
echo /k8s auto.k8s >>/etc/auto.master
systemctl restart autofs

Next start artifactory. This BTW will not run docker, there is no reverse proxy, which is OK for right now since I’m only after debian repository configuration at the moment. I just learned from Mark that artifactory now by default uses the first part of the docker repository as the repository, so no wildcards required if you name stuff right. For instance docker.bintray.io/jfrog/artifactory-jcr would now pull from the jfrog repository located on the docker.bintray.io remote repository. Or something like that. I’ll find out when I get there.
Running artifactory

docker run --name artifactory-jcr -v /k8s/artifactory:/var/opt/jfrog/artifactory -p 80:8081 docker.bintray.io/jfrog/artifactory-jcr:latest
# after it starts stop it and fix permissions since we know what they are now
chown -Rc 1030:1030 /k8s/artifactory
chmod og-w /k8s/artifactory

Create repositories:

curl -uadmin:password -X PATCH "http://artifactory:8081/artifactory/api/system/configuration" -H “Content-Type:application/yaml” -T - <<EOF
remoteRepositories:
  ubuntu:
    type: generic
    unusedArtifactsCleanupPeriodHours: 336
    url: http://us.archive.ubuntu.com/ubuntu
  ubuntu-docker: null
  docker-download:
    type: generic
    unusedArtifactsCleanupPeriodHours: 336
    url: https://download.docker.com/
  google-packages:
    type: generic
    unusedArtifactsCleanupPeriodHours: 336
    url: https://packages.cloud.google.com/
  docker-hub:
    type: docker
    unusedArtifactsCleanupPeriodHours: 336
    url: https://registry-1.docker.io/
    repoLayout: simple-default
    enableTokenAuthentication: true
virtualRepositories:
  docker:
    type: docker
    repositories:
    - docker-hub
security:
  anonAccessEnabled: true
EOF

On any machine you want to use the new cache:
sudo sed -i.bak 's~/.*archive.ubuntu.com~/artifactory/artifactory~;'\ 's~https://download.docker.com/linux/ubuntu~http://artifactory/artifactory/ubuntu-docker~' /etc/apt/sources.list

You can log into the docker repo if you add:

    "insecure-registries" : ["artifactory.deepthot.aa:8081","artifactory:8081"]

to /etc/docker/daemon.json

Kubernetes worker node on Deepthot

How to set up a kubernetes worker node on deepthot

Hopefully this is going to be easy peasy since I already have kubernetes on deepthot

So, I created a VM 4 core 8 gig 16 gig. Installed ubuntu 18.04 btrfs. Joined to deepthot.aa. Docker as outlined in the abovelinked blog post. Name k8s-worker-1. Wow, that was really easy. I got my config working flawlessly.

Tried kubeadm and kubectl as snaps. No go. I did notice that docker was still running on btrfs, boo. Restarted docker. I need to install docker-ce, kubeadm, kubectl, and kubelet per the instructions in yesterday’s blog.

If you don’t have a local artifactory set up

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
add-apt-repository -n \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
add-apt-repository \
	"deb http://apt.kubernetes.io/ kubernetes-xenial main"

If you do have a local artifactory set up

curl -fsSL http://artifactory:8081/artifactory/docker-download/linux/ubuntu/gpg | apt-key add -
curl http://artifactory:8081/artifactory/google-packages/apt/doc/apt-key.gpg | apt-key add -
add-apt-repository -n \
  "deb [arch=amd64] http://artifactory:8081/artifactory/docker-download/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
add-apt-repository \
	"deb http://artifactory:8081/artifactory/google-packages/apt kubernetes-xenial main"
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common -y
apt full-upgrade docker-ce=18.06.2~ce~3-0~ubuntu {kubectl,kubelet,kubeadm}=1.15.5-00 kubernetes-cni -y --allow-downgrades -y
apt-mark hold docker-ce kubectl kubelet kubeadm
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker

On the control node type kubeadm token create --print-join-command copy the output of the command and paste it into the worker node.

sudo kubeadm join cluster-endpoint:6443 --token blah --discovery-token-ca-cert-hash blah blah
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

And woot, tiller is on, so yay
Easy peasy

Written with StackEdit.

How Simple Does It Have to Be Before It’s Mine?

 Hi, Joy here, I'm tired.  I'm too tired to polish this.  However bare with me, the text below is *trying* to be me.  It's not l...