Thursday, November 21, 2019

Gitlab Kubernetes

Current Configuration

Note: Helm 2.16.0 is broken don’t use it. That was the snap as of 11/21/19 emphasized text
This is a total win. I’ve got the cert manager working great. Internal and external. This works with the helm chart as shipped to deploy/upgrade simply:

helm repo add gitlab https://charts.gitlab.io/
helm repo update
helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab

Here’s the current configure:

# values-minikube.yaml  
# This example intended as baseline to use Minikube for the deployment of GitLab  
# - Services that are not compatible with how Minikube runs are disabled  
# - Configured to use 192.168.99.100, and nip.io for the domain  
  
# Minimal settings  
global:  
  ingress:  
    configureCertmanager: false  
    class: "nginx"  
  tls:  
      secretName: {}  
  annotations:  
    kubernetes.io/ingress.class: "nginx"  
    cert-manager.io/cluster-issuer : "ca-issuer"  
  
  hosts:  
    domain: deepthot.aa  
    externalIP: 192.168.42.64  
    https: false  
    gitlab:  
      name: gitlab.deepthot.aa  
      https: true  
    registry:  
      name: registry.deepthot.aa  
      https: true  
    minio:  
      name: minio.deepthot.aa  
      https: true  
  
# Don't use certmanager, we'll use our own  
certmanager:  
  install: false  
  
nginx-ingress:  
  enabled: false  
  
gitlab-runner:  
  install: false  
  certsSecretName: deepthot-aa-crt  
  
gitlab:  
  unicorn:  
    ingress:  
      tls:  
        secretName: gitlab-gitlab-tls  
  
registry:  
  ingress:  
    tls:  
      secretName: gitlab-registry-tls  
  
minio:  
  ingress:  
    tls:  
      secretName: gitlab-minio-tls

Old configuration

Well, yet again I was making things too hard, ish. Where I am right now is I’ve set up a values.yaml file

# values-minikube.yaml  
# This example intended as baseline to use Minikube for the deployment of GitLab  
# - Services that are not compatible with how Minikube runs are disabled  
# - Configured to use 192.168.99.100, and nip.io for the domain  
  
# Minimal settings  
global:  
  ingress:  
    configureCertmanager: false  
    class: "nginx"  
  hosts:  
    domain: deepthot.aa  
    externalIP: 192.168.42.64  
    https: false  
    gitlab:  
      name: gitlab.deepthot.aa  
      https: false  
    registry:  
      name: registry.deepthot.aa  
      https: false  
    minio:  
      name: minio.deepthot.aa  
      https: false  
    hosts:  
# Don't use certmanager, we'll self-sign  
certmanager:  
  install: false  
# Use the `ingress` addon, not our Ingress (can't map 22/80/443)  
nginx-ingress:  
  enabled: true  
# GitLab Runner isn't a big fan of self-signed certificates  
gitlab-runner:  
  install: false  

Then I type helm upgrade --install -f values-deepthot.yaml --namespace gitlab gitlab . However I had to muck with the helm chart to work with the kubernetes I’m running here 1.16.2. The thing is they deleted several deprecated things and all of the helm charts I’ve played with have not been updated. I’m going to downgrade kubernetes and then get back to here.

Right now it is configured for a self-signed cert. That makes gitlab runner not function. So I need to generate a cert and get the deepthot.aa cert into it. I’ll do that after kubernetes downgrading though.

Downgrading was really easy. Just follow kubeadm-upgrade/ I downgraded the worker first, dunno if it was necessary, but all went clean. Now to try the helmchart from the repo again.

I think it’s working as well as it ever did. Now though it needs to have a real cert.
Wildcard_certificates that’s very problematical. OK. New plan, move my lan onto a delegated subdomain of deepthot.org. That’s a big production. I was planning on doing this anyway, but for the time being I really don’t need all that many certs. So, let’s do them by hand.

#!/bin/bash

dn=$(hostname -d)
me=$(hostname -f)
ns="gitlab"
ip="192.168.42.64"

kubectl create namespace "${ns}"
for host in gitlab registry minio; do
 ipa dnsrecord-add "${dn}" $host --a-rec "${ip}"
 ipa host-add $host."${dn}"
 ipa service-add HTTP/$host."${dn}"
 ipa service-add-host --hosts="${me}" HTTP/$host."${dn}"
 sudo ipa-getcert request -r -f $host."${dn}".crt -k $host."${dn}".key -N CN=$host."${dn}" -D $host."${dn}" -K HTTP/$host."${dn}" -w
 sudo chown denebeim: $host."${dn}".crt $host."${dn}".key
 kubectl create secret tls gitlab-$host-tls --cert=./$host."${dn}".crt --key=./$host."${dn}".key --namespace="${ns}"
 cat /etc/ipa/ca.crt >>"${host}"."${dn}".crt
done
kubectl --namespace ${ns} create secret generic deepthot-aa-crt --from-file={,{gitlab,minio,registry}.}deepthot.aa.crt
helm upgrade --install -f values.yaml --namespace "${ns}" "${ns}" gitlab/gitlab 

Next step is to get gitlab runner up. It is not liking the custom CA that we have here.
gitlab-runner/issues/3968 describes the problem. There were a few workarounds, I tried figuring out the root cause, then I tried the solutions. The documentation says I should be concatenating the chain of certs.

I did this, replaced all the certs, didn’t help. Finally I looked at the stored cert in kubernetes. It was encrypted, but you know, it looked like the length was about the length of the key before it was concatenated. Also, I noticed the logs that came out when it was sucking the key in was saying ‘Updating local copy of SSL certificate “gitlab/gitlab-gitlab-tls” with missing intermediate CA certs’ Interesting. So I reread the bug report and one of the comments mentioned working around the problem by creating a generic secret. As you can see above the secrets we were building were tls secrets. So, I tried making a generic secret from the gitlab one we used to make the tls secret earlier. That worked. This is what you do:

cat /etc/ipa/ca.crt >>gitlab.deepthot.aa.crt
kubectl \
  --namespace gitlab \
  create secret generic gitlab-deepthot-aa-crt \
  --from-file=gitlab.deepthot.aa.crt
helm upgrade --install -f values.yaml --namespace gitlab gitlab gitlab/gitlab \
	--set gitlab-runner.install=true \
	--set gitlab-runner.certsSecretName=gitlab-deepthot-aa-crt

The first script has been modified to include concatenating /etc/ipa/ca.crt onto all of the certs that it is making. However
there’s still issues. Basically reading on the internet someone mentioned how problematical it is to use your own CA. Every container on the planet has it’s own copy of SSL stuff. You can’t put your CA into all of them. I ran into the same sort of trouble at work. There I just rebuilt everything and put our CA into the base container. The solution is to get us on a public DNS we control and get let’s encrypt working. But that’s a pretty big thing, and I guess all this stuff can wait until then.

I guess that means I’m done with gitlab for the moment. I’ll call this a win.

No comments:

Post a Comment

How Simple Does It Have to Be Before It’s Mine?

 Hi, Joy here, I'm tired.  I'm too tired to polish this.  However bare with me, the text below is *trying* to be me.  It's not l...