Monday, October 28, 2019

Kubernetes Dynamic Storage Provisioning

Today we’re going to set up Dynamic Storage on our kubernetes cluster. This will allow pods on my cluster to ask for storage to be mounted. I’m planning on using NFS as the backing storage.

Create the share

Go into the synology GUI and create the share. Set it up the same way other things are. Namely no squashing, R/W access by local admins and admins, docker, and media for LDAP groups. ssh into the san and change the directory mode of the just created dir to 755.

Create the persistent dynamic volume manager

I find this hard to believe, but according to the documentation all I have to do to create an NFS dynamic volume handler is just helm install stable/nfs-client-provisioner --set nfs.server=$(getent hosts h2gt2g|awk '{print $1}') --set nfs.path=/volume1/k8s

Yes, it really was that easy. What was hard was trying to figure out how to use it. Even that was easy once I stopped overthinking it.

First nfs-pvc.yaml, the persistent volume claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "nfs-client"
  resources:
    requests:
      storage: 1Mi

Next pod.yaml, a pod to use the persistent volume claim

# This mounts the nfs volume claim into /mnt and continuously
# overwrites /mnt/index.html with the time and hostname of the pod.

apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-busybox
spec:
  replicas: 2
  selector:
    name: nfs-busybox
  template:
    metadata:
      labels:
        name: nfs-busybox
    spec:
      containers:
      - image: busybox
        command:
          - sh
          - -c
          - 'while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $(($RANDOM % 5 + 5)); done'
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
          # name must match the volume name below
          - name: nfs
            mountPath: "/mnt"
      volumes:
      - name: nfs
        persistentVolumeClaim:
          claimName: nfs

And now we test

denebeim@frankiemouse:~/kubernetes/storage$ kubectl create -f nfs-pvc.yaml -f pod.yaml 
persistentvolumeclaim/nfs created
replicationcontroller/nfs-busybox created
denebeim@frankiemouse:~/kubernetes/storage$ cat foo/
default-nfs-pvc-07001ca1-5d11-42e3-9316-33ca23e79d13/ #recycle/                                             
denebeim@frankiemouse:~/kubernetes/storage$ cat foo/default-nfs-pvc-07001ca1-5d11-42e3-9316-33ca23e79d13/index.html 
Tue Oct 29 04:43:46 UTC 2019
nfs-busybox-824rh

Written with StackEdit.

Sunday, October 27, 2019

Ingress Controller for Deepthot

Ingress Controller

The first thing you need to talk to your cluster is an ingress controller. That’s what opens a connection from the world to the kubernetes network. kubectl has a proxy which is what is allowing me to connect like the dashboard to my web browser at the moment. But that’s just for control. To connect something real, like gitlab you need something else. And there’s a ton of them.

I’m reading kubernetes top ingress controllers to see if it’s any good. I’m not sure how much which ingress controller matters for me since I’m so small. I think whatever is easiest.

I hadn’t really considered this. However the article suggests you want two ingress controllers. One for LAN traffic and one for internet traffic. Good point. Right now, of course I just need internal traffic, but configuring it so it will be easy to add a second one later sounds like a good idea.

After reading the document I’m coming to the same conclusion as I did before reading the document. It’s either NGINX or Traefik. NGINX has been the reverse proxy I’ve been using for quite awhile. Setting it up here will be painless according to the article. However Traefik seems to be the new boy in town that all the cool kids are playing with. Well, I made a post on facebook asking opinions, not a bunch of people stepping up. I think I’m going to try traefik and see if I like it.

I’m going to try to do this supposedly working demo https://docs.traefik.io/user-guides/crd-acme/ it’s running out of a docker-compose. I’m going to try to wing it. Also, I’m not going to muck with https as of yet since I don’t feel like figuring out how to make this present on the internet just yet. So, I’m going to blow off the let’s encrypt stuff and all that. So, here we go.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default
denebeim@frankiemouse:~/kubernetes/traefik$ kubectl apply -f ingressroute.yaml 
customresourcedefinition.apiextensions.k8s.io/ingressroutes.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/ingressroutetcps.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/middlewares.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/tlsoptions.traefik.containo.us created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created

Whoami service demo

apiVersion: v1
kind: Service
metadata:
  name: traefik

spec:
  ports:
    - protocol: TCP
      name: web
      port: 8000
    - protocol: TCP
      name: admin
      port: 8080
    - protocol: TCP
      name: websecure
      port: 4443
  selector:
    app: traefik

---
apiVersion: v1
kind: Service
metadata:
  name: whoami

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami
denebeim@frankiemouse:~/kubernetes/traefik$ kubectl apply -f whoami.yaml 
service/traefik created
service/whoami created
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: traefik-ingress-controller

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.Address=:8000
            - --entrypoints.websecure.Address=:4443
            - --providers.kubernetescrd
            - --certificatesresolvers.default.acme.tlschallenge
            - --certificatesresolvers.default.acme.email=foo@you.com
            - --certificatesresolvers.default.acme.storage=acme.json
            # Please note that this is the staging Let's Encrypt server.
            # Once you get things working, you should remove that whole line altogether.
            - --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
            - name: web
              containerPort: 8000
            - name: websecure
              containerPort: 4443
            - name: admin
              containerPort: 8080

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80
denebeim@frankiemouse:~/kubernetes/traefik$ kubectl apply -f deploy.yaml 
serviceaccount/traefik-ingress-controller created
deployment.apps/traefik created
deployment.apps/whoami created

Well, these all came up, but we know that it’s not reachable from the internet (it better not be!) and so the let’s encrypt cert must not be available. But it looks like the container came up anyway. We’ll see if it’s working. If not we’ll try removing the TLS stuff.

Need to install a proxy I guess sudo kubectl port-forward --address 0.0.0.0 service/traefik 8000:8000 8080:8080 443:4443 -n default this doesn’t return. I’m not sure how this is supposed to be set up for real, it’s kinda important.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: simpleingressroute
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`localhost`) && PathPrefix(`/notls`)
    kind: Rule
    services:
    - name: whoami
      port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutetls
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`localhost`) && PathPrefix(`/tls`)
    kind: Rule
    services:
    - name: whoami
      port: 80
  tls:
    certResolver: default
denebeim@frankiemouse:~/kubernetes/traefik$ curl http://localhost:8000/notls
Hostname: whoami-bd6b677dc-xptqx
IP: 127.0.0.1
IP: 10.244.1.5
RemoteAddr: 10.244.1.4:54508
GET /notls HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.58.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: localhost:8000
X-Forwarded-Port: 8000
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-74dfb956c5-w45x9
X-Real-Ip: 127.0.0.1

Well, lookie there. That seems to have worked like a champ. So, I think it works ok with no tls even when tls is configured but still broken.

Traefik is pretty cool, but I still don’t know how to get from it to the actual system. The kubectl proxy thing was ok for playing, but I think you need something real to do it.

Something real, and it doesn’t take traefik

This way works.
helm install --name metallb stable/metallb

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: default
  name: metallb-config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.42.50-192.168.42.99

kubectl apply -f config.yaml
Metallb is a bare metal load balancer. Cheap, easy. Just define pools for it to divvy out from.

kubectl apply -f https://k8s.io/examples/service/load-balancer-example.yaml
denebeim@frankiemouse:~/kubernetes/metalLB$ kubectl apply -f https://k8s.io/examples/service/load-balancer-example.yaml
deployment.apps/hello-world created
kubectl expose deployment hello-world --type=LoadBalancer --name=my-service
kubectl get services my-service
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
my-service   LoadBalancer   10.101.43.9   192.168.42.50   8080:30388/TCP   23s
denebeim@frankiemouse:~/kubernetes/metalLB$ curl 192.168.42.50:8080
Hello Kubernetes!

That’s creating 5 services that just echo hello. Then exposing the deployment, which talks to the metallb and gets an address.

And I’m too exhausted now.

Written with StackEdit.

Persistent Volumes for Deepthot

Written with StackEdit.

Tuesday, October 22, 2019

Back to Basics for Another Try

Introduction

After several attempts at getting nfs and automount directories working on my server deepthot, I’ve decided to KISS. I’m beginning from the beginning and sticking to stuff I know. Get all that working, and then increase the complexity. Here we go

Steps

  1. Create two VMs one for NFS one for a Client
  2. Create an NFS server V3
  3. Connect to it with the client
  4. Set up automount with local files
  5. Upgrade the NFS to v4
  6. Install FreeIPA
  7. Kerberos the NFS
  8. Configure automount for kerberos nfs
  9. Switch NFS to NAS

Create VMs

This is really standard stuff, just create a VM using all defaults, which is 1 CPU, 1G core, 16G disk. Grab my ssh certs from github during the install. Log in, and do sudo apt update; sudo apt dist-upgrade -y. Reboot. Make sure you can nfs in.
Do the same thing for the other machine.
One odd thing I noticed: when I did the install and rebooted, I wasn’t able to see the dns entry. Earlier this morning I noticed that it showed up after I’d upgraded and then rebooted. Hopefully that will remain true. Sadly not so much. The other machine was already in the system as nfs, I haven’t deleted it and I don’t really know how to remove a machine from FreeIPA yet, so, we’ll make another machine nfs-1.
note: too damn slow. Switch to 2 CPUs.

Create an NFS server

I’ve decided to try this without going for V3 first NFSv4Howto. This, of course, uses the /exports bind for all of the exported directories. Synology doesn’t want to do that, but I think I’m going to try doing it this way since this is how I’ve gotten it to work in the past reliably. We’ll get to that when we finally bring the nfs server into it.
OK, now we set up the filesystem on the server. So, on nfs install the nfs server, create the export directory and put it in /etc/fstab
apt install -y nfs-kernel-server
mkdir -p /export/home
echo /home/    /export/home   none    bind  0  0 >>/etc/fstab
mount -a
root@nfs-1:~# ls -l /export/home
total 0
drwxr-xr-x 1 nfs-1 nfs-1 184 Oct 13 18:07 nfs-1
looks good!
Ok, set up nfs security. We don’t actually need to do anything since what we’re doing just makes the default explicit. However later we will be setting up kerberos, and for that it will need to be turned on. Here’s the command: sed -i '/SVCGSSD/s/"[^"]*"/"no"/' /etc/default/nfs-kernel-server
Next thing is to create the /etc/exports file with the filesystems we want exported. If it was a pre NFSv4 you would only have one export here. Now we have to export the root of the exports. I’ve never been really clear on what is going on here, but here we go. This isn’t idempotent, so beware.
cat <<EOF >>/etc/exports
    /export      192.168.0.0/16(rw,fsid=0,no_subtree_check,sync)
    /export/home 192.168.0.0/16(rw,nohide,insecure,no_subtree_check,sync)
EOF
BUG? uh huh, turns out 0.0.0.0/0 isn’t working as a netmask. I’m wondering if this has been my problem all along… Remember to check next time you play here. And I did check, it helped some, but it didn’t suddenly start working, continue…
NFS isn’t mapping users properly. Googling around, found linuxmint chat that gave me a clue to try this:
sudo mkdir -m 755 -p /var/lib/nfs/rpc_pipefs/nfs
sudo mkdir -p /run/rpc_pipefs/nfs
do chown statd:nogroup /var/lib/nfs/rpc_pipefs /run/rpc_pipefs
Nope, this isn’t working either. I did see some notes in the nfsV4 howto that was mentioning the id mapping wan’t working without kerberos. So, I’m going to continue on following that howto to see if this clears up. First though, taking snapshots of nfs-1 and client

Install freeipa client on the nfs server and client

Referring to howtoforge we install freeipa client on both of the vms we’re playing with today
hostnamectl set-hostname nfs-1.deepthot.aa
apt-get install freeipa-client oddjob-mkhomedir -y
ipa-client-install --mkhomedir
kinit denebeim
root@nfs-1:~# ipa host-show client.deepthot.aa
  Host name: client.deepthot.aa
  Principal name: host/client.deepthot.aa@DEEPTHOT.AA
  Principal alias: host/client.deepthot.aa@DEEPTHOT.AA
  SSH public key fingerprint: SHA256:MT1M9wL8e8tSrn+5RszB9fsbRoe7ArQpIWkLAwddeXE root@client (ssh-rsa), SHA256:LgxSoZbg9KteI0ft/SfrysWvyg/ZG9pL8ByI0clT0nk root@client (ssh-dss),
                              SHA256:GoAgmh+95/eltBZTmvOq4834fold/ReDklxAu44slqU root@client (ecdsa-sha2-nistp256), SHA256:3U+rUUTOpHzhHi1LqrJHOLb4XnFsOBs7+OJZXK07acM root@client (ssh-ed25519)
  Password: False
  Keytab: False
  Managed by: client.deepthot.aa
So, installing the actual nfs on ipa stuff I’m referring again to using-ipa-to-provide-automount-maps-for-nfsv4-home-directories
floundering still sadly. I think I found the problem. On the NFS server /etc/idmap.conf was missing Domain=deepthot.aa from the [General] section. I added that and everything started working. It was present on the client already. I assume it’s important for there too. I’m not sure that’s going to help when I get back to using the synology for nfs since that configuration is correct there.
I’m tired of screwing around with synology. I’ve found something that works. Namely:
ipa automountkey-mod default auto.nethome --key=\* --info='-fstype=nfs4,rw,sec=sys,soft,rsize=8192,wsize=8192 h2gt2g.deepthot.aa:/volume1/homes/&'
This is nfs4 without using kerberos. The idmapping seems to be working great. This means the synology is able to do kerberos, but no idea what’s going on. It would be nice to see exactly what messages are being sent and so on, but to be honest I don’t want to screw with wireshark at the moment, and that is probably what it would take.
Here is the commands to create the autofs tables
ipa automountlocation-add phoenix
ipa automountmap-add-indirect phoenix --parentmap=auto.master --mount=/nethome auto.nethome
ipa automountkey-add phoenix auto.nethome --key=* --info="-fstype=nfs4,rw,sec=sys,soft,rsize=8192,wsize=8192,no_root_squash h2gt2g.deepthot.aa:/volume1/homes/&"

Adding a new user to the system

Configuration for adding a new user

I’m setting this up so that home directories are automounted. I don’t believe that there’s an easy way to get mkhomedir configure this. What I want to do is make a home directory on the NAS and initialize it from /etc/skel like most programs like adduser and useradd do. To do this I’m going to create an automount entry to mount the root home directories on the nas /volume1/homes to the homes directory in root’s home directory.
ipa automountkey-add phoenix auto.direct --key=/root/homes --info='-fstype=nfs4,rw,sec=sys,soft,rsize=8192,wsize=8192,no_root_squash h2gt2g.deepthot.aa:/volume1/homes'
systemctl restart autofs

Adding a new user

Actually adding the user would be, creating the user in ipa, then creating the home directory. This should be able to be done from the root account on any client.
echo password|ipa user-add --first='Fnord' --last='Ublech'  --password --all fnord
cp -a /etc/skel homes/fnord
chown -R fnord@deepthot.aa: homes/fnord
So, I think we’re done for the moment with NFS. Seems to be working fine.

This configuration works

On the synology
/volume1/homes *(rw,async,no_wdelay,crossmnt,insecure,no_root_squash,insecure_locks,sec=sys:krb5:krb5i:krb5p,anonuid=1025,anongid=100)
/volume1/video *(rw,sync,no_wdelay,crossmnt,insecure,no_root_squash,insecure_locks,sec=sys:krb5:krb5i:krb5p,anonuid=1025,anongid=100)
/volume1/public *(rw,async,no_wdelay,crossmnt,insecure,no_root_squash,insecure_locks,sec=sys:krb5:krb5i:krb5p,anonuid=1025,anongid=100)
/volume1/music *(rw,async,no_wdelay,crossmnt,insecure,no_root_squash,insecure_locks,sec=sys:krb5:krb5i:krb5p,anonuid=1025,anongid=100)
/volume1/photo *(rw,async,no_wdelay,crossmnt,insecure,no_root_squash,insecure_locks,sec=sys:krb5:krb5i:krb5p,anonuid=1025,anongid=100)root@h2gt2g:/etc# 
total 876
drwxr-xr-x 16 root  root                4096 Oct 21 19:20 .
drwxr-xr-x 23 root  root                4096 Oct 20 09:12 ..
drwxr-xr-x 10 root  root                4096 Oct 20 17:39 homes
drwxrwsr-x  4 root  media@deepthot.aa   4096 Oct 22 18:50 music
drwxrwsr-x  3 root  media@deepthot.aa   4096 Sep  2 22:58 photo
drwxrwsr-x  3 root  media@deepthot.aa   4096 Oct 22 18:44 public
drwxrwsr-x  4 root  media@deepthot.aa   4096 Oct 22 18:34 video
That configuration is setting everything to have no mucking with the user by nfs. And making the directories group writable and sticky. So, anyone belonging to the media group will be able to write, all files created will be owned by the media group. I don’t know if this will work from windows. I may have to install nfs on it.
Written with StackEdit.

Sunday, October 20, 2019

FreeIPA Sudo

FreeIPA Sudo

Turns out to be trivial.
  • Go to policy->sudo->sudo_rules
  • Click on add enter a name and click on add and edit
  • Select ‘any’ on all the radio buttons, except ‘who’.
  • On the ‘who’ button select the specified group to be admins
Easy peasy, done.
Written with StackEdit.

How Simple Does It Have to Be Before It’s Mine?

 Hi, Joy here, I'm tired.  I'm too tired to polish this.  However bare with me, the text below is *trying* to be me.  It's not l...