cluster overview
I should probably add some distinction between dev and prod mode for next time, oops.
Here follows a short overview of the current state of the Kubernetes cluster that constitutes my infrastructure. I link to the Flux bits and pieces that install and configure them. It’s more personal documentation than anything useful for others.
$ k gns
NAME STATUS AGE
cert-manager Active 105d
chog Active 105d
default Active 105d
enbi Active 105d
external-dns Active 105d
flux-system Active 105d
furpoll Active 105d
ingress-nginx Active 105d
kube-node-lease Active 105d
kube-public Active 105d
kube-system Active 105d
kv Active 105d
linkding Active 105d
miniflux Active 105d
minio-kala Active 105d
minio-operator Active 105d
nossa Active 105d
outline Active 105d
postgres-cassax Active 105d
shynet Active 105d
static Active 105d
cert-manager
$ k cert-manager ga
NAME READY STATUS RESTARTS AGE
pod/cert-manager-58dd99f969-knhn9 1/1 Running 1 (46d ago) 105d
pod/cert-manager-cainjector-55cd9f77b5-jlqwh 1/1 Running 1 (46d ago) 105d
pod/cert-manager-webhook-7987476d56-xfvn5 1/1 Running 1 (46d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.90.202.51 <none> 9402/TCP 105d
service/cert-manager-cainjector ClusterIP 10.90.246.184 <none> 9402/TCP 105d
service/cert-manager-webhook ClusterIP 10.90.71.86 <none> 443/TCP,9402/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 1/1 1 1 105d
deployment.apps/cert-manager-cainjector 1/1 1 1 105d
deployment.apps/cert-manager-webhook 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-58dd99f969 1 1 1 105d
replicaset.apps/cert-manager-cainjector-55cd9f77b5 1 1 1 105d
replicaset.apps/cert-manager-webhook-7987476d56 1 1 1 105d
Standard install of cert-manager.
This works really reliably. A couple weeks ago I had all my certs expire at once — it turned out a bug in my DNS provider caused all my domains to have an invalid CAA set, so none of them were renewing.
chog
$ k chog ga
NAME READY STATUS RESTARTS AGE
pod/chog-deploy-8697ddd585-7p5rz 2/2 Running 3 (46d ago) 49d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/chog-deploy 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/chog-deploy-64bbcf5b65 0 0 0 85d
replicaset.apps/chog-deploy-6bd86f7689 0 0 0 105d
replicaset.apps/chog-deploy-8697ddd585 1 1 1 49d
A little Discord bot which takes care of some sundries for me and Annie.
- Install via kustomize
- CUE-based setup, build with enbi
default
$ k default ga
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.90.0.1 <none> 443/TCP 105d
enbi
$ k enbi ga
NAME READY STATUS RESTARTS AGE
pod/enbi-controller-manager-54c5d7fdb5-xxj7l 1/1 Running 0 14d
pod/podwatcher-647d55454f-647d55454f-5nw2f 0/1 Completed 0 15h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/enbi-controller-manager-metrics-service ClusterIP 10.90.179.225 <none> 8443/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/enbi-controller-manager 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/enbi-controller-manager-54c5d7fdb5 1 1 1 14d
replicaset.apps/enbi-controller-manager-649b47b448 0 0 0 49d
replicaset.apps/enbi-controller-manager-6b65b4fdfb 0 0 0 49d
NAME STATUS COMPLETIONS DURATION AGE
job.batch/podwatcher-647d55454f-647d55454f Complete 1/1 108s 15h
NAME IMAGE TAG SYSTEM STATUS BUILD NODE AGE
nixbuild.enbi.hrzn.ee/podwatcher-5f9d56bc9f nossa.ee/talya/kv:211c28182fcb5ae15cd102388fa823e890acdd42 x86_64-linux Succeeded kala 15h
nixbuild.enbi.hrzn.ee/podwatcher-647d55454f nossa.ee/talya/kv:dc48721dd93d0a0d305453c9c68b50d18b3ca022 x86_64-linux Failed kala 15h
nixbuild.enbi.hrzn.ee/podwatcher-79c6b5548b nossa.ee/talya/nossa:6181112bcc45ee79482b619f0a6e838dcb3be43b x86_64-linux Succeeded kala 14d
enbi, Operator that builds images on demand.
- Install via kustomize
- CUE-based setup, build with itself
- kubebuilder manifests for CRD, policies, etc.
external-dns
$ k external-dns ga
NAME READY STATUS RESTARTS AGE
pod/external-dns-58f666b7f9-7gww2 2/2 Running 0 15d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/external-dns ClusterIP 10.90.161.171 <none> 7979/TCP,8080/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/external-dns 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/external-dns-58f666b7f9 1 1 1 15d
replicaset.apps/external-dns-5cf8f9499b 0 0 0 15d
replicaset.apps/external-dns-6fdcd965dd 0 0 0 15d
replicaset.apps/external-dns-7d9bc99bbb 0 0 0 15d
Standard install of ExternalDNS, with fork of external-dns-bunny-webhook for the heavy lifting.
ExternalDNS itself works reliably, the Bunny.net provider has needed so much work. It works OK for me now (after quite a few head-scratchers), but I wouldn’t recommend it to you.
flux-system
$ k flux-system ga
NAME READY STATUS RESTARTS AGE
pod/flux-operator-5f966f8fbc-6dqgg 1/1 Running 4 (15d ago) 105d
pod/helm-controller-5cbb6bd454-5pb8n 1/1 Running 4 (15d ago) 105d
pod/kustomize-controller-757dff9666-ck9hm 1/1 Running 4 (15d ago) 105d
pod/notification-controller-55d7f99bf9-r855t 1/1 Running 4 (15d ago) 105d
pod/source-controller-88749965c-lht44 1/1 Running 4 (15d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/flux-operator ClusterIP 10.90.248.179 <none> 8080/TCP 105d
service/notification-controller ClusterIP 10.90.245.193 <none> 80/TCP 105d
service/source-controller ClusterIP 10.90.113.157 <none> 80/TCP 105d
service/webhook-receiver ClusterIP 10.90.4.185 <none> 80/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/flux-operator 1/1 1 1 105d
deployment.apps/helm-controller 1/1 1 1 105d
deployment.apps/kustomize-controller 1/1 1 1 105d
deployment.apps/notification-controller 1/1 1 1 105d
deployment.apps/source-controller 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/flux-operator-5f966f8fbc 1 1 1 105d
replicaset.apps/helm-controller-5cbb6bd454 1 1 1 105d
replicaset.apps/kustomize-controller-757dff9666 1 1 1 105d
replicaset.apps/notification-controller-55d7f99bf9 1 1 1 105d
replicaset.apps/source-controller-88749965c 1 1 1 105d
Standard install of Flux. It works the way I want it to, and I’ve never had problems prying it apart or debugging it when I’ve done questionable things.
furpoll
$ k furpoll ga
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/furpoll 07 7,10,13,18,21,23 * * * Australia/Melbourne False 0 48m 105d
Very simple private message checker. This keeps getting into failing loops because of hostIP shenanigans and OpenSMTPD not listening on the mesh interface since it hasn’t come up yet.
- Install via kustomize
- CUE-based setup for cronjob, build with enbi
ingress-nginx
$ k ingress-nginx ga
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-controller-67cc5cc697-rv4hg 1/1 Running 0 42d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.90.99.47 51.161.136.132 80:32080/TCP,443:32443/TCP 105d
service/ingress-nginx-controller-admission ClusterIP 10.90.28.90 <none> 443/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-67cc5cc697 1 1 1 42d
replicaset.apps/ingress-nginx-controller-778b7dfb8f 0 0 0 105d
Standard install of Ingress NGINX. Now retired, hah. :/
- Install via Helm, kustomize
- Configured with annotations throughout
kube-node-lease
$ k kube-node-lease ga
No resources found in kube-node-lease namespace.
kube-public
$ k kube-public ga
No resources found in kube-public namespace.
kube-system
$ k kube-system ga
NAME READY STATUS RESTARTS AGE
pod/coredns-6d668d687-pncqs 1/1 Running 0 15d
pod/helm-install-flux-operator-42sn6 0/1 Completed 0 15d
pod/local-path-provisioner-869c44bfbd-7pg5b 1/1 Running 0 15d
pod/metrics-server-7bfffcd44-kczwl 1/1 Running 1 (46d ago) 50d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.90.0.10 <none> 53/UDP,53/TCP,9153/TCP 105d
service/kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 105d
service/metrics-server ClusterIP 10.90.91.59 <none> 443/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 1/1 1 1 105d
deployment.apps/local-path-provisioner 1/1 1 1 105d
deployment.apps/metrics-server 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-6d668d687 1 1 1 15d
replicaset.apps/local-path-provisioner-869c44bfbd 1 1 1 15d
replicaset.apps/metrics-server-7bfffcd44 1 1 1 50d
NAME STATUS COMPLETIONS DURATION AGE
job.batch/helm-install-flux-operator Complete 1/1 16s 15d
kv
$ k kv ga
NAME READY STATUS RESTARTS AGE
pod/kv-dcbf57c54-msngk 3/3 Running 0 15h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kv ClusterIP 10.90.216.102 <none> 80/TCP,81/TCP,9090/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kv 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/kv-6f5c6d7fc6 0 0 0 15h
replicaset.apps/kv-bf8bd6978 0 0 0 15h
replicaset.apps/kv-dcbf57c54 1 1 1 15h
- Install via kustomize
- CUE-based setup, build with enbi
linkding
$ k linkding ga
NAME READY STATUS RESTARTS AGE
pod/linkding-6b55479478-4j8bx 1/1 Running 1 (46d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/linkding ClusterIP 10.90.133.1 <none> 9090/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/linkding 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/linkding-5666f9b576 0 0 0 105d
replicaset.apps/linkding-6b55479478 1 1 1 105d
replicaset.apps/linkding-7954846cb4 0 0 0 105d
linkding is a nice little self-hosted bookmark manager. I wrote some CUE to configure it in k8s more nicely.
miniflux
$ k miniflux ga
NAME READY STATUS RESTARTS AGE
pod/miniflux-b76bbb9-2hw8v 1/1 Running 0 46d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/miniflux ClusterIP 10.90.105.109 <none> 8080/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/miniflux 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/miniflux-b76bbb9 1 1 1 105d
Miniflux is a self-hosted RSS reader. As above.
minio-kala and `minio-operator
$ k minio-kala ga
NAME READY STATUS RESTARTS AGE
pod/myminio-pool-kala-0 2/2 Running 2 (46d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/minio ClusterIP 10.90.127.99 <none> 443/TCP 105d
service/myminio-console ClusterIP 10.90.149.242 <none> 9443/TCP 105d
service/myminio-hl ClusterIP None <none> 9000/TCP 105d
NAME READY AGE
statefulset.apps/myminio-pool-kala 1/1 105d
$ k minio-operator ga
NAME READY STATUS RESTARTS AGE
pod/minio-operator-58fb9bb48-k6ddz 1/1 Running 1 (46d ago) 105d
pod/minio-operator-58fb9bb48-qjwns 0/1 Pending 0 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/operator ClusterIP 10.90.85.65 <none> 4221/TCP 105d
service/sts ClusterIP 10.90.196.22 <none> 4223/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/minio-operator 1/2 2 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/minio-operator-58fb9bb48 2 2 1 105d
MinIO is — was? — an S3-compatible object store. MinIO Operator deploys that into Kubernetes automatically, setting up clustering etc.
I’m probably going to try replacing it with Garage soon.
nossa
$ k nossa ga
NAME READY STATUS RESTARTS AGE
pod/nossa-67dc84b64d-wrnpj 3/3 Running 0 12d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nossa ClusterIP 10.90.193.94 <none> 80/TCP,81/TCP,9090/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nossa 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/nossa-67dc84b64d 1 1 1 12d
nóssa is my code store! Now more than ever, we must not rely on corporations to host our source.
- Install via kustomize
- CUE-based setup, build with enbi
outline
$ k outline ga
NAME READY STATUS RESTARTS AGE
pod/outline-864ffd9f49-frb8j 1/1 Running 1 (46d ago) 49d
pod/outline-redis-0 1/1 Running 1 (46d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/outline ClusterIP 10.90.11.37 <none> 3000/TCP 105d
service/outline-redis ClusterIP None <none> 6379/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/outline 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/outline-668fb7849d 0 0 0 105d
replicaset.apps/outline-864ffd9f49 1 1 1 49d
NAME READY AGE
statefulset.apps/outline-redis 1/1 105d
Outline is a self-hostable knowledge base/wiki. I don’t like it very much, but the solution you have is better than the one you don’t.
- Install via kustomize
- CUE-based setup, uses a fork I build manually
postgres-cassax
$ k postgres-cassax ga
NAME READY STATUS RESTARTS AGE
pod/postgres-0 1/1 Running 1 (46d ago) 105d
pod/postgres-operator-75d58485b9-p6nxk 1/1 Running 1 (46d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgres ClusterIP 10.90.104.198 <none> 5432/TCP 105d
service/postgres-config ClusterIP None <none> <none> 105d
service/postgres-operator ClusterIP 10.90.130.173 <none> 8080/TCP 105d
service/postgres-repl ClusterIP 10.90.47.202 <none> 5432/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/postgres-operator 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/postgres-operator-75d58485b9 1 1 1 105d
replicaset.apps/postgres-operator-85d7544db4 0 0 0 105d
NAME READY AGE
statefulset.apps/postgres 1/1 105d
NAME IMAGE CLUSTER-LABEL SERVICE-ACCOUNT MIN-INSTANCES AGE
operatorconfiguration.acid.zalan.do/postgres-operator-configuration ghcr.io/zalando/spilo-17:4.0-p2 cassax postgres-pod -1 105d
NAME TEAM VERSION PODS VOLUME CPU-REQUEST MEMORY-REQUEST AGE STATUS
postgresql.acid.zalan.do/postgres cassax 17 1 10Gi 100m 100Mi 105d Running
Zalando’s Postgres Operator just work™s? I’m surprised but so far I’ve had no issues, even when using replication. (I particularly like that you can have it drop generated secrets into target namespaces.)
shynet
$ k shynet ga
NAME READY STATUS RESTARTS AGE
pod/shynet-celeryworker-7c85c7c7b7-4k95g 1/1 Running 1 (46d ago) 105d
pod/shynet-redis-0 1/1 Running 1 (46d ago) 105d
pod/shynet-webserver-6897985f7f-6zdv5 1/1 Running 2 (46d ago) 105d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/shynet-redis ClusterIP None <none> 6379/TCP 105d
service/shynet-webserver-service ClusterIP 10.90.167.165 <none> 8080/TCP 105d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/shynet-celeryworker 1/1 1 1 105d
deployment.apps/shynet-webserver 1/1 1 1 105d
NAME DESIRED CURRENT READY AGE
replicaset.apps/shynet-celeryworker-7c85c7c7b7 1 1 1 105d
replicaset.apps/shynet-celeryworker-cc8bdf8d5 0 0 0 105d
replicaset.apps/shynet-webserver-6897985f7f 1 1 1 105d
replicaset.apps/shynet-webserver-7bf96ff48c 0 0 0 105d
NAME READY AGE
statefulset.apps/shynet-redis 1/1 105d
Shynet is a “privacy-friendly” self-hosted web analytics service. It gives me a very basic idea of how much traffic my sites get.
I’d de-duplicate the Redises between Outline and Shynet one day, or, I’ll just not ¯\_(ツ)_/¯
static
$ k static ga
No resources found in static namespace.
This namespace only exists for the Flux kustomization, oops.
$ k % gi -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
kv kv nginx kivikakk.ee 51.161.136.132 80, 443 105d
linkding linkding nginx tracks.hrzn.ee 51.161.136.132 80, 443 105d
miniflux miniflux nginx rss.hrzn.ee 51.161.136.132 80, 443 105d
minio-kala s3.hrzn.ee nginx s3.hrzn.ee,s3-console.hrzn.ee 51.161.136.132 80, 443 105d
minio-kala static-comrak.ee nginx comrak.ee 51.161.136.132 80, 443 105d
minio-kala static-eka.kivikakk.ee nginx eka.kivikakk.ee 51.161.136.132 80, 443 105d
minio-kala static-f.hrzn.ee nginx f.hrzn.ee 51.161.136.132 80, 443 105d
minio-kala static-hrzn.ee nginx hrzn.ee 51.161.136.132 80, 443 105d
minio-kala static-kindnessalliance.love nginx kindnessalliance.love 51.161.136.132 80, 443 105d
minio-kala static-kinu.ee nginx kinu.ee 51.161.136.132 80, 443 105d
minio-kala static-lottia.net nginx lottia.net 51.161.136.132 80, 443 105d
nossa nossa nginx nossa.ee 51.161.136.132 80, 443 105d
outline outline nginx adc.hrzn.ee 51.161.136.132 80, 443 105d
shynet shynet-webserver-ingress nginx shynet.hrzn.ee 51.161.136.132 80, 443 105d
The kustomization configures all the ingresses.
