You might be aware that Ingress NGINX is retiring next month (that’s tomorrow!). I really didn’t want to bother with replacing it, but I also didn’t particularly like the sound of running something barely maintained to begin with never receiving another patch again. After a half-hearted attempt to understand why I would want to use Istio, I’ve completed a migration to Envoy Gateway and will briefly document the changes in manifest form in case anyone else wants to copy-paste.

I should probably add some distinction between dev and prod mode for next time, oops.

Here follows a short overview of the current state of the Kubernetes cluster that constitutes my infrastructure. I link to the Flux bits and pieces that install and configure them. It’s more personal documentation than anything useful for others.

$ k gns
NAME              STATUS   AGE
cert-manager      Active   105d
chog              Active   105d
default           Active   105d
enbi              Active   105d
external-dns      Active   105d
flux-system       Active   105d
furpoll           Active   105d
ingress-nginx     Active   105d
kube-node-lease   Active   105d
kube-public       Active   105d
kube-system       Active   105d
kv                Active   105d
linkding          Active   105d
miniflux          Active   105d
minio-kala        Active   105d
minio-operator    Active   105d
nossa             Active   105d
outline           Active   105d
postgres-cassax   Active   105d
shynet            Active   105d
static            Active   105d

Update on enbi (source) — it now monitors Pods with annotations describing which flake to build and what tag that build should produce. When it notices one failing to start due to a missing image matching the annotations, it creates a NixBuild matching the requirements, which in turn runs the build and loads it into the cluster! Successful builds clean up after themselves, though I’m leaving around the NixBuild objects themselves for now. Failing builds leave the Job/Pod in place for troubleshooting.

Updating the version of one of my apps which use my standard pattern for building Docker images with Nix is now just a matter of changing the tag in one place (e.g.); the cluster figures out building it and moving to the new release without downtime.

This has been a fun one week sojourn into writing Kubernetes operators :) The API is pretty neat, controller-runtime feels clean, and it was enjoyable discovering how many assumptions I had to unlearn while negotiating where the controller was running, where its jobs were to be scheduled, how to move data around, and the like.

Still pre-alpha, but tonight I got the first complete run of a little Kubernetes controller I’ve been wanting!

Wahoo yipee etc.! Right now we have a CRD which triggers a Nix build of a given flake URL, expected to produce a Docker or OCI image — it chooses a node which can build for the target system, spawns a Job which builds the target, and then imports it into the node’s container registry. We assume that something like Spegel is running and so any node that needs the image will pick it up.

The “hard” part (other than writing directly against the k8s API for the first time) was getting the Nix stuff to work well vis-à-vis building in a container while caching everything nicely — the flakes themselves, as well as whatever ends up in the store, as much of it will be reused between versions. Thankfully all the tooling is Cool As Fuck and it was actually really easy. We create a locally-provisioned PersistentVolume per node and stuff $HOME/.cache/nix and the Nix store in there. For now we use a chroot store, but I’d like to try an overlay store in future to avoid potentially duplicating whatever comes along in the nixos/nix image. Importing into the node’s container store is as simple as mounting the host /run and locating containerd’s socket — it differs depending on your k8s distro, and I’m developing on kind while deploying to k3s.

I still have to clean it up in this state, and have plans after this to remove the CustomResourceDefinition and trigger builds automatically when needed, getting the source details from annotations on the Deployment, but I’m happy. I don’t particularly like manually executing builds, nor do I want to stand up a registry and pre-build everything. My cluster runs on two architectures, but whether any given revision of an application will actually ever run on either, both, or any(!) of those is a matter of the particular scheduling constraints for the application and the state of the cluster at any given moment. Rather than waste energy pre-building and storing, let’s build on-demand instead! 💛🤍💜🖤

Today it was finally time to write a policy file for one of my Anubis instances. I use Timoni as a fairly thin wrapper over CUE to write templates for my own k8s deployments, and I found it really shone in this particular instance. I’ll just tl;dr and show the code; here’s an excerpt from my blog engine’s bundle.cue, which is the “entrypoint” for compiling its manifests:

anubis: {
  secretName: "anubis-20250816-071240"
  policy: permitPaths: [{
    name:       "permit-atom-xml"
    path_regex: "^/atom\\.xml$"
  }, {
    name:       "permit-feed-xml"
    path_regex: "^/feed\\.xml$"
  }]
}

I’m aiming to expose just a minimum of configurability first. Here’s how the schema side of that is defined in config.cue:

anubis?: {
  // Needs to already exist in the target namespace. Should have key
  // "ED25519_PRIVATE_KEY_HEX".
  secretName: string

  policy?: {
    permitPaths: *[] | [... close({
      name:       string
      path_regex: string
    })]
  }
}

I grabbed the default root bot policy file from https://github.com/TecharoHQ/anubis/blob/main/data/botPolicies.yaml, and converted it to CUE with cue import botPolicies.yaml. Then we put it in the templates package, add a way to inject our config, and use the config to expand upon the defaults:

package templates

#AnubisBotPolicies: {
  #config: #Config

  //# Anubis has the ability to let you import snippets of configuration into the main
  //# configuration file. This allows you to break up your config into smaller parts
  //# that get logically assembled into one big file.

// ...

  }, if #config.kv.anubis.policy.permitPaths != _|_ for setting in #config.kv.anubis.policy.permitPaths {
    name:       setting.name
    path_regex: setting.path_regex
    action:     "ALLOW"
  }, {

// ...

Finally, the bit I really like: creating the ConfigMap (which gets mounted as a volume) with the policy YAML:

#AnubisConfigMap: timoniv1.#ImmutableConfig & {
	Config=#config: #Config
	#Kind:          timoniv1.#ConfigMapKind
	#Meta:          #config.metadata
	#Suffix:        "-anubis-env"

	#Data: {
		"policy.yml": yaml.Marshal(#AnubisBotPolicies & {#config: Config})
	}
}

Note the careful lack of hand-written YAML at any stage! 💛🤍💜🖤