Still pre-alpha, but tonight I got the first complete run of a little Kubernetes controller I’ve been wanting!
Wahoo yipee etc.! Right now we have a CRD which triggers a Nix build of a given flake URL, expected to produce a Docker or OCI image — it chooses a node which can build for the target system, spawns a Job which builds the target, and then imports it into the node’s container registry. We assume that something like Spegel is running and so any node that needs the image will pick it up.
The “hard” part (other than writing directly against the k8s API for the first time) was getting the Nix stuff to work well vis-à-vis building in a container while caching everything nicely — the flakes themselves, as well as whatever ends up in the store, as much of it will be reused between versions. Thankfully all the tooling is Cool As Fuck and it was actually really easy. We create a locally-provisioned PersistentVolume per node and stuff $HOME/.cache/nix and the Nix store in there. For now we use a chroot store, but I’d like to try an overlay store in future to avoid potentially duplicating whatever comes along in the nixos/nix image. Importing into the node’s container store is as simple as mounting the host /run and locating containerd’s socket — it differs depending on your k8s distro, and I’m developing on kind while deploying to k3s.
I still have to clean it up in this state, and have plans after this to remove the CustomResourceDefinition and trigger builds automatically when needed, getting the source details from annotations on the Deployment, but I’m happy. I don’t particularly like manually executing builds, nor do I want to stand up a registry and pre-build everything. My cluster runs on two architectures, but whether any given revision of an application will actually ever run on either, both, or any(!) of those is a matter of the particular scheduling constraints for the application and the state of the cluster at any given moment. Rather than waste energy pre-building and storing, let’s build on-demand instead! 💛🤍💜🖤
