kivikakk.ee

some things to make guix (system) less painful

I’m grateful to Nemin for writing about his experiences with Guix! I’ve been thinking about it for months at this stage, and have an aarch64 VM from the 1.5.0 release. It’s felt awkward and I haven’t known how to get started, particularly because of how slow things seem to be.

Some things that have really helped me speed things up:

Use Guix Moe’s substitutes

“Guix Moe”: Build farm and mirrors for community channels introduces it. You may be wondering where exactly to put the (simple-service 'guix-moe …) stanza.

Wonder no more. Your /etc/config.scm will have a (services …) declaration. There will probably be a list of (service …) declarations in a list inside that. Add it there. Mine looks like this:

[…]
;; Uncomment the line below to add an SSH server.
(service openssh-service-type)
;; Add support for the SPICE protocol, which enables dynamic
;; resizing of the guest screen resolution, clipboard
;; integration with the host, etc.
(service spice-vdagent-service-type)
;; Use the DHCP client service rather than NetworkManager.
(service dhcpcd-service-type)
(simple-service 'guix-moe guix-service-type
(guix-extension (authorized-keys (list (plain-file
"guix-moe-old.pub"
"(public-key (ecc (curve Ed25519) (q #374EC58F5F2EC0412431723AF2D527AD626B049D657B5633AAAEBC694F3E33F9#)))")
;; New key since 2025-10-29.
(plain-file
"guix-moe.pub"
"(public-key (ecc (curve Ed25519) (q #552F670D5005D7EB6ACF05284A1066E52156B51D75DE3EBD3030CD046675D543#)))")))
(substitute-urls '("https://cache-cdn.guix.moe")))))
;; Remove some services that don't make sense in a VM.
(remove (lambda (service)
(let ((type (service-kind service)))
[…]

I’ve included lots of context so you don’t get lost. This gets you some more pre-built packages. sudo guix system reconfigure /etc/config.scm.

Change your guix channel source to Codeberg

I could die and git.guix.gnu.org wouldn’t notice. Get acquainted with Guix Home. We’ll be customising the Guix channel itself per 6.2 Using a Custom Guix Channel, and doing it in Guix Home per 13.3.9 Guix Home Services.

Because we’re replacing the default, we don’t want to just add a channel. Add this service to your home-environment, similar to above:

(service home-channels-service-type
(list (channel
(name 'guix)
(url "https://codeberg.org/guix/guix.git")
(branch "master")
(introduction
(make-channel-introduction
"9edb3f66fd807b096b48283debdcddccfea34bad"
(openpgp-fingerprint
"BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))
; any other channels here
))

I pulled that introduction from https://hpc.guix.info/static/doc/complex-deployment/channels.scm. Looks safe to me!

guix home reconfigure blah.scm.

Hey, saayix has some nice software

I really wanted Ghostty, and it turns out someone’s got it packaged and up-to-date! https://codeberg.org/look/saayix/src/commit/b9b22f9ca03a452c4d8801b200da0095f54090e8/modules/saayix/packages/terminals.scm

You can add one of the channel declarations directly where indicated above. Then guix home reconfigure again (you can see the result in ~/.config/guix/channels.scm), and then guix pull to fetch the channel. Now you can e.g. #:use-module (saayix packages terminals) and add ghostty to your (packages …) list!

reminder to nix-collect-garbage

tail-end of “nix-collect-garbage” output, showing 1.5 terabytes freed🐰🐰🐰

enbi: fully operational!

Update on enbi (source) — it now monitors Pods with annotations describing which flake to build and what tag that build should produce. When it notices one failing to start due to a missing image matching the annotations, it creates a NixBuild matching the requirements, which in turn runs the build and loads it into the cluster! Successful builds clean up after themselves, though I’m leaving around the NixBuild objects themselves for now. Failing builds leave the Job/Pod in place for troubleshooting.

Updating the version of one of my apps which use my standard pattern for building Docker images with Nix is now just a matter of changing the tag in one place (e.g.); the cluster figures out building it and moving to the new release without downtime.

This has been a fun one week sojourn into writing Kubernetes operators :) The API is pretty neat, controller-runtime feels clean, and it was enjoyable discovering how many assumptions I had to unlearn while negotiating where the controller was running, where its jobs were to be scheduled, how to move data around, and the like.

nix build → nb → enbi

Still pre-alpha, but tonight I got the first complete run of a little Kubernetes controller I’ve been wanting!

Screenshot of a terminal, showing a Nix build in progress. At the top of the screen a Kubernetes CRD called “nixbuild.enbi.hrzn.ee” is visible, and at the bottom, the Nix build process can be seen producing a layered Docker image, which is then imported.

Wahoo yipee etc.! Right now we have a CRD which triggers a Nix build of a given flake URL, expected to produce a Docker or OCI image — it chooses a node which can build for the target system, spawns a Job which builds the target, and then imports it into the node’s container registry. We assume that something like Spegel is running and so any node that needs the image will pick it up.

The “hard” part (other than writing directly against the k8s API for the first time) was getting the Nix stuff to work well vis-à-vis building in a container while caching everything nicely — the flakes themselves, as well as whatever ends up in the store, as much of it will be reused between versions. Thankfully all the tooling is Cool As Fuck and it was actually really easy. We create a locally-provisioned PersistentVolume per node and stuff $HOME/.cache/nix and the Nix store in there. For now we use a chroot store, but I’d like to try an overlay store in future to avoid potentially duplicating whatever comes along in the nixos/nix image. Importing into the node’s container store is as simple as mounting the host /run and locating containerd’s socket — it differs depending on your k8s distro, and I’m developing on kind while deploying to k3s.

I still have to clean it up in this state, and have plans after this to remove the CustomResourceDefinition and trigger builds automatically when needed, getting the source details from annotations on the Deployment, but I’m happy. I don’t particularly like manually executing builds, nor do I want to stand up a registry and pre-build everything. My cluster runs on two architectures, but whether any given revision of an application will actually ever run on either, both, or any(!) of those is a matter of the particular scheduling constraints for the application and the state of the cluster at any given moment. Rather than waste energy pre-building and storing, let’s build on-demand instead! 💛🤍💜🖤

conclusion: patch OpenSSH!

John Goerzen’s Easily Using SSH with FIDO2/U2F Hardware Security Keys came up yesterday, and I thought it was a good time to fix my mess of private keys. I already own a YubiKey 5C Nano, which sits in my laptop at all times, as well as a 5C NFC, which I figured I could use hopefully with both my phone (NFC) and tablet (USB-C) for SSH when needed.

The ideal was to drop all non-SK keys, and use move to using agent forwarding exclusively when authenticating between hosts — rarely needed, but nice for some remote-to-remote scps or git+ssh pushes. (Agent forwarding is traditionally frowned upon, since someone who has or gains access to your VPS can use your socket to get your agent to auth things, but that issue is greatly reduced when user presence is verified on each use, viz. requiring you to touch your key.)

Turns out all was pretty much that easy! Just two minor hiccups:

  • Terminus on iOS supports FIDO2 keys, no payment required (despite what some search results say; looks like it was maybe a paid-only feature during beta but since not). Non-resident Ed25519 keys work very well over NFC on iPhone, but not over USB-C on iPad. The only reference I can find is this from their “ideas” page:

     Unfortunately, iPads and iPhones with USB-C cannot be compatible with OpenSSH-generate FIDO2-based keys. Please generate new FIDO2-based keys in the Termius app. These keys are supported in OpenSSH and all Termius apps.

    Upon testing, Terminus generates a non-resident ECDSA, and that works just great. So, in the end, I have three private keys: an Ed25519 for the 5C Nano, and an Ed25519 and ECDSA for the 5C NFC for use with NFC and USB-C respectively.

  • The OpenSSH bundled in macOS (at time of writing, OpenSSH_9.9p2, LibreSSL 3.3.6) doesn’t support the use of these keys. I haven’t checked whether it’s non-resident SKs specifically or what, or whether it’s the version or just a matter of what support is compiled in.

    NixOS/nix-darwin 25.05 carries an OpenSSH_10.0p2, OpenSSL 3.4.1 11 Feb 2025, and it does!

    Using agent forwarding without losing what’s left of one’s humanity implies getting your ssh-agent setup working nicely. How?

I looked into a few different ways, but opted for the simplest: patching OpenSSH (!?). The thought process is as follows:

  • /System/Library/LaunchAgents/com.openssh.ssh-agent.plist will put an SSH_AUTH_SOCKET in your environment, which launches the system-provided ssh-agent when first addressed.
  • This is a nice, macOS-y way to do things, but we don’t want to go down any rabbit hole that involves disabling SIP, so we can’t just swap the binary path (or binary itself) out.
    • Even if we could, we still need to add launchd support to the ssh-agent in Nix. Thankfully, the needed changes are small, and easy to find; just look for __APPLE_LAUNCHD__ in ssh-agent.c.
  • In my experience, disabling the system-provided launch agent doesn’t work reliably.

We apply two patches:

  1. Add the __APPLE_LAUNCHD__ bits into the Nix-provided OpenSSH.
  2. Change SSH_AUTHSOCKET_ENV_NAME to "VYX_SSH_AUTH_SOCK".

Finally, we install our own launchd user agent (modelled upon the system one, but with our binary), which puts the socket in the VYX_SSH_AUTH_SOCK env var instead. This means we don’t need to worry about the system launch agent; it’ll only get triggered/used when something calls the system ssh binary.

Head teed!