La herramienta del día es nftrace. I couldn’t work out why some pods weren’t able to communicate with each other across the Tailscale mesh. Suspected ACLs, suspected routes weren’t getting installed correctly (p.s. ip route show table 52 (!?)), suspected local firewalls, suspected so much. tcpdump only gets you so far.

Finally, on the target node:

$ doas -s
# nix shell nixpkgs#nftrace nixpkgs#nftables
# nftrace add ip daddr 10.59.1.213
# nftrace monitor

Try the request that isn’t making it through a bunch of times until you can isolate the exact sequence. ^C, nftrace remove, and read carefully:

trace id daac839a inet nftrace-table nftrace-chain packet: iif "tailscale0" ip saddr
    100.67.157.26 ip daddr 10.59.1.213 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 32261
    ip protocol tcp ip length 60 tcp sport 33233 tcp dport 9090 tcp flags == syn tcp
    window 64480
trace id daac839a inet nftrace-table nftrace-chain rule ip daddr 10.59.1.213 meta nftrace
    set 1 (verdict continue)
trace id daac839a inet nftrace-table nftrace-chain policy accept
trace id daac839a ip filter FORWARD packet: iif "tailscale0" oif "cni0" ip saddr 100.67.157.26
    ip daddr 10.59.1.213 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 32261 ip length 60 tcp
    sport 33233 tcp dport 9090 tcp flags == syn tcp window 64480
trace id daac839a ip filter FORWARD rule  counter packets 44827 bytes 28768164 jump
    KUBE-ROUTER-FORWARD (verdict jump KUBE-ROUTER-FORWARD)
trace id daac839a ip filter KUBE-ROUTER-FORWARD rule ip daddr 10.59.1.213  counter packets
    5001 bytes 6279235 jump KUBE-POD-FW-FIAOHC4WHRKERAQ6 (verdict jump
    KUBE-POD-FW-FIAOHC4WHRKERAQ6)
trace id daac839a ip filter KUBE-POD-FW-FIAOHC4WHRKERAQ6 rule  counter packets 5 bytes 300
    jump KUBE-NWPLCY-ZYSQVVSY5LQY7Q46 (verdict jump KUBE-NWPLCY-ZYSQVVSY5LQY7Q46)
trace id daac839a ip filter KUBE-NWPLCY-ZYSQVVSY5LQY7Q46 rule  limit rate 10/minute burst 10
    packets meta mark & 0x00010000 != 0x00010000 counter packets 5 bytes 300 log prefix
    "DROP by policy monitoring/prometheus-k8s" group 100 (verdict continue)
trace id daac839a ip filter KUBE-POD-FW-FIAOHC4WHRKERAQ6 rule  meta mark & 0x00010000 !=
    0x00010000 limit rate 10/minute burst 10 packets counter packets 5 bytes 300 log group
    100 (verdict continue)
trace id daac839a ip filter KUBE-POD-FW-FIAOHC4WHRKERAQ6 rule  meta mark & 0x00010000 !=
    0x00010000 counter packets 5 bytes 300 reject (verdict drop)

What’s that? log prefix "DROP by policy monitoring/prometheus-k8s"?? Guuaaaaauuuuu.

This is part 1 of x in a series.

I have spent most of my life avoiding DevOps-y type things. At GitHub I got familiar enough with kubectl to help debug the applications I had deployed on it, but that was almost a decade ago and I don’t remember a single bit of it.

Most of the things I run I deploy with a really simple systemd unit definition in the Nix module. Here’s an excerpt from the one for the Elixir app this blog ran on:

{
  systemd.services.kv = {
    description = "kv";
    enableStrictShellChecks = true;
    wantedBy = [ "multi-user.target" ];
    after = [ "kv-migrations.service" ];
    requires = [
      "postgresql.service"
      "kv-migrations.service"
    ];
  
    script = ''
      export KV_STORAGE_ROOT="$STATE_DIRECTORY"
      ${envVarScript}
      ${cfg.package}/bin/kv-server
    '';
  
    serviceConfig = {
      User = cfg.user;
      ProtectSystem = "strict";
      PrivateTmp = true;
      UMask = "0007";
      Restart = "on-failure";
      RestartSec = "10s";
      StateDirectory = "kv";
      StateDirectoryMode = "0750";
    };
  
    inherit environment;
  };
}

It’s very basic, and it worked beautifully! I love that, with NixOS, you can package a reproducible build (with all its dependencies), deployment strategy, and configuration schema all in one place. It’s so damn clean, and it works wonderfully for homelab- or personal services-scale systems. (For more, try Xe’s All Systems Go! talk, Writing your own NixOS modules for fun and (hopefully) profit.)

The downside is that this is not exactly a high-availability setup. When any of the dependencies of a service like this change — such as a new cfg.package, or change in environment — the result is that the existing service is stopped, the service is swapped out, and then the new one is started.

There can often be 10–30 seconds between the stop and start, depending on how much else the nixos-rebuild has to do. And while a failing build won’t leave you with a stopped service — you won’t even get that far — if the build succeeds, but the new service fails to come up for some reason, then you’ll be scrambling fast.

This being NixOS, getting your service back up is as easy as switching to the previous generation, and can be done very fast, but still, it’s not great. Realising this, and still very much wanting to use Nix as a build orchestrator in places where this isn’t an acceptable trade-off, it was time to learn a devops.

Structurally, Kubernetes seems relatively sound, giving us language for defining the shape of a deployed system upon many different axes. It is very YAML and it is very containers, neither of which I am the hugest fan of, but I felt pretty sure there would be tools to help with the former, and Nix my beloved has beautiful solutions for the latter.

If, like me before the start of this exercise, you don’t really know about the model Kubernetes gives you to work with, you might find useful David Ventura’s blog post, A skeptic’s first contact with Kubernetes. If I had found it before and not immediately after coming this far it would’ve been super helpful -_-

One thing worth mentioning is that, as a Very Nix Person (and Very Dissociated Person), I really need my infrastructure to be described in a version-controlled way. Ideally, I would be able to tie all of my infra back into the same place (which is vyx, a Nix flake).

So I decide to start up a cluster and begin experimenting. I hate Docker, Inc. with a passion — I will never forgive them for getting rid of Docker for Mac’s cute whale — plus I want to learn somewhere where I can actually deploy things, so I decide to start with k3s on my VPS. How I chose k3s to begin with, I’m not so sure — maybe because it has relatively few options exposed in its Nix module. Lightweight sounds good, and it’s a “certified Kubernetes distribution”. Whatever that means, it must be good!

NixOS has the option services.k3s.manifests, which is described as “auto-deploying manifests”. Perhaps this is the magic sauce I need to get my infrastructure as code!?

(The answer is, no, it isn’t — the entire cluster is restarted when you change its values, because NixOS. Teehee.)

Nonetheless, I struggled through writing some early manifests this way. Writing YAML in Nix is way better than writing YAML, and very easy to parameterise, extract functions, and so on. I had seen mention of Helm charts here and there, and while I felt like one day I would need to come to terms with them, I preferred to leave that until as late an opportunity as possible. As a bonus, using k3s auto-deploying manifests in this way meant I could write a NixOS module to deploy an application in Kubernetes, without a single line of raw YAML.

So, terrible in many respects — now bringing down an entire cluster on each change instead of just the relevant services (!!!) — but an introduction nonetheless. We are now at the point of siguiente:

• Decided to turn that homelab server into a gaming PC instead, haha psyche! Instead decided to learn better how to cross-build things and operate k3s without trying to shove everything through a NixOS module.

Part 2 will cover building our own software ready for orchestration (using Nix — we won’t write a single Dockerfile, promise, and as a little bit of a spoiler, we won’t write a single Go template either), and the unique fun presented by developing on aarch64-darwin while largely deploying to x86_64-linux. :)

Just a quick field report. I wanted to try ExternalDNS; I use bunny.net (obviously), and found external-dns-bunny-webhook! Perfect!

Only, it crash looped on startup. After asking myself whether I was really bothered enough to try getting a development environment setup for this (and then asking Annie, who helped resolve it to a yes), I rewrote the build process and development environment in Nix and got a fork with a fix going.

Given the sorry state of GitHub, I figured it was time to start opening “pull requests” there in the form of issues that contain instructions for fetching from off-site. Included here for posterity.

I really wanted to get this working for me, so I ended up forking the project with my fix and build environment. The webhook currently doesn’t handle root records being handed to it by ExternalDNS correctly, and will panic if that happens.

You can find the source here: https://nossa.ee/~talya/external-dns-bunny-webhook.

If you’d like to pull the relevant commit, this should do the trick:

$ git fetch https://nossa.ee/~talya/external-dns-bunny-webhook acb21849b8292222d7e0c85ac8d3dea913147bad
$ git cherry-pick FETCH_HEAD

The patch is inline here if you’d rather apply it directly with git am:

(elided)

I have a longer post about k8s cooking, but for now—

An hour and a half ago, I found this to-do created around 2am this morning:

I decided that, as annoying as dealing with Discord and OAuth and all kinds of things is, if I can’t make this happen and have it deployed and managed within the same sit as thinking about it, I need to know that.

The idea in that ellipsis isn’t a very new one: I look at Twitter and like to drop links to art I see there in Discord to my wife/puppygirl/bestie/whatever is going on between us that day. Twitter’s embeds aren’t very good; vxtwitter.com/fxtwitter.com make them better, and so I use them habitually by editing the links manually after pasting/before sending. Automate it with love.

In 90 minutes, I have an Elixir app, reproducibly built with Nix, reproducibly OCI-packaged, and deployed into k8s.

(The curious will find that secret created here — I bring up NixOS with any new secret definitions, vyx creating them via sops, before pushing/having Flux reconcile any new items referring to them.)

It works really well! It’s the same setup for my blog these days (— can’t believe my blog is on k8s, but oh well —), and soon nóssa too.

no highly esteemed deed is commemorated here… nothing valued is here. What is here was dangerous and repulsive to us.