kivikakk.ee

2 results —

deps_nix, not mix2nix!

elixir nix

Oh my goodness. Turns out deps_nix exists, and it obviates not only half of the hacks from yesterday, but also many, many more I’d just accepted as unavoidable, and something I’d have to build some tooling around. Well, looks like someone already has :)

Using Phoenix LiveView 1.1 with Nix

elixir nix

LiveView v1.1 has been released! I like to keep on top of Phoenix’s updates; the ecosystem moves at a reasonable and steady pace, and the direction is in general good. (Not sure I care for colocated hooks yet, but change tracking in comprehensions is extremely welcome, it’s nice to be able to ditch @types/phoenix_live_view, and <.portal> looks neat!)

LazyHTML is a new dependency, and you bet it tries to download pre-compiled dependencies. I never want that, and Nix is so good to me by simply Not Permitting It:

lazy_html> Running phase: unpackPhase

lazy_html> unpacking source archive /nix/store/qlgi3hx2syzhzq2r8l5b8xd42zzwrvb5-lazy_html-0.1.3

lazy_html> source root is lazy_html-0.1.3

lazy_html> Running phase: patchPhase

lazy_html> Running phase: updateAutotoolsGnuConfigScriptsPhase

lazy_html> Running phase: configurePhase

lazy_html> Running phase: buildPhase

lazy_html> ** (File.Error) could not make directory (with -p) "/homeless-shelter/Library/Caches/elixir_make": no such file or directory

lazy_html>     (elixir 1.18.4) lib/file.ex:346: File.mkdir_p!/1

lazy_html>     (elixir_make 0.9.0) lib/elixir_make/artefact.ex:22: ElixirMake.Artefact.cache_dir/0

lazy_html>     (elixir_make 0.9.0) lib/elixir_make/artefact.ex:85: ElixirMake.Artefact.archive_path/3

lazy_html>     (elixir_make 0.9.0) lib/mix/tasks/compile.elixir_make.ex:226: Mix.Tasks.Compile.ElixirMake.download_or_reuse_nif/3

lazy_html>     (elixir_make 0.9.0) lib/mix/tasks/compile.elixir_make.ex:169: Mix.Tasks.Compile.ElixirMake.run/1

lazy_html>     (mix 1.18.4) lib/mix/task.ex:495: anonymous fn/3 in Mix.Task.run_task/5

lazy_html>     (mix 1.18.4) lib/mix/tasks/compile.all.ex:117: Mix.Tasks.Compile.All.run_compiler/2

lazy_html>     (mix 1.18.4) lib/mix/tasks/compile.all.ex:97: Mix.Tasks.Compile.All.compile/4

Close enough, you get the idea — elixir_make tried to create $HOME/Library/Caches/elixir_make (because this is nix-darwin), and if it had succeeded, would have failed to then actually download anything. How will we figure this out?

tl;dr

Step one: stop LazyHTML from trying to use pre-compiled dependencies

Assuming you’ve produced deps.nix with mix2nix, we patch out the pre-compilation configuration from LazyHTML’s mix.exs:

let

  mixNixDeps = import ./deps.nix {

    inherit lib beamPackages;



    overrides = final: prev: {

      lazy_html = prev.lazy_html.overrideAttrs {

        patches = [

          ./lazy_html-mix.exs.patch

        ];

      };

    };

  };

diff --git a/mix.exs b/mix.exs

index fa0be6f..0f40d16 100644

--- a/mix.exs

+++ b/mix.exs

@@ -24,11 +24,6 @@ defmodule LazyHTML.MixProject do

           "LEXBOR_VERSION" => @lexbor_version

         }

       end,

-      # Precompilation

-      make_precompiler: {:nif, CCPrecompiler},

-      make_precompiler_url: "#{@github_url}/releases/download/v#{@version}/@{artefact_filename}",

-      make_precompiler_filename: "liblazy_html",

-      make_precompiler_nif_versions: [versions: ["2.16"]]

     ]

   end

 

@@ -42,7 +37,6 @@ defmodule LazyHTML.MixProject do

     [

       {:fine, "~> 0.1.0"},

       {:elixir_make, "~> 0.9.0"},

-      {:cc_precompiler, "~> 0.1", runtime: false},

       {:ex_doc, "~> 0.36", only: :dev, runtime: false}

     ]

   end

elixir_make does have a config key, :force_build, but we’d have to patch LazyHTML to include it either way: this build is completely independent of your application, so setting it in your own config won’t do anything at Nix build time, and there’s no environment variable equivalent like RUSTLER_PRECOMPILED_FORCE_BUILD_ALL.

And the result:

lazy_html> Running phase: unpackPhase

lazy_html> unpacking source archive /nix/store/qlgi3hx2syzhzq2r8l5b8xd42zzwrvb5-lazy_html-0.1.3

lazy_html> source root is lazy_html-0.1.3

lazy_html> Running phase: patchPhase

lazy_html> applying patch /nix/store/vk0kvy6ih5nwr75k7yzi8k8nnggycy6w-lazy_html-mix.exs.patch

lazy_html> patching file mix.exs

lazy_html> Running phase: updateAutotoolsGnuConfigScriptsPhase

lazy_html> Running phase: configurePhase

lazy_html> Running phase: buildPhase

lazy_html> make: git: No such file or directory

lazy_html> make: *** [Makefile:50: /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0] Error 127

lazy_html> ** (Mix) Could not compile with "make" (exit status: 2).

lazy_html> You need to have gcc and make installed. Try running the

lazy_html> commands "gcc --version" and / or "make --version". If these programs

lazy_html> are not installed, you will be prompted to install them.

lazy_html>

LazyHTML wraps Lexbor, a C library used for its HTML engine, and here it is trying to download its source at compile-time. (If you make git available to the build, Nix will block its network access.)

Step two: fetch Lexbor’s source ourselves

Let’s declare the version of Lexbor we want, fetch that from GitHub, and then put it where LazyHTML’s Makefile would clone it to. We also check that the version we’ve declared matches the one in LazyHTML’s mix.exs, so we’ll notice if it changes.

(You could try to extract this automatically, but you’ll have to update the hash manually either way. I’ll also skip the step where this fails because the build wants cmake and just give it cmake.)

let

  mixNixDeps = import ./deps.nix {

    inherit lib beamPackages;



    overrides = final: prev: {

      lazy_html = prev.lazy_html.overrideAttrs (

        prevAttrs:

        let

          lexborVersion = "2.4.0";

          lexbor = pkgs.fetchFromGitHub {

            owner = "lexbor";

            repo = "lexbor";

            tag = "v${lexborVersion}";

            hash = "sha256-wsm+2L2ar+3LGyBXl39Vp9l1l5JONWvO0QbI87TDfWM=";

          };

        in

        {

          nativeBuildInputs = prevAttrs.nativeBuildInputs ++ (with pkgs; [ cmake ]);



          prePatch = ''

            # ensure the version is in sync.

            grep -q '@lexbor_version "${lexborVersion}"' mix.exs

            mkdir -p _build/c/third_party/lexbor

            cp -r ${lexbor} _build/c/third_party/lexbor/${lexborVersion}

            chmod -R u+w _build/c/third_party/lexbor/${lexborVersion}

          '';



          patches = [

            ./lazy_html-mix.exs.patch

          ];

        }

      );

    };

  };

Now we successfully build Lexbor as part of LazyHTML, yay! But then:

lazy_html> [ 99%] Building C object CMakeFiles/lexbor_static.dir/source/lexbor/utils/http.c.o

lazy_html> [ 99%] Building C object CMakeFiles/lexbor_static.dir/source/lexbor/utils/warc.c.o

lazy_html> [100%] Linking C static library liblexbor_static.a

lazy_html> [100%] Built target lexbor_static

lazy_html> make[1]: Leaving directory '/private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0/build'

lazy_html> clang++ -shared -fPIC -fvisibility=hidden -std=c++17 -Wall -Wextra -Wno-unused-parameter -Wno-comment -I/nix/store/62pchbdm94l23xcd9c7z76pywli4v9r0-erlang-27.3.4.1/lib/erlang/erts-15.2.7/include -I/private/tmp/nix-build-fine-0.1.2.drv-0/fine-0.1.2/include -I/private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0/source -O3 -undefined dynamic_lookup -flat_namespace /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/c_src/lazy_html.cpp /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0/build/liblexbor_static.a -o /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/prod/lib/lazy_html/priv/liblazy_html.so

lazy_html> /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/c_src/lazy_html.cpp:3:10: fatal error: 'fine.hpp' file not found

lazy_html>     3 | #include <fine.hpp>

lazy_html>       |          ^~~~~~~~~~

lazy_html> 1 error generated.

lazy_html> make: *** [Makefile:38: /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/prod/lib/lazy_html/priv/liblazy_html.so] Error 1

lazy_html> ** (Mix) Could not compile with "make" (exit status: 2).

lazy_html> You need to have gcc and make installed. Try running the

lazy_html> commands "gcc --version" and / or "make --version". If these programs

lazy_html> are not installed, you will be prompted to install them.

lazy_html>

LazyHTML depends on Fine, and if you squint at that last output, you’ll see this argument to clang++:

-I/private/tmp/nix-build-fine-0.1.2.drv-0/fine-0.1.2/include

Fine reports its own include directory in Fine.include_dir/0 like this:

defmodule Fine do

  # [...]



  @include_dir Path.expand("include")



  @doc """

  Returns the directory with Fine header files.

  """

  @spec include_dir() :: String.t()

  def include_dir(), do: @include_dir

end

This resolves the absolute path of the include dir relative to the current working directory at compile time. If you’re building Fine in the same filesystem as its dependent, this works great — Mix changes the cwd to each project’s root as it goes. If not, you’ll have a compiled-in path that may not exist when you try to use it, which is exactly what’s happened above — that’s the Fine sandbox path being used while we’re building in LazyHTML’s sandbox.

Step three: get Fine to report its installed location, not its build-time one

Just one more patch. Just one. I swear. I can stop any time I want.

let

  mixNixDeps = import ./deps.nix {

    inherit lib beamPackages;



    overrides = final: prev: {

      lazy_html = prev.lazy_html.overrideAttrs (

        prevAttrs:

        let

          lexborVersion = "2.4.0";

          lexbor = pkgs.fetchFromGitHub {

            owner = "lexbor";

            repo = "lexbor";

            tag = "v${lexborVersion}";

            hash = "sha256-wsm+2L2ar+3LGyBXl39Vp9l1l5JONWvO0QbI87TDfWM=";

          };

        in

        {

          nativeBuildInputs = prevAttrs.nativeBuildInputs ++ (with pkgs; [ cmake ]);



          prePatch = ''

            # ensure the version is in sync.

            grep -q '@lexbor_version "${lexborVersion}"' mix.exs

            mkdir -p _build/c/third_party/lexbor

            cp -r ${lexbor} _build/c/third_party/lexbor/${lexborVersion}

            chmod -R u+w _build/c/third_party/lexbor/${lexborVersion}

          '';



          patches = [

            ./lazy_html-mix.exs.patch

          ];

        }

      );



      fine = prev.fine.overrideAttrs {

        patches = [

          ./fine-lib-fine.ex.patch

        ];

      };

    };

  };

diff --git a/lib/fine.ex b/lib/fine.ex

index 277b983..22c8d4f 100644

--- a/lib/fine.ex

+++ b/lib/fine.ex

@@ -8,11 +8,9 @@ defmodule Fine do

 

   @moduledoc readme_docs

 

-  @include_dir Path.expand("include")

-

   @doc """

   Returns the directory with Fine header files.

   """

   @spec include_dir() :: String.t()

-  def include_dir(), do: @include_dir

+  def include_dir(), do: Application.app_dir(:fine, "include")

 end

Instead of resolving something about cwd at compile-time, we resolve the installed application path at runtime. Lovely! 🍴

This now builds successfully! Yay!

And then your application fails bring-up when you’re using the compiled release:

ERROR! the application :lazy_html has a different value set for key :inspect_extra_newline during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behavior than expected:



  * Compile time value was set to: true

  * Runtime value was not set



To fix this error, you might:



  * Make the runtime value match the compile time one



  * Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.clean lazy_html --build"



  * Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag





Runtime terminating during boot ({<<"aborting boot">>,[{'Elixir.Config.Provider',boot,2,[]}]})



Crash dump is being written to: erl_crash.dump...⏎

Oop. Turns out LazyHTML’s config.exs declares a single config var to avoid some behaviour in test:

import Config



# We disable the extra newline in test env, because it breaks doctests.

config :lazy_html, :inspect_extra_newline, config_env() != :test

The actual use of this config has the correct default (since this config isn’t used in your dependent application):

  if Application.compile_env(:lazy_html, :inspect_extra_newline, true) do

    defp separator(), do: line()

  else

    defp separator(), do: empty()

  end

But, because we compiled LazyHTML by itself, that explicit true got compiled in, and the lack of runtime value for it means we crash out.

Step four: declare LazyHTML’s config in our application

In your config.exs:

config :lazy_html, :inspect_extra_newline, true

That’s it!

Egumi? Isn’t that a lot of work?

It sure is. Here’s the PRs I opened this morning to make some of this unnecessary:

LiveView also now supplies its own types (the DefinitelyTyped ones under @types/phoenix_live_view are good but not ~official~!), but not in a way that seems compatible with certain TypeScript moduleResolution configurations, so:

Aside

Technically, a lot of this is on us — we don’t actually need LazyHTML in our build, since it’s marked only: :test — but mix2nix doesn’t exclude it on that basis. phoenix_live_view also declares it as a dependency (albeit with optional: true), but you could do some override shenanigans to excise it entirely.

But: (a) I’ll do what I feel like, and (b) I actually use Floki in my application code, and it has support for a Lexbor-based parser which until now I’ve been unable to use precisely because it entailed all this. Now I can just port my Floki uses to LazyHTML and drop it entirely!

Edit: Fine v0.1.3 no longer includes the includes and it’s all my fault!

Dw bb. Nix gotchu. Get rid of the old patch — our new one is a bit more (and a bit less) dynamic.

fine = prev.fine.overrideAttrs (prevAttrs: {

  prePatch = ''

    sed -i -e "s|@include_dir Path\.expand(\"c_include\")|@include_dir \"${prevAttrs.src}/c_include\"|" lib/fine.ex

  '';

});

include was renamed to c_include — the only reason include was making it into Fine’s application directory was because that’s the directory name used by Erlang .hrl files.

How to solve this? We replace the call to Path.expand/1 with the path to the c_include directory in the source!

I really like knowing this happened right. Let’s decompile the resulting Elixir.Fine.beam file and pull Fine.include_dir/0’s contents:

iex(1)> with {:ok, {Fine, [{:abstract_code, {:raw_abstract_v1, code}}]}} <-

               :beam_lib.chunks(~c"/nix/store/nx1bl8jzsg1ns6yfv237spxbd45n9jxs-fine-0.1.3/lib/erlang/lib/fine-0.1.3/ebin/Elixir.Fine.beam", [:abstract_code]),

             [body] <- for({:function, _, :include_dir, _, body} <- code, do: body),

          do: :erl_prettypr.format(:erl_syntax.form_list(body), ribbon: 100) |> IO.puts()

() -> <<"/nix/store/d54xr3cssgmsj9kwkd04gzvx1jbi2pc3-fine-0.1.3/c_include">>

:ok

iex(2)>

Looks good to me! One question that arises — will this include path definitely be present for all uses of the resulting derivation? It is of course in a different store path — we couldn’t create a store path that referred to itself, after all. I think the answer is no — there’s nothing in the result derivation that tells Nix that the source must stick around.

The answer is to also specify the source derivation in propagatedNativeBuildInputs. What a mouthful. As a quick reviser, for a given derivation, the following definitions generally hold:

  • buildInputs — dependencies that must exist in the runtime environment.
  • nativeBuildInputs — dependencies that must exist in the build environment.
  • propagatedBuildInputs — dependencies that must exist in the runtime environment, and the runtime environment of any downstream user of this package.
  • propagatedNativeBuildInputs — dependencies that must exist in the build environment, and the build environment of any downstream user of this package.

So:

fine = prev.fine.overrideAttrs (prevAttrs: {

  propagatedNativeBuildInputs = [prevAttrs.src];

  prePatch = ''

    sed -i -e "s|@include_dir Path\.expand(\"c_include\")|@include_dir \"${prevAttrs.src}/c_include\"|" lib/fine.ex

  '';

});

The resulting derivation now has a file nix-support/propagated-native-build-inputs, which lists the source derivation, matching the path compiled into Elixir.Fine.beam. This should mean that any build that depends on fine will be assured of the include directory’s existence at build-time. :)

Maybe this’ll do!
1