LiveView v1.1 has been released! I like to keep on top of Phoenix’s updates; the ecosystem moves at a reasonable and steady pace, and the direction is in general good. (Not sure I care for colocated hooks yet, but change tracking in comprehensions is extremely welcome, it’s nice to be able to ditch @types/phoenix_live_view, and <.portal> looks neat!)
LazyHTML is a new dependency, and you bet it tries to download pre-compiled dependencies. I never want that, and Nix is so good to me by simply Not Permitting It:
lazy_html> Running phase: unpackPhase
lazy_html> unpacking source archive /nix/store/qlgi3hx2syzhzq2r8l5b8xd42zzwrvb5-lazy_html-0.1.3
lazy_html> source root is lazy_html-0.1.3
lazy_html> Running phase: patchPhase
lazy_html> Running phase: updateAutotoolsGnuConfigScriptsPhase
lazy_html> Running phase: configurePhase
lazy_html> Running phase: buildPhase
lazy_html> ** (File.Error) could not make directory (with -p) "/homeless-shelter/Library/Caches/elixir_make": no such file or directory
lazy_html> (elixir 1.18.4) lib/file.ex:346: File.mkdir_p!/1
lazy_html> (elixir_make 0.9.0) lib/elixir_make/artefact.ex:22: ElixirMake.Artefact.cache_dir/0
lazy_html> (elixir_make 0.9.0) lib/elixir_make/artefact.ex:85: ElixirMake.Artefact.archive_path/3
lazy_html> (elixir_make 0.9.0) lib/mix/tasks/compile.elixir_make.ex:226: Mix.Tasks.Compile.ElixirMake.download_or_reuse_nif/3
lazy_html> (elixir_make 0.9.0) lib/mix/tasks/compile.elixir_make.ex:169: Mix.Tasks.Compile.ElixirMake.run/1
lazy_html> (mix 1.18.4) lib/mix/task.ex:495: anonymous fn/3 in Mix.Task.run_task/5
lazy_html> (mix 1.18.4) lib/mix/tasks/compile.all.ex:117: Mix.Tasks.Compile.All.run_compiler/2
lazy_html> (mix 1.18.4) lib/mix/tasks/compile.all.ex:97: Mix.Tasks.Compile.All.compile/4
Close enough, you get the idea — elixir_make tried to create $HOME/Library/Caches/elixir_make (because this is nix-darwin), and if it had succeeded, would have failed to then actually download anything. How will we figure this out?
tl;dr
Step one: stop LazyHTML from trying to use pre-compiled dependencies
Assuming you’ve produced deps.nix with mix2nix, we patch out the pre-compilation configuration from LazyHTML’s mix.exs:
let
mixNixDeps = import ./deps.nix {
inherit lib beamPackages;
overrides = final: prev: {
lazy_html = prev.lazy_html.overrideAttrs {
patches = [
./lazy_html-mix.exs.patch
];
};
};
};
diff --git a/mix.exs b/mix.exs
index fa0be6f..0f40d16 100644
--- a/mix.exs
+++ b/mix.exs
@@ -24,11 +24,6 @@ defmodule LazyHTML.MixProject do
"LEXBOR_VERSION" => @lexbor_version
}
end,
- # Precompilation
- make_precompiler: {:nif, CCPrecompiler},
- make_precompiler_url: "#{@github_url}/releases/download/v#{@version}/@{artefact_filename}",
- make_precompiler_filename: "liblazy_html",
- make_precompiler_nif_versions: [versions: ["2.16"]]
]
end
@@ -42,7 +37,6 @@ defmodule LazyHTML.MixProject do
[
{:fine, "~> 0.1.0"},
{:elixir_make, "~> 0.9.0"},
- {:cc_precompiler, "~> 0.1", runtime: false},
{:ex_doc, "~> 0.36", only: :dev, runtime: false}
]
end
elixir_make does have a config key, :force_build, but we’d have to patch LazyHTML to include it either way: this build is completely independent of your application, so setting it in your own config won’t do anything at Nix build time, and there’s no environment variable equivalent like RUSTLER_PRECOMPILED_FORCE_BUILD_ALL.
And the result:
lazy_html> Running phase: unpackPhase
lazy_html> unpacking source archive /nix/store/qlgi3hx2syzhzq2r8l5b8xd42zzwrvb5-lazy_html-0.1.3
lazy_html> source root is lazy_html-0.1.3
lazy_html> Running phase: patchPhase
lazy_html> applying patch /nix/store/vk0kvy6ih5nwr75k7yzi8k8nnggycy6w-lazy_html-mix.exs.patch
lazy_html> patching file mix.exs
lazy_html> Running phase: updateAutotoolsGnuConfigScriptsPhase
lazy_html> Running phase: configurePhase
lazy_html> Running phase: buildPhase
lazy_html> make: git: No such file or directory
lazy_html> make: *** [Makefile:50: /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0] Error 127
lazy_html> ** (Mix) Could not compile with "make" (exit status: 2).
lazy_html> You need to have gcc and make installed. Try running the
lazy_html> commands "gcc --version" and / or "make --version". If these programs
lazy_html> are not installed, you will be prompted to install them.
lazy_html>
LazyHTML wraps Lexbor, a C library used for its HTML engine, and here it is trying to download its source at compile-time. (If you make git available to the build, Nix will block its network access.)
Step two: fetch Lexbor’s source ourselves
Let’s declare the version of Lexbor we want, fetch that from GitHub, and then put it where LazyHTML’s Makefile would clone it to. We also check that the version we’ve declared matches the one in LazyHTML’s mix.exs, so we’ll notice if it changes.
(You could try to extract this automatically, but you’ll have to update the hash manually either way. I’ll also skip the step where this fails because the build wants cmake and just give it cmake.)
let
mixNixDeps = import ./deps.nix {
inherit lib beamPackages;
overrides = final: prev: {
lazy_html = prev.lazy_html.overrideAttrs (
prevAttrs:
let
lexborVersion = "2.4.0";
lexbor = pkgs.fetchFromGitHub {
owner = "lexbor";
repo = "lexbor";
tag = "v${lexborVersion}";
hash = "sha256-wsm+2L2ar+3LGyBXl39Vp9l1l5JONWvO0QbI87TDfWM=";
};
in
{
nativeBuildInputs = prevAttrs.nativeBuildInputs ++ (with pkgs; [ cmake ]);
prePatch = ''
grep -q '@lexbor_version "${lexborVersion}"' mix.exs
mkdir -p _build/c/third_party/lexbor
cp -r ${lexbor} _build/c/third_party/lexbor/${lexborVersion}
chmod -R u+w _build/c/third_party/lexbor/${lexborVersion}
'';
patches = [
./lazy_html-mix.exs.patch
];
}
);
};
};
Now we successfully build Lexbor as part of LazyHTML, yay! But then:
lazy_html> [ 99%] Building C object CMakeFiles/lexbor_static.dir/source/lexbor/utils/http.c.o
lazy_html> [ 99%] Building C object CMakeFiles/lexbor_static.dir/source/lexbor/utils/warc.c.o
lazy_html> [100%] Linking C static library liblexbor_static.a
lazy_html> [100%] Built target lexbor_static
lazy_html> make[1]: Leaving directory '/private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0/build'
lazy_html> clang++ -shared -fPIC -fvisibility=hidden -std=c++17 -Wall -Wextra -Wno-unused-parameter -Wno-comment -I/nix/store/62pchbdm94l23xcd9c7z76pywli4v9r0-erlang-27.3.4.1/lib/erlang/erts-15.2.7/include -I/private/tmp/nix-build-fine-0.1.2.drv-0/fine-0.1.2/include -I/private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0/source -O3 -undefined dynamic_lookup -flat_namespace /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/c_src/lazy_html.cpp /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/c/third_party/lexbor/2.4.0/build/liblexbor_static.a -o /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/prod/lib/lazy_html/priv/liblazy_html.so
lazy_html> /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/c_src/lazy_html.cpp:3:10: fatal error: 'fine.hpp' file not found
lazy_html> 3 | #include <fine.hpp>
lazy_html> | ^~~~~~~~~~
lazy_html> 1 error generated.
lazy_html> make: *** [Makefile:38: /private/tmp/nix-build-lazy_html-0.1.3.drv-0/lazy_html-0.1.3/_build/prod/lib/lazy_html/priv/liblazy_html.so] Error 1
lazy_html> ** (Mix) Could not compile with "make" (exit status: 2).
lazy_html> You need to have gcc and make installed. Try running the
lazy_html> commands "gcc --version" and / or "make --version". If these programs
lazy_html> are not installed, you will be prompted to install them.
lazy_html>
LazyHTML depends on Fine, and if you squint at that last output, you’ll see this argument to clang++:
-I/private/tmp/nix-build-fine-0.1.2.drv-0/fine-0.1.2/include
Fine reports its own include directory in Fine.include_dir/0 like this:
defmodule Fine do
@include_dir Path.expand("include")
@spec include_dir() :: String.t()
def include_dir(), do: @include_dir
end
This resolves the absolute path of the include dir relative to the current working directory at compile time. If you’re building Fine in the same filesystem as its dependent, this works great — Mix changes the cwd to each project’s root as it goes. If not, you’ll have a compiled-in path that may not exist when you try to use it, which is exactly what’s happened above — that’s the Fine sandbox path being used while we’re building in LazyHTML’s sandbox.
Step three: get Fine to report its installed location, not its build-time one
Just one more patch. Just one. I swear. I can stop any time I want.
let
mixNixDeps = import ./deps.nix {
inherit lib beamPackages;
overrides = final: prev: {
lazy_html = prev.lazy_html.overrideAttrs (
prevAttrs:
let
lexborVersion = "2.4.0";
lexbor = pkgs.fetchFromGitHub {
owner = "lexbor";
repo = "lexbor";
tag = "v${lexborVersion}";
hash = "sha256-wsm+2L2ar+3LGyBXl39Vp9l1l5JONWvO0QbI87TDfWM=";
};
in
{
nativeBuildInputs = prevAttrs.nativeBuildInputs ++ (with pkgs; [ cmake ]);
prePatch = ''
grep -q '@lexbor_version "${lexborVersion}"' mix.exs
mkdir -p _build/c/third_party/lexbor
cp -r ${lexbor} _build/c/third_party/lexbor/${lexborVersion}
chmod -R u+w _build/c/third_party/lexbor/${lexborVersion}
'';
patches = [
./lazy_html-mix.exs.patch
];
}
);
fine = prev.fine.overrideAttrs {
patches = [
./fine-lib-fine.ex.patch
];
};
};
};
diff --git a/lib/fine.ex b/lib/fine.ex
index 277b983..22c8d4f 100644
--- a/lib/fine.ex
+++ b/lib/fine.ex
@@ -8,11 +8,9 @@ defmodule Fine do
@moduledoc readme_docs
- @include_dir Path.expand("include")
-
@doc """
Returns the directory with Fine header files.
"""
@spec include_dir() :: String.t()
- def include_dir(), do: @include_dir
+ def include_dir(), do: Application.app_dir(:fine, "include")
end
Instead of resolving something about cwd at compile-time, we resolve the installed application path at runtime. Lovely! 🍴
This now builds successfully! Yay!
And then your application fails bring-up when you’re using the compiled release:
ERROR! the application :lazy_html has a different value set for key :inspect_extra_newline during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behavior than expected:
* Compile time value was set to: true
* Runtime value was not set
To fix this error, you might:
* Make the runtime value match the compile time one
* Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.clean lazy_html --build"
* Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag
Runtime terminating during boot ({<<"aborting boot">>,[{'Elixir.Config.Provider',boot,2,[]}]})
Crash dump is being written to: erl_crash.dump...⏎
Oop. Turns out LazyHTML’s config.exs declares a single config var to avoid some behaviour in test:
import Config
config :lazy_html, :inspect_extra_newline, config_env() != :test
The actual use of this config has the correct default (since this config isn’t used in your dependent application):
if Application.compile_env(:lazy_html, :inspect_extra_newline, true) do
defp separator(), do: line()
else
defp separator(), do: empty()
end
But, because we compiled LazyHTML by itself, that explicit true got compiled in, and the lack of runtime value for it means we crash out.
Step four: declare LazyHTML’s config in our application
In your config.exs:
config :lazy_html, :inspect_extra_newline, true
That’s it!
Egumi? Isn’t that a lot of work?
It sure is. Here’s the PRs I opened this morning to make some of this unnecessary:
LiveView also now supplies its own types (the DefinitelyTyped ones under @types/phoenix_live_view are good but not ~official~!), but not in a way that seems compatible with certain TypeScript moduleResolution configurations, so:
Aside
Technically, a lot of this is on us — we don’t actually need LazyHTML in our build, since it’s marked only: :test — but mix2nix doesn’t exclude it on that basis. phoenix_live_view also declares it as a dependency (albeit with optional: true), but you could do some override shenanigans to excise it entirely.
But: (a) I’ll do what I feel like, and (b) I actually use Floki in my application code, and it has support for a Lexbor-based parser which until now I’ve been unable to use precisely because it entailed all this. Now I can just port my Floki uses to LazyHTML and drop it entirely!
Edit: Fine v0.1.3 no longer includes the includes and it’s all my fault!
Dw bb. Nix gotchu. Get rid of the old patch — our new one is a bit more (and a bit less) dynamic.
fine = prev.fine.overrideAttrs (prevAttrs: {
prePatch = ''
sed -i -e "s|@include_dir Path\.expand(\"c_include\")|@include_dir \"${prevAttrs.src}/c_include\"|" lib/fine.ex
'';
});
include was renamed to c_include — the only reason include was making it into Fine’s application directory was because that’s the directory name used by Erlang .hrl files.
How to solve this? We replace the call to Path.expand/1 with the path to the c_include directory in the source!
I really like knowing this happened right. Let’s decompile the resulting Elixir.Fine.beam file and pull Fine.include_dir/0’s contents:
with {:ok, {Fine, [{:abstract_code, {:raw_abstract_v1, code}}]}} <-
:beam_lib.chunks(~c"/nix/store/nx1bl8jzsg1ns6yfv237spxbd45n9jxs-fine-0.1.3/lib/erlang/lib/fine-0.1.3/ebin/Elixir.Fine.beam", [:abstract_code]),
[body] <- for({:function, , :include_dir, , body} <- code, do: body),
do: :erl_prettypr.format(:erl_syntax.form_list(body), ribbon: 100) |> IO.puts()
() -> <<"/nix/store/d54xr3cssgmsj9kwkd04gzvx1jbi2pc3-fine-0.1.3/c_include">>
:ok
Looks good to me! One question that arises — will this include path definitely be present for all uses of the resulting derivation? It is of course in a different store path — we couldn’t create a store path that referred to itself, after all. I think the answer is no — there’s nothing in the result derivation that tells Nix that the source must stick around.
The answer is to also specify the source derivation in propagatedNativeBuildInputs. What a mouthful. As a quick reviser, for a given derivation, the following definitions generally hold:
buildInputs — dependencies that must exist in the runtime environment.nativeBuildInputs — dependencies that must exist in the build environment.propagatedBuildInputs — dependencies that must exist in the runtime environment, and the runtime environment of any downstream user of this package.propagatedNativeBuildInputs — dependencies that must exist in the build environment, and the build environment of any downstream user of this package.
So:
fine = prev.fine.overrideAttrs (prevAttrs: {
propagatedNativeBuildInputs = [prevAttrs.src];
prePatch = ''
sed -i -e "s|@include_dir Path\.expand(\"c_include\")|@include_dir \"${prevAttrs.src}/c_include\"|" lib/fine.ex
'';
});
The resulting derivation now has a file nix-support/propagated-native-build-inputs, which lists the source derivation, matching the path compiled into Elixir.Fine.beam. This should mean that any build that depends on fine will be assured of the include directory’s existence at build-time. :)
Maybe this’ll do!
