Build and development environment

1 Introduction

Originally this project started out as a fork of the Nixkell project skeleton at commit 29e78ce (Some tidying (#36), 2024-03-18), which gave us a working Haskell development environment with many of the infrastructural issues sorted out with Nix. The Nixkell project encourages wiping the Nixkell-specific history, but we decided to keep that history for additional context.

2 Nix

Using Nix helps us set up a reproducible development environment, as well as package up Lilac as a full-fledged Nix package (so that anyone with Nix can install it by just cloning the repo and then running a Nix command to build and install it).

2.1 Sources (Nix dependencies)

The file nix/sources.json (generated by Niv) declares the Nix packages snapshot we use for reproducibility. We have to update this snapshot periodically (for example, to use a newer GHC version).

update-nix:
	niv init
	niv update nixpkgs --branch Nixpkgs version

The niv tool is available in our development shell.

pkgs.niv

The niv init updates the niv-generated nix/sources.nix. The niv update updates the nix/sources.json.

In niv's terminology, the nixpkgs above is a "package" (nix/sources.json declares all packages we want to use). The --branch ... flag tells niv to use a particular Git branch for the update (because Nixpkgs is a Git repository, we can use Git branches to track a particular version). Of course, the branch is mutable, so we have to also track the particular Git SHA that the branch was pointing to at the time of running the niv update command; this information is also stored in nix/sources.json. For now we track the nixos-YY.MM release branch because it's the stable branch.

nixos-26.05

The nix/ folder houses Nix files for our development environment. The most important is nix/sources.nix which is the starting point, as it defines the universe of imported Nix functions and packages we want to use in order to set up our environment for Lilac.

The nix/sources.nix references nix/sources.json.

nix-repl> sources = import ./nix/sources.nix

nix-repl> sources
{ __functor = «lambda @ /Users/l/prog/lilac/nix/sources.nix:198:43»; "git-hooks.nix" = { ... }; "gitignore.nix" = { ... }; nixpkgs = { ... }; }

Importing sources.nix gives us git-hooks.nix, gitignore.nix, and nixpkgs (we can ignore __functor as that appears to be an internal thing). This is because nix/sources.json has those three entries (you use niv add to add new entries). The latter is in quotes because it has a dot in its name (and the Nix language normally treats the dot as an attribute access operator).

The nixpkgs is a tarball of the set of all Nixpkgs packages we have access to.

nix-repl> builtins.attrNames sources.nixpkgs
[ "branch" "description" "homepage" "outPath" "owner" "repo" "rev" "sha256" "type" "url" "url_template" ]

nix-repl> sources.nixpkgs.type
"tarball"

nix-repl> sources.nixpkgs.description
"Nix Packages collection"

If you evaluate sources.nixpkgs.branch in the nix-repl you will get the same Nixpkgs branch name as the one in Nixpkgs version.

The gitignore.nix is a tarball of the gitignore.nix project, which is just a collection of Nix functions that allows us to ignore files in .gitignore when packaging up our own source code repositories (in our case, Lilac itself).

Lastly, git-hooks.nix allows us to integrate https://pre-commit.com Git hooks with Nix. See nix:pre-commit-checks to see how it's used.

2.2 Build entrypoint

Let's look at nix/release.nix because this is our "entrypoint" for building Lilac.

{ lilacGitVersion ? "unknown" }:
let
  pkgs = import ./default.nix { inherit lilacGitVersion; }; # 1
in pkgs.lilacPkgs.lilac

This file is used for creating our release builds. It ultimately resolves to a single derivation we define in nix:Lilac derivation. So building this with make:Build lilac with Nix results in just a single package. If we were to put pkgs.lilacPkgs instead of pkgs.lilacPkgs.lilac, Nix will build all derivations it finds under pkgs.lilacPkgs, not just the pkgs.lilacPkgs.lilac one (this may be useful for debugging, for example).

The default.nix at 1 is the default file used when invoking nix-build . (just as how shell.nix is the file for nix-shell). This behavior is just how Nix behaves; for our purposes we won't be using default.nix directly, because nix/release.nix pulls it in.

{ lilacGitVersion ? "unknown" }:
let sources = import ./sources.nix; # 2
in import sources.nixpkgs {
  system = builtins.currentSystem;
  config = { };
  overlays = [
    (final: prev: {
      lilacPkgs = import ./packages.nix { # 3
        pkgs = prev;
        fetchFromGitHub = prev.fetchFromGitHub;
        inherit lilacGitVersion;
      };
    })
  ];
}

Anyway, default.nix imports our sources.nix (generated by niv) at 2, and also defines our overlays.

Overlays are a standard Nix pattern for modifying existing Nix packages. In our case, we are importing the universe of all Nixpkgs packages from nix/sources.nix, and then injecting our own lilac package into it.

The overlay in turn imports nix/packages.nix at 3, which is where most of the heavy lifting is done. In the overlay, we inject a new Haskell package called lilacPkgs by evaluating the function in nix/packages.nix, which returns what's known as an attrset in Nix, which is just a key/value map.

3 packages.nix

We call this file packages.nix because it is a function which returns two very important derivations, devShell 4 and lilac 5.

{ pkgs, fetchFromGitHub, lilacGitVersion }:
let
  nix:devShellEnv
  nix:scripts
  nix:tools
  nix:ghc
  nix:ourHaskell
  nix:ghcVersion
  nix:Haskell overlays
  nix:projectRoot
  nix:lilacSource
  nix:buildForCI
  nix:buildFaster
  nix:pre-commit-checks
in {
  nix:devShell derivation # 4
  nix:Lilac derivation # 5
}

There's a lot to unpack here; see ghc for most of the discussions around taming the Haskell ecosystem with Nix and devShell for additional environment setup concerns for local development.

3.1 devShell

The devShell is the development environment, and is based on the pkgs.mkShell function.

devShell = pkgs.mkShell {
  buildInputs = [ devShellEnv ]; # 6
  shellHook = ''
    # Create a local cabal config that points to Nix store. This prevents
    # cabal from downloading and instead uses the GHC package DB.
    export CABAL_CONFIG="$PWD/cabal.config.nix"
    echo "package-db: $(ghc --print-libdir)/package.conf.d" > cabal.config.nix
    echo "with-compiler: $(which ghc)" >> cabal.config.nix

    export LD_LIBRARY_PATH=${devShellEnv}/lib:$LD_LIBRARY_PATH
    # Used when building with cabal.
    export LILAC_PROJECT_ROOT=$PWD
    # Set up Git pre-commit hooks.
    ${pre-commit-check.shellHook}
    logo
  '';
};

The devShellEnv at 6 is another derivation, which is just a collection of symlinks of all build dependencies for Lilac, plus some additional scripts and tools, and GHC (ghc). This is the main "magic" of a Nix dev environment, where suddenly you gain access to certain binaries --- all thanks to these symlinks on the path.

devShellEnv = pkgs.buildEnv {
  name = "lilac-devShellEnv";
  paths = scripts ++ tools ++ [ ghc ];
};

There are three main things we want to add to the PATH: scripts, tools, and GHC.

The scripts are the custom shell scripts we defined earlier in nix/scripts.nix.

scripts = import ./scripts.nix { inherit pkgs; };

The tools are packages from Nixpkgs that we want available in our development environment. Some tools are categorized as Haskell packages (under the haskellPackages prefix). We use ourHaskell to specify these tools, to make Nix compile these tools with the same version of GHC that we are using in the GHC version.

tools = [
  nix:tools-fe
  nix:tool-niv
  nix:tool-cabal
  # 7
  ourHaskell.hspec-discover
  nix:tool-editorconfig-checker
  nix:tool-shellcheck
  nix:tool-shfmt
  nix:tool-nixfmt
  nix:tool-ormolu
  nix:tool-parinfer-rust
  nix:tool-cspell
  nix:tool-pikchr
];

7 is needed to run our tests.

This leaves us with ghc, which we cover in the next section.

3.2 ghc

In nix:ghc (referenced by nix/packages.nix) we use ghc that Nixpkgs provides, but teach it about the Haskell dependencies (libraries) that Lilac uses. This means you can invoke ghci, and then load any library you want (needed by Lilac) into it. See the file .ghc.environment.<arch>-<GHC version> (written by Cabal at the project root) for a list of such dependencies.

hlib = pkgs.haskell.lib;
ghc = ourHaskell.ghc.withPackages
  (_: hlib.getHaskellBuildInputs ourHaskell.lilac);

The ourHaskell bit is the GHC version we want to use from Nixpkgs, but with a few overlays to tweak it a bit.

lib = pkgs.lib;

# 8
ourHaskell = pkgs.haskell.packages.${ghcVersion}.extend
  (lib.composeManyExtensions haskellOverlays);

The ghcVersion ultimately comes from GHC version. It's just the string ghc plus the numeric bits of GHC version without the dots in it, because this is how Nixpkgs names GHC versions.

ghcVersionWithoutDots = lib.replaceStrings [ "." ] [ "" ] "GHC version";
ghcVersion = "ghc" + ghcVersionWithoutDots;

Going back to ourHaskell, the haskellOverlays are straightforward enough. The first overlay injectLilac adds a new Haskell package called lilac. The other one is tweakLilacBuild which adjusts the build process so that the Template Haskell code can use the LILAC_GIT_VERSION environment variable to use as the output for the --version flag.

haskellOverlays = [
  disableHaddockGlobally
  heavyDeps
  jailbreaks
  injectExternalPackages
  injectLilac
  tweakLilacBuild
];

disableHaddockGlobally = final: prev: {
  mkDerivation = args: prev.mkDerivation (args // { doHaddock = false; });
};

heavyDeps = final: prev:
  let
    heavyPackages = [
      "brick"
      "fsnotify"
      "pandoc-types"
      "sandwich"
      "skylighting"
      "skylighting-core"
      "skylighting-format-ansi"
      "skylighting-format-blaze-html"
      "skylighting-format-latex"
      "th-lift-instances"
      "unicode-collation"
      "unicode-transforms"
      "vty-crossplatform"
      "wai"
      "wai-app-static"
      "wai-extra"
      "wai-websockets"
      "warp"
      "websockets"
    ];
  in lib.genAttrs heavyPackages (name: buildForCI prev.${name});

jailbreaks = final: prev: {
  jailbreaks:unicode-data
};

injectExternalPackages = final: prev: {
  injectExternalPackage:citeproc
};

injectLilac = final: prev: { # 9
  lilac = buildFaster (prev.callCabal2nix "lilac" lilacSource { });
};

tweakLilacBuild = final: prev: {
  lilac = prev.lilac.overrideAttrs (drv: {
    buildPhase = ''
      export LILAC_GIT_VERSION=${lilacGitVersion}
      echo "Using LILAC_GIT_VERSION $LILAC_GIT_VERSION"
    '' + drv.buildPhase;
  });
};

When our Nixpkgs channel (26.11? 27.05?) begins using GHC 9.12 as the default compiler version, we can drop disableHaddockGlobally because we'll be able to use the binary cache. That is, the only reason we have disableHaddockGlobally is because our compiler version does not have pre-built packages on Nix because it's too new (Nix does not build Haskell packages for newer GHC versions).

The lilacSource at 9 is the source code needed to build Lilac, and buildFaster is some Haskell build configuration options we pass in to configure the Lilac build process.

3.3 lilacSource

We want to tell Nix to use the local directory path of the project root as the starting point (this way, we can build Lilac without having to make a Git commit first --- that is, we could choose to create a Git commit, then pull this repo back down from Git at that commit to start building it, but for now we just build from disk regardless of Git).

projectRoot = ../.;

We have to use the parent directory because the projectRoot is relative to the location of nix/packages.nix, which is itself in a child folder (nix) in the actual project root.

Anyway, we are now confronted with a new problem: this means Nix will use all files in the project root. We have to pare it down to only those bits that actually matter for cabal.

We use the gitignoreFilterWith function to filter out all paths that are already defined inside the existing .gitignore. We also filter out additional files that are not needed for building Lilac, such as the woven HTML files in 13.

gnf =
  import (import ./sources.nix)."gitignore.nix" { inherit lib; }; # 10
lilacSource = lib.cleanSourceWith {
  src = projectRoot;
  name = baseNameOf projectRoot + "-src"; # 11
  filter = let # 12
    filterFunc = gnf.gitignoreFilterWith {
      basePath = projectRoot;
      # 13
      extraRules = ''
        .*
        *.html
        *.nix
        *.sh
        *.toml
        image
        Makefile
        nix
      '';
    };
  in path: type: filterFunc path type;
};

The gnf at 10 stands for "gitignore Nix functions". As for the actual .gitignore itself, we prefer to use a single .gitignore file at the root of the project.

.cpcache
.direnv
.ghc.environment*
.hie
.rebel_readline_history
Generated pre-commit config
js/*
# 14
!js/lilac.js.injectme
cabal.config.nix
cabal.project.local
dist-newstyle
orgdoc.json
result*
*~
*.css
*.html
*.larc
*.lobj
**/.*
!**/.gitattributes
!**/.gitignore
!**/.gitkeep
!.build.yml
site
site.tar.gz
vendor

gitignore:Makefile sentinels

Using nix repl we can inspect the gnf function directly:

nix-repl> nixpkgs = (import ./sources.nix).nixpkgs

nix-repl> gnf = import (import ./sources.nix)."gitignore.nix" { lib = nixpkgs.pkgs.lib; }

nix-repl> builtins.attrNames gnf
[ "gitignoreFilter" "gitignoreFilterWith" "gitignoreSource" "gitignoreSourceWith" ]

Back to lilacSource, it has a name at 11 consisting of the project root folder, plus a -src suffix. When building Lilac, this name will be used for generating a Nix store entry for all of the source code used for building the project:

...
Running phase: unpackPhase unpacking source archive /nix/store/z4lzldnnna5ghxqs84s8r87gxvr4bwnj-lilac-src
...

It's important that we use a let binding for filterFunc at 12. This memoizes the expensive part of the computation. If we don't do this, it will re-parse the .gitignore files it finds (recursively, from the project root) all over again each time it is called.

3.4 lilac attribute

Recall that the nix:Haskell overlays added the Lilac package to the top-level Nixpkgs set of packages, which we named ourHaskell 8. We expose it now as just lilac.

lilac = ourHaskell.lilac;

So this means any Nix expression that imports nix/packages.nix will be able to refer to the Lilac Haskell build as lilac. This is what we do in nix/default.nix where we decide to name the attribute set from the evaluation of nix/packages.nix as lilacPkgs (so it would be lilacPkgs.lilac). And consequently, nix/release.nix references lilacPkgs.lilac so that only this particular package gets built if we choose to feed this file into nix-build.

You can inspect the lilacPkgs.lilac derivation by running nix repl from the project root, then typing into the REPL the following example:

x = import ./nix/default.nix { lilacGitVersion = "someversion"; }
x.lilacPkgs.lilac

The first line calls the expression in default.nix with the argument for lilacGitVersion. The next line x.lilacPkgs.lilac accesses the lilac attribute which is just a derivation (an installable Nix package).

3.5 Haskell packages

3.5.1 buildFaster function

The buildFaster function customizes the first argument haskellPkg which is a Haskell package, which should be the output of calling callCabal2nix.

buildFaster = haskellPkg:
  let
    confFns = [
      hlib.dontHyperlinkSource
      hlib.dontCoverage
      hlib.dontHaddock
      hlib.disableExecutableProfiling
      hlib.disableLibraryProfiling
      hlib.dontBenchmark
    ];

  in lib.pipe haskellPkg confFns;

The general idea is to speed up the build by opting out of some things. We want to optimize the build for speed when developing locally, because we want to build Lilac as quickly as possible to tighten the feedback loop when working with tangle-loop.sh.

You may want to consult the GHC documentation to make sense out of some of these flags. For example, see runtime system (RTS) options for the RTS flags.

3.5.2 buildForCI function

This is similar to nix:buildFaster, but is slightly different. It was created because SourceHut CI kept failing with OOM errors (Haddock being the main culprit). We had to enable library profiling because some packages failed to build without it.

buildForCI = haskellPkg:
  let
    confFns = [
      hlib.dontHyperlinkSource
      hlib.dontCoverage
      hlib.dontHaddock
      hlib.disableExecutableProfiling
      hlib.dontCheck
      hlib.dontBenchmark
    ];

  in lib.pipe haskellPkg confFns;

3.5.3 External Haskell packages from beyond Nixpkgs

Sometimes we might want to use a Haskell package that's not in Nixpkgs (for example, if the version in Nixpkgs is broken and the fix for that package is already in upstream but not yet in Nixpkgs). And so we allow users to optionally specify packages by the GitHub commit.

To add something directly from GitHub (such as if the latest version on Hackage is still too old for us), we can put the following inside the injectExternalPackages section in nix:Haskell overlays.

  foo-bar =
    pkgs.haskell.lib.dontCheck (
      prev.callCabal2nix "foo-bar" (
        builtins.fetchFromGitHub {
          owner = "x";
          repo = "y";
          rev = "0000000000000000000000000000000000000000";
          sha256 = "00000";
        }
      ) {}
    );

For example, the above will override the default version of foo-bar with the one defined via Git.

3.5.4 Removing version bounds

You can remove version bound checks on package foo-package like this inside the jailbreaks section of nix:Haskell overlays.

foo-package = pkgs.haskell.lib.doJailbreak(prev.foo-package);

Jailbreaking is a common technique to work around packages that sometimes have unnecessarily strict version bounds.

3.6 Diagramming tools

We use Pikchr to draw SVG drawings. Originally we wanted to use Asymptote, but ruled against it because it requires a full-blown TeX distribution (because it relies on LaTeX for rendering text). Pikchr is just a standalone binary.

pkgs.pikchr

The rules for converting a *.pikchr file into an SVG is straightforward; we just invoke the pikchr command against it.

image_sources = $(wildcard image/*.pikchr)
image_svgs = $(image_sources:.pikchr=.svg)

%.svg: %.pikchr
	@echo "Compiling Pikchr diagram: $<"
	pikchr --svg-only $< > $@

Ignore the SVG text from diffs, because they are not human-readable.

image/*.svg -diff

See fig:lilac-subcommands.pikchr for a sample illustration using the Pikchr language.

4 Known issues and workarounds

Here we document various breakages and our workarounds for them.

4.1 citeproc

The version of citeproc on Nixpkgs 25.04 is 0.8.1.1, which is too old. So pull in a newer one.

citeproc = buildFaster (prev.callCabal2nix "citeproc" (let
  repo = fetchFromGitHub {
    owner = "jgm";
    repo = "citeproc";
    rev = "e5f22f0b0f7662524ff52883582d883316f4c113";
    sha256 = "sha256-xdwUdmmiTwHn0o0BeX949FPQFftHiTUmzieijpLxruc=";
  };
in "${repo}") { });

We also have to jailbreak (disable tests) for unicode-data (a dependency of citeproc), because otherwise it fails like build-failure:unicode-data:

unicode-data = pkgs.haskell.lib.dontCheck prev.unicode-data;

error: builder for '/nix/store/2la73rm4w1rb1a6ggml9bn1j1ba9gcfj-unicode-data-0.6.0.drv' failed with exit code 1;
       last 25 log lines:
       > [WARNING] Cannot test '\124410': incompatible Unicode version (unassigned char). Expected 15.1.0, but got: 16.0.0
       >       Compare to base [✔]
       >     isNumber implies a numeric value [✔]
       >
       > Failures:
       >
       >   test/Unicode/CharSpec.hs:206:21:
       >   1) Unicode.Char.Case toUpper
       >        predicate failed on: '\411'
       >
       >   To rerun use: --match "/Unicode.Char/Case/toUpper/" --seed 583422839
       >
       >   test/Unicode/CharSpec.hs:227:21:
       >   2) Unicode.Char.Case toTitle
       >        predicate failed on: '\411'
       >
       >   To rerun use: --match "/Unicode.Char/Case/toTitle/" --seed 583422839
       >
       > Randomized with seed 583422839
       >
       > Finished in 0.7974 seconds
       > 41 examples, 2 failures
       > Test suite test: FAIL
       > Test suite logged to: dist/test/unicode-data-0.6.0-test.log
       > 0 of 1 test suites (0 of 1 test cases) passed.
       For full logs, run:
         nix-store -l /nix/store/2la73rm4w1rb1a6ggml9bn1j1ba9gcfj-unicode-data-0.6.0.drv

As of this time, unicode-data 0.6.0 only works with Unicode version 15.1.0. When they update to support 16.0.0, we should be able to enable checks again.

5 shell.nix

The entrypoint of our Nix development environment is the following shell.nix file.

let pkgs = import ./nix/default.nix { }; in pkgs.lilacPkgs.devShell

See packages.nix for a deeper discussion of the underlying Nix infrastructure, including the nix:devShell derivation, which shows how we declaratively define which particular executables are made available to our development environment (including nix/scripts.nix).

The logo in the shellHook section in nix:devShell derivation is just a custom script to show an ASCII-art Lilac logo when we enter the development environment.

{ pkgs }:
let
  logo = pkgs.writeShellScriptBin "logo" ''
    set -euo pipefail
    echo -e "\n$(tput setaf 4)"
    echo Lilac | ${pkgs.figlet}/bin/figlet
    echo -e "$(tput sgr0)\n"
  '';
  lilac = pkgs.writeShellScriptBin "lilac" ''
    set -euo pipefail
    cabal-run-invocation
  '';
  lilac-nix = pkgs.writeShellScriptBin "lilac-nix" ''
    set -euo pipefail
    result/bin/lilac "$@"
  '';
in [ logo lilac lilac-nix ]

You can use the traditional nix-shell command to use this environment, or use Direnv (recommended), which will automatically reset environment variables in your existing shell whenever you enter the project root. Direnv will also automatically reload shell.nix if any of it or its dependencies change, based on the .envrc file below:

#!/usr/bin/env bash
use nix
watch_file lilac.cabal
watch_file shell.nix
watch_dir nix

6 Makefile

We use the traditional make command to encode some common operations. This way we don't have to memorize exactly how to invoke certain commands, and we can also use the filesystem as a cache to try to avoid re-computing already-computed outputs. See Sentinels for a discussion of this caching behavior.

make:Project setup
make:Org files
make:Run tests
make:Run linters
make:Compile sources
make:Tangle sources
make:Generate SVGs
make:Weave
make:Publish
make:Start frontend development environment (CLJS REPL)
make:Build prod JS
make:Build lilac with Cabal
make:Install lilac with Cabal
make:Build lilac with Nix
make:Install lilac with Nix
make:Update Nix dependencies
make:Clean

6.1 Sentinels

In a Makefile, given a rule

thing: dep1 dep2
	do-something

make will assume that the output of do-something will be a file named thing and will only run do-something if either dep1 dep2 have a newer timestamp than thing.

For cases where do-something does not result in generating the file thing, we use "sentinel" files (files that we touch at the end of a build rule) so that make won't re-run the rule if it sees that the sentinel file is still newer than the listed dependencies.

gitignore:Makefile sentinel:lint
gitignore:Makefile sentinel:build
gitignore:Makefile sentinel:build-release
gitignore:Makefile sentinel:tangle
gitignore:Makefile sentinel:weave
publish
install
install-release

6.2 Project setup

Make Git order the diffs according to git-orderfile.

gitconfig:
	git config diff.orderfile .git-orderfile
.PHONY: gitconfig

6.3 Org files

The Org files are all of the files that make up this project. We could discover these files like in tangle-loop.sh, but we list them out here explicitly for now.

org_files = \
	index.org \
	cli.org \
	build.org \
	compile.org \
	parse.org \
	tangle.org \
	types.org \
	weave.org \
	serve.org \
	fe.org \
	user-manual.org \
	plan.org \
	release-notes.org
backend_org_files = $(filter-out fe.org,$(org_files))

Because we tangle Lilac with itself, we want to separate out just the tangling mechanism from the rest. For this purpose we have backend_org_files. This way, we can isolate whether Lilac can tangle itself correctly before worrying about also tangling the frontend bits. See make:Tangle sources for this separation.

6.4 Run tests

The tests are discovered by Hspec automatically.

{-# OPTIONS_GHC -F -pgmF hspec-discover #-}

This allows cabal test to run all of the tests without having to specify them manually.

We make use of a spec hook to run the discovered tests in parallel.

module SpecHook (hook) where

import Test.Hspec (Spec, parallel)

hook :: Spec -> Spec
hook = parallel

The Makefile rule just calls cabal test for running these tests. The options passed to cabal test are recommended by Hspec. In particular, the --format=progress option condenses the output to just show periods (.) when a test case is passing, because most of the time, most tests are passing and it's too verbose to list all passing tests on each line (as is the default behavior).

test: $(org_files)
	cabal test --test-show-details=direct --test-option=--format=progress
	touch test

6.5 Run linters

We use the pre-commit framework to organize all of our linters into a single place, so that we can just use the pre-commit tool to execute them all. See Linters via pre-commit for details on what the various linters do.

lint: $(archive_file)
	make:lilac lint
	pre-commit run --all-files
	touch lint # 15

Aside: note that pre-commit isn't defined as a tool in the nix:devShellEnv. This is because we only define it in the mkShell wrapper in nix:devShell derivation.

Because running the linter doesn't produce any evidence on disk of a successful linter run, we have to do this ourselves with touch lint at 15. And because we don't want this file in nix:lilacSource, we make Git ignore it here:

lint

6.6 Compile Lilac with Lilac

Lilac can compile Org files into object files (*.lobj) and use those object files to create archive files (*.larc). Compilation helps us catch errors within object files, especially for finding broken (internal) links to other Org file headings, code blocks, etc. See Compile for a deeper discussion.

6.6.1 Object files

obj_files = $(org_files:%.org=%.lobj)

make:Batch compile Org files
make:Build archive

The naive way of building object files involves a rule like this:

%.lobj: %.org
	lilac compile $<

But this doesn't work well, because it will invoke the lilac CLI multiple times, once for each Org file that needs to be compiled because the : creates individual targets. This is why we have make:Batch compile Org files instead:

$(obj_files) &: $(org_files)
	lilac compile $?
	touch $(obj_files) # 16

This batch compilation works by specifying the &: (grouped targets). This way, if any object file is outdated (older than the org_files), we can invoke lilac compile ... just once on all needing-to-be-compiled Org files (specified with the $? automatic variable). This doesn't matter too much when running the tangle-loop.sh because that reacts as soon as one of the Org files is saved, but it does matter more when we compile everything from scratch (such as when we wipe the object files as a sanity check).

We have to update the timestamps of all object files at the end of the batch compilation at 16. To see why this is needed, consider the following. Suppose the timestamps could be like this for the Org files:

If we now see that b.org is newer, and then run lilac compile b.org, we'll only update the b.lobj timestamp (say, to 2).

However, now a.lobj has an older timestamp of 0, compared to b.org which still has a timestamp of 1. And so Make will think it will need to run the batch compilation again (it will think this way forever until we manually also update the timestamp of a.org). To avoid this scenario from happening, we run touch $(obj_files) so that now all objects get the timestamp update. So in this new scenario, the timestamps will look like

and all of the object files will be newer than the Org files, which will correctly inform Make that it won't need to run batch compilation again.

6.6.2 Archive

The make:Build archive rule says we should rebuild lilac.larc if any of the object files are missing or newer than lilac.larc.

archive_file := lilac.larc

$(archive_file): $(obj_files)
	lilac archive $^ --out-file $@

In order to create the lilac.larc archive file of the entire Lilac project, for example, do make lilac.larc.

6.6.3 Cleanup

The clean rule cleans up compilation artifacts, for debugging.

clean:
	rm -rf \
		$(archive_file) \
		$(obj_files)
.PHONY: clean

This rule is used in f:tangle-loop:clean_build_files.

6.7 Tangle Lilac with Lilac

We can tangle Lilac with itself.

tangle: $(archive_file)
	lilac tangle $(archive_file) --out-dir .
	touch tangle

The lilac script is the one we created in nix/scripts.nix to automatically build Lilac with Cabal.

The tangle file is a sentinel file to help GNU Make avoid running the tangling operation if there is nothing to do, very much like gitignore:Makefile sentinel:lint.

tangle

Making Lilac tangle itself is advantageous because it's faster than running org-babel-tangle inside Orgmode in Emacs. However, doing this means we lose out on being able to run the tangling function automatically on save, which we historically did with org-auto-tangle by having #+auto_tangle: t in our Org file.

So we need another way of tangling every time this file (build.org) is changed. We can do that with Watchman, but that would be unnecessarily heavy-handed because we don't need to watch a directory or multiple files. Instead we can just have a script which sleeps one second and checks the timestamp of build.org (and other files we care about), and if it's different than before, tangle it (and also run tests and weave and lint as needed). That's what we do in tangle-loop.sh.

set -euo pipefail

SCRIPT_ROOT="$(dirname "$(realpath "$0")")"

f:tangle-loop:get_tangled_program_files
f:tangle-loop:clean_build_files
f:tangle-loop:reset_tangled_program_files
f:tangle-loop:reset_everything
f:tangle-loop:tangle
f:tangle-loop:run_all_checks
f:tangle-loop:watch

main()
{
	__test_hs=1
	__want_to_lint=1
	if [[ "${1:-}" == "fe" ]]; then
		__test_hs=0
		shift
	fi
	if [[ "${1:-}" == "no-lint" ]]; then
		__want_to_lint=0
		shift
	fi
	__ok_to_lint=1

	readarray -t org_files < <(find . -type f -regex '.+\.org$')
	watch "${org_files[@]}"
}

main "$@"

The main() entrypoint above sets up two global variables __test_hs and __ok_to_lint, before entering the loop in f:tangle-loop:watch.

watch()
{
	local s1
	local s2
	local files=("$@")

	s1="$(stat -c %Y "${files[@]}")"

	run_all_checks # 17
	while true; do
		s2="$(stat -c %Y "${files[@]}")"
		if [[ ! "${s1}" == "${s2}" ]]; then
			s1="${s2}"
			run_all_checks
		fi
		sleep 1
	done
}

f:tangle-loop:watch checks for timestamps on the files we're interested in every second, and runs f:tangle-loop:run_all_checks if it detects a modification timestamp change.

run_all_checks()
{
	tangle || true  # 18
	if tangle; then # 19
		if ((__test_hs)); then
			if ! make -C "${SCRIPT_ROOT}" test; then # 20
				reset_everything
				return
			else
				__ok_to_lint=1
			fi
		fi
	fi

	echo "----------------------------------------"

	if ! time make -C "${SCRIPT_ROOT}" weave; then
		reset_everything
		return
	else
		__ok_to_lint=1
	fi

	echo "----------------------------------------"

	if ((__want_to_lint)) && ((__ok_to_lint)); then
		make -C "${SCRIPT_ROOT}" lint || true
	fi
}

f:tangle-loop:run_all_checks tangles first, runs all tests, weaves, and finally runs the linter. If any of the earlier steps fail, then the later steps abort execution.

f:tangle-loop:run_all_checks calls tangle() twice (at 18 and 19), because if we modify Lilac's behavior (by editing a Haskell block), the newly-compiled Lilac will be created in the first invocation of tangle(). The second invocation will then use the new Lilac version to tangle itself again. It's a simple (albeit ugly) way of ensuring that we're always tangling with the latest Lilac version.

f:tangle-loop:run_all_checks also runs the unit tests at 20, as a convenience function, but only if the previous call to tangle succeeds. This way, we only try to run the unit tests if Lilac can tangle itself. As for the unit tests, the exit code is ignored with || true because oftentimes the tests break when we're adding new tests (without this, we would break out of the while loop, which is not desirable).

The initial call to run_all_checks() in the watch() function (outside of the while loop) at 17 is there for convenience, just in case we've modified the build.org file outside of a tangle-loop.sh session.

tangle()
{
	mapfile -t tangled_program_files < <(get_tangled_program_files) # 21
	if ! time make -C "${SCRIPT_ROOT}" tangle; then
		reset_everything
		return 1
	fi
	__ok_to_lint=1
	return 0
}

f:tangle-loop:tangle tangles source code; however it also resets the tangled_program_files global variable, because it may be the case that the program files are slightly different than before when we tangle new code blocks from an Org file.

If anything goes wrong during tangling, we call f:tangle-loop:reset_everything to reset all tangled code blocks to their previously known (good) state in f:tangle-loop:reset_tangled_program_files, and also delete any intermediate build artifacts to get us to a clean state.

reset_everything()
{
	reset_tangled_program_files
	clean_build_files
}

f:tangle-loop:reset_everything calls reset_tangled_program_files() and also clean_build_files(), which cleans up incremental build artifacts like *.lobj and *.larc files. This is needed if we make a change to the structure of data:OrgDoc or anything else that's serialized out to a file and read back in. For example, if we rename a field to a new name, then any existing build artifact with the old name will become invalid and fail to parse; in this case the best thing to do is remove the old build artifact (via clean_build_files()) and build these files again from scratch.

The f:tangle-loop:reset_tangled_program_files function assumes that the current Git commit has a working Lilac version. Based on this assumption, it runs git checkout on tangled files at 22 if there are any errors. So if it turns out we edited Lilac's own sources and broke the build, we want to undo the broken (tangled) changes, which we can do by simply resetting the Haskell source code back to the checked-in state with git checkout. This is safe because the changes we want to try out are always stored in the Org files (build.org, etc), because we use Literate Programming. If we didn't use Literate Programming we would be very cautious of resetting any source code files, but we don't have that worry here.

reset_tangled_program_files()
{
	__ok_to_lint=0
	echo >&2 "Error encountered; cleaning Haskell tangle paths."
	>/dev/null git checkout -- "${tangled_program_files[@]}" \ 22
		Makefile \
		.gitignore \
		nix
}

Note that the tangled_program_files is created by calling f:tangle-loop:get_tangled_program_files at 21, which only looks at the tangled Cabal, Haskell, Nix, or shell files. This means you can make changes to the original Org files to make Lilac compile or pass tests again, as needed, by tweaking those code blocks responsible for these file types.

get_tangled_program_files()
{
	git -C "${SCRIPT_ROOT}" grep -E '^#\+header: :tangle' -- '*.org' \
		| awk '{print $3}' \
		| grep '\.\(cabal\|hs\|nix\|sh\)$'
}

Lastly, f:tangle-loop:clean_build_files just runs make:Clean.

clean_build_files()
{
	echo >&2 "Cleaning build files."
	make -C "${SCRIPT_ROOT}" clean
}

6.8 Weave Lilac with Lilac

For weaving, we use the doc directory.

weave: $(archive_file) $(image_svgs)
	lilac weave $< --out-dir . --write-css # 23
	touch weave

Because f:lilacWeave writes CSS to the css subdirectory, the CSS ends up getting written to css/default.css. This css directory will not exist in CI (when Lilac is cloned and built on a fresh machine), so we have to create it. Otherwise, the make:Build prod JS rule will fail because the cfg:figwheel-main.edn expects the css directory to exist.

We make Git ignore the weave file because it's a sentinel file.

weave

We add the --write-css flag at 23 because we want to regenerate (every time) the CSS file from Haskell sources in f:genCss, in case anything in there changed. However we don't do the same for JavaScript (we don't supply --write-js), because we don't want to overwrite the same js/lilac.js path used by make:Start frontend development environment (CLJS REPL).

To clarify, there are 2 flavors of JavaScript artifacts, development and production. The development version is not a single JavaScript file and is not optimized; this allows us to get a ClojureScript REPL where we can play around with a direct connection to a browser. The production flavor is a single, optimized JavaScript file and does not have ClojureScript REPL support. The production flavor is copied out to another file js/lilac.js.injectme by sh:update-injectable. It is this separate file which is injected into Haskell in f:lilacJs, so that Lilac is able to generate this file on its own.

When we're developing Lilac with Lilac, we generally want to only look at the development version of the JavaScript because this allows REPL interaction. This means we cannot use the --write-js flag to Lilac in make:Weave, because doing so would make Lilac overwrite the development file (which shares the same js/lilac.js target path that Lilac targets with the --write-js flag) with its own (pre-baked) JS file.

For serving the woven (development) HTML pages through a server (instead of making the browser fetch it via the file:// protocol), see Frontend.

6.9 Build lilac

There are two ways to build lilac. The first option, build, builds with cabal, Haskell's default tooling for building Haskell programs.

build: $(org_files) lilac.cabal
	cabal build
	touch build

This results in building the binary. The binary will end up in an obscure path like ./dist-newstyle/build/.../lilac. You can get this path with cabal list-bin lilac. To run the binary from that path, use cabal run lilac -- [LILAC OPTIONS...] where [LILAC OPTIONS ...] are the options for lilac. The double dash is required to tell cabal to stop interpreting command line options (to leave them to be handled by lilac).

The above ceremony with cabal run might be a little bit annoying during development. This is why nix/scripts.nix includes lilac which acts as a shell wrapper around those various cabal-centric options for us. Specifically, the cabal-run-invocation uses some default flags:

• -j to use all available CPU cores for cabal

• +RTS to mark the start of runtime system options

• -N to use all available CPU cores for lilac

• -RTS to mark the end of runtime system options

cabal run lilac -v0 -j -- +RTS -N -RTS "$@" 24

We have to ignore the build file because it's just a sentinel file.

build

Another option is to build with Nix, which wraps everything up for public consumption, as a Nix package.

LILAC_GIT_VERSION := $(shell git describe --abbrev=10 --always --dirty)
build-release: $(org_files) lilac.cabal
	nix-build ./nix/release.nix \
		--argstr lilacGitVersion $(LILAC_GIT_VERSION)
	touch build-release

build-release

You can run the version of Lilac built with Nix via result/bin/lilac, which we wrap as lilac-nix in the nix/scripts.nix as well.

6.10 Install lilac

You can install Lilac with Cabal or Nix. If you're on Nix already, using Nix is probably better because it'll show up in your nix-env -q.

6.10.1 Cabal

install: $(org_files) lilac.cabal
	cabal install --overwrite-policy=always
	touch install

6.10.2 Nix

For Nix, we just have to reuse what's already been built with make:Build lilac with Nix.

install-release: $(org_files) result build-release
	nix-env -i ./result
	touch install-release

6.11 Publish to lilac.funloop.org

Makefile:create-tarball

SRHT_TOKEN != cat .srht-pages-token 2>/dev/null || echo ""

publish: weave site.tar.gz
	@if [ -z "$(SRHT_TOKEN)" ]; then \
		echo "Error: .srht-pages-token is empty or missing." >&2; \
		exit 1; \
	fi
	@echo "Uploading site to SourceHut Pages..."
	@curl --oauth2-bearer "$(SRHT_TOKEN)" \
		-Fcontent=@site.tar.gz \
		https://pages.sr.ht/publish/lilac.funloop.org
	touch publish

The tarball is mainly composed of the site (woven contents) plus the vendored stuff.

site.tar.gz: $(archive_file) weave vendor/mathjax vendor/mathjax-newcm-font vendor/source-fonts
	rm -f site.tar site.tar.gz
	lilac weave $< --out-dir site --write-css --write-js
	tar cvf site.tar -C site .
	tar rvf site.tar vendor
	gzip site.tar

vendor/source-fonts:
	./vendor-source-fonts.sh

vendor/mathjax vendor/mathjax-newcm-font &:
	./vendor-mathjax.sh

The vendoring is needed because SourceHut does not allow us to pull in scripts from CDNs. We have to be self-contained. Lilac has a lilac init --only-local-imports command which gives us some scripts to download MathJax and also Source fonts.

set -euo pipefail

MAIN_PACKAGE="mathjax"
FONT_PACKAGE="mathjax-newcm-font"
VERSION="4.1.2"

mkdir -p vendor
pushd vendor

download_tarball()
{
	local package
	local version
	local url_prefix
	local tarball
	package="${1:-}"
	version="${2:-}"
	tarball="${package}-${version}.tgz"
	url_prefix="${3:-}"

	if [[ -d "${package}" ]]; then
		echo "${package} directory already exists"
		return 1
	fi

	curl -OL "${url_prefix}/-/${tarball}"
	tar xzvf "${tarball}"
	mv package "${package}"
	rm "${tarball}"
}

download_tarball \
	"${MAIN_PACKAGE}" \
	"${VERSION}" \
	"https://registry.npmjs.org/mathjax"
download_tarball \
	"${FONT_PACKAGE}" \
	"${VERSION}" \
	"https://registry.npmjs.org/@mathjax/${FONT_PACKAGE}"

set -euo pipefail

url_sans="https://raw.githubusercontent.com/adobe-fonts/source-sans/3.052R"
url_serif="https://raw.githubusercontent.com/adobe-fonts/source-serif/4.005R"
url_code="https://raw.githubusercontent.com/adobe-fonts/source-code-pro/2.042R-u%2F1.062R-i%2F1.026R-vf"

mkdir -p vendor/source-fonts
pushd vendor/source-fonts

curl -OL "${url_sans}/WOFF2/VF/SourceSans3VF-Italic.otf.woff2"
curl -OL "${url_sans}/WOFF2/VF/SourceSans3VF-Upright.otf.woff2"

curl -OL "${url_serif}/WOFF2/VAR/SourceSerif4Variable-Italic.otf.woff2"
curl -OL "${url_serif}/WOFF2/VAR/SourceSerif4Variable-Roman.otf.woff2"

curl -OL "${url_code}/WOFF2/VF/SourceCodeVF-Italic.otf.woff2"
curl -OL "${url_code}/WOFF2/VF/SourceCodeVF-Upright.otf.woff2"

cat <<EOF >source-fonts.css
text:source-fonts.css
EOF

Finally we need to actually use these vendored assets by referring to them in every woven HTML file. That's what we do in custom-html-head.

[htmlHead]
onlyLocalImports = true
injection = """
<link rel="stylesheet" href="/vendor/source-fonts/source-fonts.css">
<script>
  window.MathJax = {
      loader: { paths: { "mathjax-newcm": "/vendor/mathjax-newcm-font" } },
      output: { font: "mathjax-newcm" },
  tex: {
      tags: 'ams'
  },
  options: {
      ignoreHtmlClass: 'verbatim|code'
  }
};
</script>
<script id="MathJax-script" async="" src="/vendor/mathjax/tex-mml-chtml.js"></script>
"""

7 Linters via pre-commit

Git has an elaborate system of hooks that it can run every time you do something important with it. A common hook is a pre-commit hook, which will run just before creating a new commit. It's useful to add linters and other things in here.

There is a project called https://pre-commit.com which is in the business of standardizing Git hooks. One of the selling points is that for the various formatters and linters that it already supports, by default it only runs them against changed files in the Git index. This optimization is annoying to set up separately for each formatter or linter, so it's nice to let pre-commit handle it for us.

In addition, pre-commit supports all of the usual Git hooks, not just pre-commit. This can come in handy..

7.1 git-hooks.nix

The https://github.com/cachix/git-hooks.nix project brings pre-commit into Nix, so that we can define all of the hook infrastructure in a reproducible way with Nix (such as using specific versions of linters, formatters, etc).

For us we already have the set of linters and formatters we are interested in defined inside our development shell environment in nix:devShellEnv, so we tell git-hooks to reuse those same tools.

While formatting isn't important for tangling (the computer doesn't care about formatting), it's important for weaving because we want code blocks in woven output to still look decent and consistent.

gitHooksPkgs = import (import ./sources.nix)."git-hooks.nix";
pre-commit-check = gitHooksPkgs.run {
  src = projectRoot;
  hooks = {
    hook-editorconfig-checker
    hook-nixfmt
    hook-ormolu
    hook-shfmt
    hook-shellcheck
    hook-parinfer-rust
    hook-unit-test
    hook-cspell
    hook-sort-file-contents
  };
};

toolOpts = opts: builtins.concatStringsSep " " opts;

By default the above tools will only run on changed files going into a new commit (and these changed file names will be passed to each check). If you want to run against all files to check for any existing violations, run pre-commit run --all-files.

The pre-commit tool writes a configuration file at .pre-commit-config.yaml. We ignore this for Git because it has local machine-specific paths in it (and because it's also just autogenerated by pre-commit based on our Nix configuration above, thereby making it easy enough to reproduce as needed if we delete it by accident).

.pre-commit-config.yaml

7.2 EditorConfig

EditorConfig is a specification for setting some universal properties about files in a project.

root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true

[Makefile]
indent_style = tab
tab_width = 8

[*.sh]
indent_style = tab
tab_width = 8
[vendor-source-fonts.sh]
indent_style = unset

[js/lilac.js.injectme] # 25
insert_final_newline = false
trim_trailing_whitespace = false
end_of_line = unset
charset = unset

The root = true is required because the EditorConfig specification is version-control agnostic and does not understand Git's notion of a project or repository root. When EditorConfig plugins check whether a file complies with EditorConfig configuration, it simply searches for the current or any parent folders for the .editorconfig file. Setting the root = true tells the plugin to stop searching upwards beyond the current file.

25 prevents EditorConfig from trying to interpret the lilac.js.injectme file (see gitattributes-ignore-compiled-js) as a text file.

editorconfig-checker = {
  enable = true;
  package = devShellEnv;
};

We use the standalone editorconfig-checker Go tool to check for conformance.

pkgs.editorconfig-checker

7.3 Nix

nixfmt is the official formatter for the Nix language. We use this to format all of our Nix files.

nixfmt-rfc-style = {
  enable = true;
  package = devShellEnv;
  excludes = [ "nix/sources.nix" ];
  entry = let opts = [ "--check" ];
  in "${devShellEnv}/bin/nixfmt ${toolOpts opts}";
};

ourHaskell.nixfmt

7.4 Haskell

Haskell has quite a few different formatters in the ecosystem. We use Ormolu because it is simple (no configuration is allowed), and also tries to minimize leading indentation.

ormolu = {
  enable = true;
  package = devShellEnv;
  entry = let opts = [ "--mode" "check" "--check-idempotence" ];
  in "${devShellEnv}/bin/ormolu ${toolOpts opts}";
};

ourHaskell.ormolu

7.5 Shell

We use shfmt for formatting and ShellCheck for linting.

shfmt = {
  enable = true;
  package = devShellEnv;
  entry = let
    opts =
      [ "--indent" "0" "--binary-next-line" "--func-next-line" "--diff" ];
  in "${devShellEnv}/bin/shfmt ${toolOpts opts}";
};

pkgs.shfmt

shellcheck = {
  enable = true;
  package = devShellEnv;
};

pkgs.shellcheck

7.6 ClojureScript

For ClojureScript we use parinfer-rust to check for indentation issues.

pkgs.parinfer-rust

parinfer-rust = {
  name = "parinfer-rust";
  enable = true;
  always_run = true;
  entry = "./parinfer-rust.sh";
  files = "\\.(clj|cljs|cljc|edn)$";
};

set -euo pipefail

RET=0

main()
{
	for file in "$@"; do
		# shellcheck disable=SC2094
		if ! parinfer-rust <"$file" | diff -u "$file" - 2>&1; then
			RET=1
		fi
	done

	exit ${RET}
}

main "$@"

7.7 Audit tangled vs tracked files

We should detect whether we've accounted for all tracked files via Literate Programming. That is, we want to detect any files that are checked into source control but which are not being tangled from an Org file, because that's an anti-pattern and goes directly against LP principles.

Of course, there can be exceptions, such as generated code (code spat out by a binary), binary files (like images), and metadata such as .gitmodules. We account for these exceptions in the arguments to git-ls-files in toml:RawProjectConfig → 1.

The concerns above are addressed by the lint subcommand of the Lilac CLI, as described in Linting tangled paths.

Originally we integrated lilac lint into pre-commit, but then moved away from it because it was observed that pre-commit's invocation of lilac lint would bust the Cabal cache, even if there were no changes to Lilac's source code. So instead we just call it as a separate item in the Makefile.

lilac lint $(archive_file)

7.8 Run lilac unit test before pushing

Running unit tests requires compiling the tests, which takes more than a few seconds. So it's a little bit resource intensive and we don't want to run it on every commit. So we instead run it when we git push to save our work to a remote, by putting in the pre-push stage at 26.

unit-test = {
  name = "unit-test.sh";
  enable = true;
  always_run = true;
  entry = "./unit-test.sh";
  pass_filenames = false;
  stages = [ "pre-push" ]; # 26
};

set -euo pipefail

SCRIPT_ROOT="$(dirname "$(realpath "$0")")"

shell-func-is-wip

main()
{
	>/dev/null pushd "${SCRIPT_ROOT}"

	shell-skip-if-wip
	shell-fail-if-dirty-tree
	unit-test-run
}

main "$@"

We don't want to run unit tests if the current commit (HEAD in Git) is a work-in-progress (WIP) commit.

is_wip()
{
	git show -s --format=%s HEAD \
		| grep -qi '^\(update\|wip\|fixup\|f \|s \)'
}

if is_wip; then
	echo >&2 "skipping (WIP commit)"
	return
fi

We can't just invoke make test directly to run unit tests for this hook, because it may be the case that the working tree is dirty. If the tree is dirty, the results of the unit test may no longer reflect what we are pushing to the remote (which would only be the clean commits, not any of the dirty stuff), so we need to abort if the tree is dirty.

if [[ -n $(git status --porcelain) ]]; then
	echo >&2 "$0: dirty tree"
	return 1
fi

For invoking the unit tests, we run make test (like we normally do during development).

if make test; then
	echo >&2 "OK"
else
	echo >&2 "$0: failed"
	return 1
fi

7.9 Spellcheck (cspell)

Use cspell for checking spelling. cspell understands Org syntax, so it knows to skip over words inside code blocks and code or verbatim styles. This obviously helps with only checking spelling in the places we care about.

cspell = {
  enable = true;
  package = devShellEnv;
  files = "\\.org$";
};

---
$schema: https://raw.githubusercontent.com/streetsidesoftware/cspell/main/cspell.schema.json
version: '0.2'
dictionaryDefinitions:
    - name: lilac-dict
      path: './lilac-dict.txt'
      addWords: true
dictionaries:
    - lilac-dict
patterns:
  - name: org-block
    pattern: '^#\+begin_[\s\S]*?^#\+end_'
  - name: org-metadata
    pattern: '^#\+[\S]+[\s\S]*?$'
  - name: org-links
    pattern: '\[\[[\s\S]*?\]\]'
  - name: org-emphasis-code
    pattern: '~[^~\n]+?~'
  - name: org-emphasis-verbatim
    pattern: '=[^=\n]+?='
  - name: org-citations
    pattern: '\[cite[^\]]*\]'
  - name: mathjax-display
    pattern: '\\\[[\s\S]*?\\\]'
  - name: mathjax-inline
    pattern: '\\\([^\n]*?\\\)'

overrides:
  - filename: "**/*.org"
    ignoreRegExpList:
      - org-block
      - org-metadata
      - org-links
      - org-emphasis-code
      - org-emphasis-verbatim
      - org-citations
      - mathjax-display
      - mathjax-inline

ignorePaths:
    - '**/*.cljs'
    - '**/*.css'
    - '**/*.hs'
    - '**/*.js'
    - '/lilac-dict.txt'

pkgs.cspell

7.9.1 Custom vocabulary

Use a custom "Lilac" vocabulary that has some words that cspell doesn't recognize by default.

Aeson
attrset
backlink
backlinks
Citeproc
Citeproc's
combinators
csljson
cweave
CWEB
dedented
detangle
Direnv
Elisp
Figwheel
GTOC
Hackage
Hspec
jailbreaking
keywordize
larc
lobj
monoid
Nixkell
Nixpkgs
nonparent
orderfile
orderfiles
Orgmode
Pandoc
pathspec
pathspecs
pikchr
pilcrow
pkgs
sandboxing
semigroup
Skylighting
specialisations
typeclass
weavable

7.10 Sort file contents

pre-commit comes with a hook to check that a file's contents are sorted. This is useful for checking that lilac-dict stays sorted.

sort-file-contents = {
  enable = true;
  files = "^lilac-dict\\.txt$";
  settings = { ignore-case = true; };
};

8 SourceHut CI builds

The CI runs the linter, builds the production binary, and also runs tests.

image: nixos/latest
secrets:
  - 5de45cde-db04-468b-94dd-7ef3a53eeace
sources:
  - git@git.sr.ht:~listx/lilac
tasks:
  - lint: |
      cd lilac
      nix-shell --run "make lint"
  - weave: |
      cd lilac
      rm -f image/*.svg
      nix-shell --run "make weave"
  - compile-prod-cljs: |
      cd lilac
      nix-shell --run "make prod-cljs"
  - verify-git-health: |
      cd lilac
      git diff
      if [[ -z "$(git status --porcelain)" ]]; then
        echo "Clean (no untracked files either)."
      else
        echo "Modified or untracked files present."
        exit 1
      fi
  - build: |
      cd lilac
      nix-shell --run "make build-release"
  - test: |
      cd lilac
      nix-shell --run "make test" # 27
  - print_lilac_version: |
      lilac/result/bin/lilac --version

Because the build-release rule involves building and running the test suite, the test task 27 results in a NOP. However we still keep it here just in case the behavior of make:Build lilac with Nix changes.

9 Cabal configuration

For the Haskell side of things, we use the traditional Cabal configuration file because it's simpler than adding Stack into the mix.

The cabal binary is available to us via the cabal-install package in our nix:tools.

pkgs.cabal-install

For the Haskell compiler, we use the following GHC version:

9.12.4

The above version is used throughout other configuration files and such elsewhere in this document. Just search for GHC version to see those references.

To update GHC to a newer version, we have to first check the path pkgs/development/compilers/ghc in the Nixpkgs Git branch where GHC compiler versions are defined (to see which GHC versions are available), then update the branch we are using with Niv (see make:Update Nix dependencies).

cabal-version:  2.2

name:           lilac
version:        Lilac version
license:        GPL-2.0-only
license-file:   LICENSE
build-type:     Simple
extra-source-files:
  js/lilac.js.injectme

library lilac-internal
  exposed-modules:
      Lilac.Types.Internal
  hs-source-dirs:
      src-internal
  default-extensions:
      Default GHC extensions
  ghc-options:
      GHC warning flags
  build-depends:
      aeson
    , base >=4.7 && <5
    , relude
    , text
  default-language: Language edition

library
  other-modules:
      Paths_lilac
  exposed-modules:
      Lilac.Version
      Lilac.Compile
      Lilac.Parse
      Lilac.Tangle
      Lilac.Types
      Lilac.Weave
      Lilac.Weave.Css
      Lilac.Util
      Lilac.Serve
  hs-source-dirs:
      src
  default-extensions:
      Default GHC extensions
  ghc-options:
      GHC warning flags
  build-depends:
      aeson
    , async
    , base >=4.7 && <5
    , bitwise
    , blaze-html
    , bytestring
    , citeproc
    , clay
    , containers
    , directory
    , filepath
    , file-io
    , fsnotify
    , http-types
    , lilac-internal
    , toml-parser
    , lucid2
    , megaparsec
    , network-uri
    , process
    , relude
    , skylighting
    , stm
    , template-haskell
    , text
    , time
    , transformers
    , uuid
    , wai
    , wai-websockets
    , websockets
  default-language: Language edition

executable lilac
  main-is: Main.hs
  other-modules:
      Paths_lilac
  hs-source-dirs:
      bin
  default-extensions:
      Default GHC extensions
      TemplateHaskell
  ghc-options:
      GHC warning flags
      GHC threaded flags
  build-depends:
      aeson
    , ansi-terminal
    , base >=4.7 && <5
    , bytestring
    , containers
    , directory
    , filepath
    , file-embed
    , lilac
    , network-uri
    , optparse-applicative
    , megaparsec
    , relude
    , stm
    , text
    , toml-parser
    , typed-process
    , uuid
    , wai-app-static
    , warp
  default-language: Language edition

test-suite spec
  type: exitcode-stdio-1.0
  main-is: Spec.hs
  other-modules:
      Lilac.CompileSpec
      Lilac.ParseSpec
      Lilac.TangleSpec
      Lilac.WeaveSpec
      SpecHook
      Paths_lilac
  hs-source-dirs:
      test
  default-extensions:
      Default GHC extensions
  ghc-options:
      GHC warning flags
      GHC threaded flags
  build-depends:
      base >=4.7 && <5
    , citeproc
    , containers
    , filepath
    , hspec
    , hspec-megaparsec
    , lilac
    , lilac-internal
    , megaparsec
    , network-uri
    , relude
  default-language: Language edition

We save ourselves some repetition (at least in the Org file) by declaring default GHC extensions in one place:

BangPatterns
BinaryLiterals
ConstraintKinds
DataKinds
DefaultSignatures
DeriveAnyClass
DeriveDataTypeable
DeriveFoldable
DeriveFunctor
DeriveGeneric
DeriveTraversable
DerivingStrategies
DerivingVia
DuplicateRecordFields
EmptyCase
FlexibleContexts
FlexibleInstances
FunctionalDependencies
GeneralizedNewtypeDeriving
ImportQualifiedPost
InstanceSigs
MultilineStrings
MultiParamTypeClasses
MultiWayIf
NamedFieldPuns
NoFieldSelectors
NoImplicitPrelude
NumDecimals
NumericUnderscores
LambdaCase
OrPatterns
OverloadedLabels
OverloadedRecordDot
OverloadedStrings
PatternSynonyms
RankNTypes
ScopedTypeVariables
StandaloneDeriving
StandaloneKindSignatures
TupleSections
TypeApplications
TypeOperators
ViewPatterns

9.1 GHC compiler warnings

Our strategy with GHC warnings is to first turn on as many as possible (ghc:Warn everything), followed by some flags to turn off harmless or overly pedantic warnings.

ghc:Warn everything
ghc:Disable safe Haskell
ghc:Disable missed specialisations
ghc:Disable missed extra shared libs

We turn on all warnings with -Weverything.

-Weverything

Now we disable some warnings that are not that important.

-Wno-unsafe -Wno-safe -Wno-missing-safe-haskell-mode

"Safe Haskell" is essentially a dialect of Haskell that has stronger guarantees about referential transparency. It's an interesting idea, but not one worth pursuing in this program.

-Wno-missed-specialisations
-Wno-all-missed-specialisations

These "missed specialisations" warnings are not really useful for us because we don't (as of yet) care too much about performance. Plus they seem to be about library functions we are importing.

-Wno-missed-extra-shared-lib

Since GHC 9.12.2, we've been getting warnings like the following on Mac OS. As it's benign, ignore these warnings.

<no location info>: warning: [-Wmissed-extra-shared-lib]
    dlopen(libHSunix-2.8.6.0-0912.dylib, 0x0005): tried: 'libHSunix-2.8.6.0-0912.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibHSunix-2.8.6.0-0912.dylib' (no such file), '/nix/store/b96n0hqpih5rb7cxw7cp8d1bwpq3bah5-ghc-9.12.2/lib/ghc-9.12.2/lib/aarch64-osx-ghc-9.12.2-94b7/libHSunix-2.8.6.0-0912.dylib' (no such file), '/nix/store/b96n0hqpih5rb7cxw7cp8d1bwpq3bah5-ghc-9.12.2/lib/ghc-9.12.2/bin/../lib/aarch64-osx-ghc-9.12.2-94b7/libHSunix-2.8.6.0-0912.dylib' (no such file), '$ORIGIN/../../../lib/aarch64-osx-ghc-9.12.2-94b7/libHSunix-2.8.6.0-0912.dylib' (no such file), '/usr/lib/libHSunix-2.8.6.0-0912.dylib' (no such file, not in dyld cache), 'libHSunix-2.8.6.0-0912.dylib' (no such file), '/usr/local/lib/libHSunix-2.8.6.0-0912.dylib' (no such file), '/usr/lib/libHSunix-2.8.6.0-0912.dylib' (no such file, not in dyld cache)
    It's OK if you don't want to use symbols from it directly.
    (the package DLL is loaded by the system linker
     which manages dependencies by itself).

9.2 GHC threaded runtime flags

-threaded
-rtsopts
-with-rtsopts=-N

The -threaded flag links the executable with the threaded runtime, which can manage multiple OS threads (as opposed to the default runtime which is single-threaded).

The -rtsopts option allows the processing of RTS control options given either on the command line or via the GHCRTS environment variable.

The -with-rtsopts=-N option lets the runtime choose how many simultaneous threads to use (at runtime). Typically this means using all cores on the machines.

9.3 Language edition

Several options are available to pick the "flavor" of Haskell, such as:

• Haskell98

• Haskell2010

• GHC2021

• GHC2024

We use the latest available edition, because we can.

GHC2024

9.4 Cabal project file

For options that should only affect Cabal builds, we want to specify such options inside cabal.project because then these settings won't contaminate cabal2nix (see nix:Haskell overlays).

packages: *.cabal

cabal-write-env-file
cabal-faster-dev-build
cabal-offline

Write the GHC environment file to disk. This file is useful for seeing the packages that are available to load inside ghci.

write-ghc-environment-files: always

Define -O0 to speed up Cabal builds.

optimization: False

Nix handles dependencies, so tell Cabal to work offline (it won't download any packages from Hackage).

active-repositories: none

9.5 Setup.hs file

This file is needed for Cabal.

import Distribution.Simple

main = defaultMain

10 Searching with ripgrep

The ripgrep (rg) tool is a popular tool used for searching text. While rg respects the .gitignore, we cannot solely rely on it because it will still search tangled source code. We want to tell rg to ignore those files because when we search for something we always want to look at the source of truth, which in most cases (except for generated code) is an Org file.

To make rg behave as we want, we have to provide it a separate .rgignore file.

*.cabal
*.hs
*.nix
*.lobj
*.sh
csl-styles
test/*

We can't put these inside .gitignore because doing so will remove these files from bundling it into a Nix package in nix:lilacSource (we actually want the tangled content because they are required for the build).

11 Ordering Git diffs

By default, Git orders diffs alphabetically by the filename (presumably to guarantee determinism when running git diff). However, you can change this behavior to sort the diffs in a different way, by using orderfiles (see man git-diff). Below is our orderfile:

# Org files
plan.org
release-notes.org
user-manual.org
index.org
build.org
types.org
parse.org
compile.org
tangle.org
weave.org
fe.org
cli.org
*.org

# Infrastructure
*.nix
*.json
*.sh
*.yml
.*

# Haskell
cabal.project
*.cabal
*.hs

# ClojureScript
*.edn
*.cljs

# Tests
test*

The most important files are the Org files, because they serve as the source of truth for Literate Programming in this project. So they appear first. The rest of the groupings are not strictly necessary, because the diffs there will be redundant with the diffs in the Org files; nevertheless we categorize them here for completeness.