Introduction

1 Lilac

Lilac is a literate programming [1] (LP) tool. Specifically it can read Orgmode files to tangle (i.e., extract) code blocks using Noweb [2] references, and also weave (i.e., publish) the same files to HTML for a more pleasant reading experience.

This document is mainly concerned with how Lilac is implemented. If you simply want to learn how to use Lilac, read the User Manual.

1.1 Version

The current Lilac version is

🔗

Lilac version ⇑ ⇪ ⇑

0.1.0.0

We follow the Haskell Package Versioning Policy.

1.2 Repository homepage

The Lilac source code can be found at

🔗

lilac-repo-url ⇑ ⇧

https://git.sr.ht/~listx/lilac

1.3 Project configuration for Lilac

We define the Project configuration for Lilac itself in here, as a TOML file.

🔗

toml:RawProjectConfig
🎯 Lilac.toml

name = "Lilac"
homepage = "lilac-repo-url"
repo = "lilac-repo-url"
version = "Lilac version"

nonTangledPathspecs = [ # 1
    ":(exclude,top).gitmodules", # 2
    ":(exclude,top)LICENSE",
    ":(exclude,top)*.html",
    ":(exclude,top,glob)image/*.svg",
    ":(exclude,top,glob)*.org",
    ":(exclude,top)image/PNG-Gradient.png", # 3
    ":(exclude,top)js/lilac.js.injectme", # 4
    ":(exclude,top)bib.json", # 5
    ":(exclude,top)csl-styles", # 6
    ":(exclude,top)nix/sources.json", # 7
    ":(exclude,top)nix/sources.nix",
    ":(exclude,top)resources/public" # 8
]

bibliography = ["bib.json"]
citationStyle = "csl-styles/ieee.csl"

custom-html-head

The nonTangledPathspecs are passed to Git to filter out some paths from git-ls-files, because these files are intentionally not tangled from Org files. See Linting tangled paths to see how this is used. The magic words exclude and top are documented under gitglossary.

2 skips .gitmodules because it's generated by Git.

7 skips files generated by Niv.

Image files are never tangled from Org files, so they're skipped 3. (Technically we could support translating text-based image descriptions into tangled image file outputs, but that's not supported.)

4 skips the generated JS file (built from ClojureScript).

5 skips the raw list of bibliographical references (written in CSL JSON format).

6 skips the CSL styles submodule repo.

8 skips the symlink which points back to the project root. This is used for convincing Figwheel that it's OK to serve files from the project root (it insists on only serving web assets from resources/public).

1.4 Project layout

This document references many things in other docs. All of the docs for this project are described below:

index.org (this doc)
🔗 Describes the overall project.
cli.org
🔗 Describes the implementation of the lilac CLI program.
build.org
🔗 Describes the build and development environment. Look here if you want to know how to get started hacking on this project.
parse.org
🔗 Describes the Org syntax parser. This parser allows us to create machine-friendly intermediate representations (an object (*.lobj) and archive (*.larc)) of an Org document, which are created by the compile and archive subcommands of the CLI, respectively.
types.org
🔗 Describes the types used to represent the machine-friendly intermediate representation of an Org document. The types here are used heavily by parse.org as well as all of the other pieces.
compile.org
🔗 Describes the compilation processes, which power the compile and archive subcommands. These steps check for things like broken links and the absence of any cycles in Noweb references.
tangle.org
🔗 Describes the tangle process. Most of the hard work is done during compilation, so this document is pretty short.
weave.org
🔗 Describes the weave process, which converts all of the Org documents inside an archive (*.larc) into an HTML page, styled with CSS.
fe.org
🔗 Describes the frontend framework for making the HTML page interactive with JavaScript, to go beyond what HTML and CSS can do for us.
serve.org
🔗 Describes the inner workings of the serve subcommand.

We also have additional documentation surrounding the Lilac project, not just how it's implemented.

user-manual.org
🔗 Lilac's User Manual. This is the starting place for newcomers who just want to use the lilac CLI program, without having to read all about how it's implemented internally.
plan.org
🔗 Explains where we're headed as a project.
release-notes.org
🔗 Records noteworthy changes between releases.

1.4.1 Literate programming and Noweb

This is a literate document, so prose explanations are mixed in with code. Typically, all of the content of a code block will be explained by nearby prose. Code blocks will typically make use of Noweb references.

For example, in

🔗

Foo
<<Lilac version>>

the reference Lilac version (enclosed within << and >>) is a Noweb reference. When the above is woven to HTML with Noweb enabled (the default), we get

🔗

Foo
Lilac version

which, if you're viewing this in HTML, should show a link for the Lilac version part.

If a Noweb reference is not explained nearby, search for it to find where it is defined. A Noweb reference is a single code block with the #+name: foo metadata where foo is the name of the Noweb reference (this is the name other blocks use to refer to it).

1.4.2 Code block naming conventions

We follow a particular naming convention when referring to code blocks. Because Lilac is mostly written in Haskell, many of these prefixes are specific to the Haskell programming language.

Table 1. Naming convention of code blocks.

Naming prefix	Meaning	Example

data	Haskell data declaration	data:OrgDoc
newtype	Haskell newtype declaration	newtype:Prose
eq	Mathematical equation	eq:Pythagorean Theorem
f	Function	f:lilac-main
fig	Figure	fig:PNG image
ghc	Glasgow Haskell Compiler	ghc:Warn everything
t	Test case	t:empty-input
ti	Test case input	ti:0001-blankline
tbl	Table	tbl:Marking characters
instance	Haskell typeclass instance	instance:Show OID
make	Makefile rule	make:Tangle sources
module	Module file	module:Lilac.Types
nix	Nix configuration	nix:devShell derivation
ns	Clojure(Script) namespace	ns:lilac.core
cfg	Configuration	cfg:figwheel-main.edn
sh	Shell script	sh:invoke-dev-build-watcher

2 Release strategy

The next branch is where development happens. Time to time, main is brought up to match next. The process is:

(On next) Release Notes are updated for the as-yet unreleased version.
(On next) The Lilac version is bumped, and a commit is created. A Git tag is created to point to the above commit. See Release commit and tagging.
main is fast-forwarded to HEAD of next.
(On next) Some things are moved from the Backlog into Prioritised TODOs, to be worked on for the next release cycle.

next is expected to be unstable (force-pushes will be the norm).

2.1 Release commit and tagging

stamp-new-version.sh takes the Lilac version by searching for it from this Org file, and creates a commit.

🔗

stamp-new-version.sh
🎯 stamp-new-version.sh

set -euo pipefail

version="$(awk \
	'/^#\+name: Lilac version$/{getline; getline; print}' index.org)"

git commit -am "Lilac ${version}"
git tag -a "v${version}" -m "Lilac ${version}"

3 Contributing

First, agree to the below Developer Certificate of Origin:

🔗

DCO

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Please send your patches to the mailing list. Make sure your commits have Signed-off-by: ... trailer at the bottom of every commit message to signal your agreement to the DCO above. You can use git commit -s for this to automate the generation of the trailer.

It's a good idea to read over the Git project's own guidelines for submitting patches, because they have good advice about how to work with a mailing list.

4 Miscellaneous

🔗

.gitattributes
🎯 .gitattributes

gitattributes-ignore-svgs
gitattributes-ignore-compiled-js

5 Glossary (Global)

.code-block-body: [weave.org] Main body of the block, which contains the actual code that have been fenced by #+begin_... and #+end_....
.code-block-meta: [weave.org] The name or tangle path. These attributes are so common (and important) that we always display them where possible.
.code-block-meta-raw: [weave.org] Other metadata such as the programming language syntax of the code block. These attributes are typically not as interesting, and so require the user to press a button (labeled M) in the .code-block-controls container to reveal them.
a: [weave.org] This is in its own <dl> tag.
a name: [parse.org] what's displayed to the reader, and
a target: [parse.org] where we want to take the reader when they click on the link.
ancestry: [compile.org] data:Archive → 20 A map of global cell indices of blocks to their parents (a block can have multiple parents because more than one block can refer to the same child block). This is created by f:mkBlockAncestry and later used by f:weaveBlockAncestry for making each block link back to its parents, if any.
archive: [compile.org] A self-contained collection of objects. All objects inside an archive can link to each other (headings, code blocks, etc) without any issues. Uses file extension *.larc, and is encoded as JSON.
b: [weave.org] This is also in its own (separate) <dl> tag.
backlinks: [compile.org] data:Archive → 22 A map where the keys are the paths (of objects) and the values are the backlinks to that object. We use a map because we want to organize all of the backlinks by the document they are starting out from. The actual backlinks are just edges between cells (vertices), where the edges are created by a reference (via data:LinkOid) from one cell to another. They are based on newtype:PseudoEdge which are constructed in f:oidLinkParser. See f:weaveBacklinks for how they are used during weaving.
Backlinks: [weave.org] List of backlinks to any named cell in the current document.
bibliography: [cli.org] data:ProjectConf → 6 The paths (relative to where the lilac binary will be invoked) of default bibliography files to use for citation references. Use this if you want to avoid typing relative paths for commonly used #+bibliography: ... entries inside Org files.
bibs: [compile.org] data:Archive → 26 Collection of all bibliography files used by documents. See CSL JSON (bibliographical entries). Needed for processing Citations and printing bibliographies. See Citation dependencies.
build.org: ↑ Describes the build and development environment. Look here if you want to know how to get started hacking on this project.
c: [weave.org] x
caption: [parse.org] (Optional) Displayed as the caption for the image. If there is no caption, on weaving the only name of the figure will be just the figure number, like Figure 1 or Figure 12. You can use styled text (bold, etc), links, and inline math, just like regular paragraphs (the only caveat is that you are not allowed to have newlines).
citationStyle: [cli.org] data:ProjectConf → 7 Similar to bibliography, but for the CSL style file. Unlike bibliography, this is just a single file path, not a list of files.
cli.org: ↑ Describes the implementation of the lilac CLI program.
Clojure: [fe.org] Lisp dialect which traditionally targets the JVM (Java runtime).
ClojureScript (CLJS): [fe.org] Clojure library which is able to convert Clojure code into JavaScript. In a sense this is a Clojure to JS compiler.
Closure: [fe.org] Library developed by Google which performs optimizations on JS. It is a JS to JS optimizer. ClojureScript uses this to clean up the JS code that it generates.
compile.org: ↑ Describes the compilation processes, which power the compile and archive subcommands. These steps check for things like broken links and the absence of any cycles in Noweb references.
csls: [compile.org] data:Archive → 25 Collection of CSL paths and their raw XML values (see CSL style file (XML)). We have to store the information as a Text value, because there is no ToJSON instance for the Citeproc.Types.Style type. Typically csls will have at most a handful of entries, because most (if not all) documents inside an archive will all use the same CSL style. Needed for processing Citations and printing bibliographies. See Citation dependencies.
d: [weave.org] y
fe.org: ↑ Describes the frontend framework for making the HTML page interactive with JavaScript, to go beyond what HTML and CSS can do for us.
glossary: [compile.org] data:Archive → 21 This is the combined set of all glossary terms found in all objects. The main difference is that the key includes not just the term itself (Text) but also the path of the object to identify which object has this term. This also avoids clashes because two documents could very well define the same glossary term (in different ways).
height: [parse.org] (Optional) Height of the image (as a percentage of page width or pixel count), for HTML weaving.
homepage: [cli.org] data:ProjectConf → 2 URL of the homepage. This is used as the link target to describe the name in the navbar. It could very well be the case that this target merely points to one of the other woven object files.
htmlHead: [cli.org] data:ProjectConf → 8 Allows customization of the HTML <head> element, enough to make Lilac's HTML assets completely static (no dependencies on external CDNs). See HTML <head> customization.
index.org (this doc): ↑ Describes the overall project.
isIntersecting: [fe.org] True if intersecting (visible in viewport), false if not.
lilac_maxHeadingLevel: [parse.org] f:maxHeadingLevelParser This takes an integer as a value. Using -1 means no headings are allowed at all. Using 0 means headings can have as many asterisks as desired. Using a positive integer means headings may only have that many (or less) asterisks; otherwise Lilac will fail the parse.
lilac_maxListLevel: [parse.org] f:maxListLevelParser This has the same semantics as lilac_maxHeadingLevel but is for list items, not headings.
Main: [weave.org] Holds all of the content of the document. The list of cells in the document are rendered one after another in here, top to bottom.
name: [cli.org] data:ProjectConf → 1 Name of the project. This is used in the Navigation bar.
name: [parse.org] (Optional) Used to name the cell uniquely in the surrounding Org file. Other cells can refer to this figure by this name (or as part of a global data:LinkOid). The name is optional because all figures, named or not, are always assigned a number like Figure 1 or Figure 12.
Named cells: [weave.org] List of named cells in the current document.
Navbar: [weave.org] Contains the filesystem path of the object, as well as a link to the homepage of the project data:ProjectConf → 2.
nonTangledPathspecs: [cli.org] data:ProjectConf → 5 A list of pathspecs to be sent to the git-ls-files command. This output is compared against all tangle paths in the Archive, and is used for linting.
object: [compile.org] An even more machine-friendly representation of a single Org document. Uses file extension *.lobj, and is encoded as JSON.
offsets: [compile.org] data:Archive → 23 The offset of each object's cells, in relation to the global (flattened) list of all cells in the archive. This is used to convert a GCI back into an LCI (so that we can look up the actual cell at this index inside the object).
Page metrics: [weave.org] Holds various widgets which aid in navigation. The widgets are:
parse.org: ↑ Describes the Org syntax parser. This parser allows us to create machine-friendly intermediate representations (an object (*.lobj) and archive (*.larc)) of an Org document, which are created by the compile and archive subcommands of the CLI, respectively.
plan.org: ↑ Explains where we're headed as a project.
release-notes.org: ↑ Records noteworthy changes between releases.
repo: [cli.org] data:ProjectConf → 3 URL of the source code repository (the assumption being that the Org docs and tangled files are all open source and belonging to the same Git repo).
serve.org: ↑ Describes the inner workings of the serve subcommand.
symbols: [compile.org] data:Archive → 19 A table for finding the cell by its GCI based on a given data:LinkOid. It could be the case that the user decides to link to the top-level UUID of the document data:OrgDoc → 1. This presents a problem because documents are not cells (they contain cells) so using a GCI doesn't really make sense. So instead we use negative numbers so as not to collide with the regular cell index numbers. See f:docOidsToSymbols.
Table of contents: [weave.org] Holds both the site contents and page contents, which give a macro and micro overview of the current document. See Table of contents (TOC).
tangle.org: ↑ Describes the tangle process. Most of the hard work is done during compilation, so this document is pretty short.
Tangled files: [weave.org] List of tangled files in the document.
target: [fe.org] The DOM element whose intersection with the root changed.
type: [parse.org] (Required) Used to force there to be at least 1 key/value pair in the metadata. Otherwise when the parser comes across a link to an image on its own without metadata, it is ambiguous whether the link is part of a 1-link paragraph cell, or as part of a figure cell.
types.org: ↑ Describes the types used to represent the machine-friendly intermediate representation of an Org document. The types here are used heavily by parse.org as well as all of the other pieces.
user-manual.org: ↑ Lilac's User Manual. This is the starting place for newcomers who just want to use the lilac CLI program, without having to read all about how it's implemented internally.
version: [cli.org] data:ProjectConf → 4 Human-friendly version information. This is used in the Navigation bar.
weave.org: ↑ Describes the weave process, which converts all of the Org documents inside an archive (*.larc) into an HTML page, styled with CSS.
width: [parse.org] (Optional) Width of the image (as a percentage of page width or pixel count), for HTML weaving.

6 Bibliography

[1]

D. E. Knuth, “Literate Programming,” The Computer Journal, vol. 27, no. 2, pp. 97–111, Feb. 1984, doi: 10.1093/comjnl/27.2.97.

[2]

N. Ramsey, “Literate programming simplified,” IEEE Software, vol. 11, no. 5, pp. 97–105, Sep. 1994, doi: 10.1109/52.311070.

.gitattributes
Lilac.toml
stamp-new-version.sh