In most Literate Programming systems, weaving is the act of generating a human-readable document. This usually means writing a PDF or an HTML page.
For now, Lilac's main goal is to generate HTML pages. There are several reasons:
HTML is easier to consume on the web than PDFs.
PDFs are obsessed with page numbers because they are designed to be printed on real physical paper; we don't care about page numbers at all because you don't have to print web content in order to consume it.
We want to make our HTML pages somewhat interactive, and interactivity is easier to accomplish in HTML.
Because we generate HTML, we also have to decorate it with CSS. Both concerns are addressed in this doc. There is also a sprinkling of JavaScript (via ClojureScript) in Frontend for some interactivity.
One main advantage of this model is that other "viewers" can be created, as long as they understand the same JSON. While we would prefer to use something more "typed" like Protobufs, Transit, or EDN, we choose JSON for its universality. Also, none of these alternative formats support both Haskell and ClojureScript well.
During the weaving process, we want to access various other settings that may not be apparent at the data:Cell level. To capture these values in one place, we use a data:WeaveConf. Instead of passing this around to all functions, we put it inside a ReaderT so that it is available for all weave-related functions that are using the type:HtmlWeaver. This is analogous to data:OrgParserState which is available to all parsing functions in type:OrgParser.
The originObjectPath1 is used to check whether a given data:LinkOid refers to a cell that exists in the current object being woven in f:weaveObject (so we know whether to refer to the cell with just #... as the HTML id attribute, or whether as path/to/external/object.html#... because the cell belongs to another object).
The type:HtmlWeaver is a monad. We stack ReaderT on top of Lucid's HtmlT to create the monadic environment where we can read from data:WeaveConf whenever we want.
typeHtmlWeaver ::Type->TypetypeHtmlWeaver a =HtmlT (ReaderTWeaveConfIdentity) a
Note that the inner monad is the ReaderT. This way we don't have to call lift every single time we want to use a Lucid combinator like head_ or body_.
Now we just need a starting point for using HtmlWeaver. For weaving, the final thing we want to generate is a Text value. Lucid gives us renderTextT, which can run inside a monad. This means we can use that to generate Text from type:HtmlWeaver; this is what f:renderAsText does.
f:renderAsText takes a WeaveConf and an HtmlWeaver a action, and converts it to a Text type. It's there because we cannot use any of the default rendering functions that Lucid gives us (because they operate on Lucid's HtmlT, not the HtmlWeaver we use).
The hoistHtmlT function is a convenience helper provided by Lucid, which is equivalent to hoist from Control.Monad.Morph. It applies a function to the inner monad of HtmlT (recall that type:HtmlWeaver is just a HtmlT whose inner monad is a ReaderT). So in this case the inner monad is ReaderT WeaveConf Identity, and running runReaderT will result in Identity () (where () is the a in the type signature for f:renderAsText). So then the call to hoistHtmlT gives us HtmlT Identity a, which is what Lucid's renderText expects.
f:genCss is the main top-level function which generates the CSS. There are two main parts: one is the Syntax highlighting theme2 used in Code blocks, and the other is the rest of Lilac (f:defaultCss) which uses the Clay CSS preprocessor. These two parts are concatenated together in f:genCss, so that Lilac's settings take precedence if there are any conflicts with the Skylighting parts.
These font names are reused exactly in text:source-fonts.css, which is why we parameterize them here as their own code blocks. This way we can guarantee that these stay in sync (regardless of whether we pull in the fonts from Google web fonts or from the vendored artifacts).
fontMono ::Css
fontMono =do
fontFamily ["font-mono"] [monospace]
fontSize (em 0.9)
The verbatim and code classes use a smaller 0.9 em font size, because otherwise they look too big (because typically monospaced font tends to look bigger than non-monospaced font).
These fonts are loaded into the HTML page by Google Fonts.
Clay uses its own internal Color type, which can only be constructed from integer values. However we want to talk about colors using strings because using something like #22ffff can allow Emacs to colorize it in the buffer, aiding visualization.
f:colorFrom converts a Text into a Color, defaulting to #ff0000 (red) if parsing fails.
This depends on f:hexColorParser which uses Text.Megaparsec.Char.Lexer.hexadecimal and checks the numeric bounds before retrieving the red, green, and blue values from the number using bit masks.
hexColorParser ::ColorParserColor
hexColorParser =do
_ <- char '#'
n <- hexadecimal
let c
| n >0xffffff= rgb 255255255| n <0= rgb 000|otherwise= rgb rr gg bb
where
rr =0xff.&. shiftR n 16
gg =0xff.&. shiftR n 8
bb =0xff.&. n
pure c
The ColorParser is a type synonym for a "standard" Megaparsec type defined in the documentation.
The f:weaveObject function sets up the main HTML bits (like the title, head, style sheets, etc), before finally delegating to various helper functions.
f:sortFootnoteCells sorts footnote definition cells by the order in which they are referenced 3. This way, users don't have to keep them in order in the doc (users can freely change which ones are referenced first, without having to manually reorder the footnote definitions). The ordering matters because that's the order the cells are woven.
weaveDocMeta ::M.MapTextText->HtmlWeaver ()
weaveDocMeta m =docaselookup"title" m ofNothing->pure ()
Just title ->do
h1_ [id_ "top"] $ toHtml title -- 4
hr_ [class_ "title-underline"]
f "author"$ p_ [class_ "doc-meta-author"]
f "date"$ p_ [class_ "doc-meta-date"]
where
f k element =caselookup k m ofNothing->pure ()
Just x -> element $ toHtml x
For CSS, the title gets a special black circle U+25cf symbol to make it stand out 29, and an underline css:h1 (title) underline. The css:h1 (title) underline changes the horizontal rule to look a bit subdued (without the default black color).
6 hides the block by not weaving it, if the code block is a magic "index" block which has been requested to be hidden, because this is low-level information which is already widely apparent by looking at the "Site contents" in the left sidebar 105. Index blocks are hidden by default, unless the user specifies otherwise.
The use of toHtml8 is to help Haskell infer the correct type. This is because Lucid will think that regular strings are of type Html () (this is how Lucid automatically escapes strings to be HTML-safe). The title_ function expects an Html () type, so we have to manually convert the Text we get out of lookup with toHtml.
The favicon path is hardcoded to be at the root. It's up to the user whether they want to have one or not (and if they want one, it has to be favicon.png).
We only pull in these Google fonts over the network if the user has not specified that they want local imports 10. The user is expected to supply these fonts themselves (host it themselves from the same domain where they are serving the woven HTML) if they request only local imports. See HTML <head> customization for the configuration, and Custom HTML head for how this is used during weaving.
We only import MathJax over the CDN 11 if the user does not request to only have local imports, because the MathJax JS import pulls in JS over a CDN).
MathJax is configurable, so we customize it in MathJax customization. Similarly, we disable this if the user requested local imports, with the understanding that they will provide their own configuration for MathJax. In practice, the user must also provide additional settings to tell MathJax not to use a CDN for importing fonts (because MathJax by default tries to pull fonts from a CDN).
It may sound a bit surprising to let MathJax perform the numbering when we already have this numbering information on hand from the f:mathFragmentNumberedParser. The thing that MathJax solves for us is the automatic placement of the numbers themselves on the pages, so that we don't have to do that part. The parsers we use already conform to the behavior that MathJax expects when numbering equations (namely, that \begin{equation} and \[ start numbered and unnumbered equations), so the way we number equations and the way MathJax numbers equations is the same. In other words, we can let MathJax number equations for us and those automatically generated equation numbers should line up with what we have.
14 prohibits MathJax from acting on certain HTML element classes, namely verbatim and code styles from f:weaveStyledText.
The comment line at 15 is to suppress Ormolu's suggestion to delete an empty line there (the blank line gets added because of f:padConsecutiveNowebLinks).
Widgets are small and require a much smaller heading; therefore they use a <p> element instead of something like <h6>. These widget titles are configured at 18.
The <div> element is used in many places to section off areas of content; to avoid repetition, we reset its margins to be 0 19 here for all <div> elements.
The main content's padding and such is defined in css:main-content.
On mobile, we need to stop the browser from being able to "bounce" the page when the user gestures (drags) the page with their finger. This is because the <main> element will get dragged around while the sidebar and breadcrumbs at the top will remain fixed.
Unlike the world of parsing, tangling, and compiling, we cannot use a data:LinkOid to identify cells in HTML. Instead we have to use the familiar hash syntax like #foo to refer to HTML elements (where #foo refers to an element with an id of foo). There are two types of id values we assign to cells to identify them in HTML: Cell link anchors and Numeric cell indexes.
Generating a unique name suitable for being used as the id attribute value requires us to convert some arbitrary text into a DOM ID-friendly name. For this purpose we have f:toSafeHtmlId, which follows the guidelines from this page, which says:
ID and NAME tokens must begin with a letter ([A-Za-z]) and may be followed by
any number of letters, digits ([0-9]), hyphens ("-"), underscores ("_"), colons
(":"), and periods (".").
toSafeHtmlId ::Text->Text
toSafeHtmlId = T.map onlyKeepSomeChars
where
onlyKeepSomeChars c
| isAsciiLower c || isAsciiUpper c = c
|isDigit c = c
| T.elem c "-_:."= c
|otherwise='-'
The f:getLinkAnchor function generates the unique, human-friendly ID of a data:Cell. It uses the cell's metadata where appropriate it can. For paragraphs, because they cannot have metadata we just use the local cell index 22.
Every cell has a local cell index. We can use this information to give every cell a unique ID. This ID is not human friendly, but it's good for assigning every cell a numeric ID so that we can refer to them easily from code (we also assign all of these a class of cell-idx so that we can select them all at once; see f:lciAttrs).
Originally we added f:lciAttrs (with the unique cell index as the id) to be used for f:genHeadingAncestry for detecting when certain cells are visible in the viewport, but then realized that we actually don't need it there. To clarify, we don't need to attach a unique ID to every cell and then add observers to all of them; instead we can just group non-heading cells underneath each heading together into a <div> to check for the intersection of that overall <div> with the viewport.
Although these cell index based id attributes are not used, we still encode it in f:lciAttrs because eventually we want to add keyboard-driven navigation, and for that we'll need to highlight the "current" cell, identified by its unique ID.
Headings might sound simple to weave, but because they are made up of arbitrary text, it's fairly involved. There's also the business of creating unique link anchors for each heading.
The heading creates and closes a heading-container-...<div> element, to essentially group all non-heading cells (up to the next heading) together as one entity. This is needed for Active headings TOC indicator. Specifically, grouping up cells like this lets us easily track which parts are being viewed (and we only have to keep track of these <div> elements, instead of tracking each cell individually). 23 closes the <div> elements opened up by 24 from previously woven headings; for the very last heading, we close it from 5.
A well-formed document should only have one <h1> HTML element (for the title of the document). We use h1_ when weaving the title of the document (if any), at 4. This is why we start from h2_ for headings 27.
For generating the unique link anchor, we use f:getLinkAnchor which uses both the section number 20 and the contents of its body 21. The f:getSectionNumber function just retrieves the numeric heading section number (e.g., as Text values like "1.2" or "7.3.5"). While the heading section number alone is enough to make it unique (and hence technically sufficient for creating a unique link anchor), including the contents of the body make it more human-friendly when looking at just the raw link anchor as part of a URL.
As for rendering the heading for the user to read, we again use the section number 25 and body 26. The section number is the same as when generation the link anchor, by relying on f:getSectionNumber. The body part is different because we can do much better than just grabbing the text alone; instead we can weave each of the data:PToken values in the body, with all of the rich information encoded within those values. The main workhorse for this is f:weaveProse which can weave a newtype:Prose value; this in turn delegates most of the work to f:weavePToken.
The main differentiation points between heading elements are their font sizes, and any usage of decorative symbols with f:headingPrefix. Stylistically, the main visual elements of headings are their use of sans-serif font (f:fontSans).
Thanks to Haskell we can define some repetitive things in css:Headings via a forM_ loop 28, instead of copy-pasting the same configuration across all headings.
This prefixed text is not selectable (via text selection) by the user in the browser, so it's purely for decorative visual cues. We use separately named code blocks for the <h2> and <h3> symbols because they're also reused in css:page-toc103104. This makes it easier to quickly scan the TOC for the top-level headings.
f:weaveLink is the main workhorse for links. This is used for prose and also code blocks (Noweb links and line labels). Other than the link target address (where we want to take the reader, determined by f:weaveLinkTarget), and the name of link shown to them (via f:weaveLinkDisplayName), we also need to keep track of the HTML ID of the link element for line labels 32 (the targets of line label links inside blocks just point to themselves via this HTML ID).
Links to bibliographical entries (those starting with #ref-33) are not given an internal-link class, because we want these links to be simple without any extra decorations 42.
Let's look at link display names first before looking at how we get the link targets. For the most part, the link display name is just the name field of a link 36. Most of the complications involve line labels. For internal line labels (line labels that are in code blocks in the current document), we just use the line label number 35. But for links to line labels in external code blocks, we also weave the name of that external code block 34.
For weaving the target to Line labels, there are three cases:
The actual labels themselves. This is where they are defined inside a code block, where we have to make them link to themselves 37 (similar to how headings and code blocks can link to themselves for the user to easily save the link to it).
Internal line labels 38. This is simple because we know that all line labels inside a document are unique, so we can just weave with the name of the label (modulo an ll- prefix to mean "line label") 40.
External line labels 39. We can't use f:weaveLinkTargetLinkOid because it will try to do a coarse cell-level lookup. Line labels are finer-grained than cells. We need to get the external relative path, and then skip cell-wise operations and just slap on the name of the line label (modulo the ll- prefix) to the external path.
f:showBrokenLink just encodes a basic "error message" into the link itself if the target cannot be determined.
f:weaveLinkTargetLinkOid is a little bit complicated because it needs to be able to handle links that point to things inside the doc, or to another doc somewhere else (in the archive).
The docName == "" part at 41 is needed to check that the linkOid exists in the current object's list of local symbols; this is the only way we can ensure that the linkOid is a local one internal to the current document. The f:getExternalRelPath function tells us the relative path of the external Org document we have to link to (for external links).
f:isExternalLink checks if a given link target is external to the would-be-generated HTML file (of the data:Object being woven). It considers any non-OID link to be external. File paths links are considered external (and thus opening a new tab) if the file is not an HTML file; linking to another HTML file on disk comes up when we want to link to another HTML file that was not generated by Lilac.
External links get a special superscript-like icon with an arrow sticking out of a box. This just visually indicates that the user will be leaving the current site. Whether a link is external or not is determined in f:isExternalLink during HTML generation.
The raw SVG code for the external link is taken mostly from here. That document had a typo though, where it has viewbox instead of viewBox (which is why we use the latter here).
The blaze-html library we use expects to take care of the starting and closing tags for us automatically when we use its HTML tag functions. However, for list items we cannot use these functions (H.ol, H.ul, H.li) because they require our existing data structure to be nested as well (like a tree).
But the cells we have are flat, and so we cannot use those functions. On the other hand, we already know exactly where to start and close the various tags, so it's actually very easy to insert the tags manually. This is what we do in f:weaveListMarker, which is used by f:weaveCell.
Ordered list items get a right parenthesis, just like what we require when they are written in the Org file (search for "parenthesis" in Detailed rules).
We want to make indentation match (roughly) what we see in the raw Org syntax. Thankfully, there isn't really anything to do here; the defaults from the browser (at least in Firefox) line up with the indentation style in the raw Org syntax already --- namely, the list item is always indented more than the preceding paragraph.
If a list item is empty (has no content), the browser will collapse them onto a single line. To prevent this, we set a minimum height for empty list items.
Code blocks (also just called blocks) have two major parts: the code (with any Noweb links inside them), which we call the body, along with any metadata (such as the name of the block and the tangle path).
There are several additional things worth mentioning, all having to do with how we display each block:
code-block-controls DIV, which has a button 43 to reveal all of the raw metadata, as well as a self-link to the block itself. This DIV is only displayed on hover, and also includes the Self-linked anchor symbol.
f:weaveImportantMetadata which weaves the name and tangle path (if any). This also weaves the block Ancestry, which is automatically generated (unlike the other metadata which the user must supply for each code block in the underlying Org file).
We want users to link to code blocks; that is, code blocks should be linkable just like Headings. We could use the name of the block for this (much like headings where the heading itself is a clickable link to itself), however this wouldn't work for anonymous blocks which have no name. This is why we use the link anchor symbol "🔗" with Unicode code point U+1F517 for code blocks and glossary terms (see f:weaveGlossaryTerm). We display this link symbol whenever the mouse hovers over the block.
To recap, metadata are key/value pairs that describe some characteristics about the block, such as the type of block, programming language, and so on. These are all metadata defined by the user (in manual fashion in the Org mode file). We also have some automatically generated metadata (ancestry).
f:weaveImportantMetadata weaves nothing if both the name and tangle paths are missing. 48 encodes a "direct hit" or "bullseye" emoji (🎯) and is what we use to prefix the tangle path, if any.
The user may also be interested in seeing all of the metadata, but because this is uncommon, the full metadata 46 is hidden behind a button press 43. See Toggle code block metadata to see how we make the button interactive.
f:weaveBlockMeta just weaves a single key and value pair of the metadata; it is called serially for all such pairs at 46.
The block ancestry answers the question "what other blocks refer to this block?". This is handled by f:weaveBlockAncestry which looks at a block and sees if it's a dependency of any other block, by using the ancestry field in data:Archive. If other blocks do depend on this block, we link back to those blocks, which we call "parents", with f:weaveParentBlock. If the parent block itself does not have any parents, we call it a "root" block.
If a block is anonymous (does not have a name) (at 49), then it cannot have any parents because no block would be able to name it (via a Noweb reference) as a child.
f:weaveParentBlock makes two distinctions: whether a block belongs to the current document being woven, and whether that block itself has any parents of its own.
This makes the code block body area hidden by default, by making f:weaveBlockBody start out in "collapsed" form 53. f:weaveInlineBlockControls weaves the toggle-code-block-body button to allow the showing (and hiding) of the code block body.
Unfortunately the syntax highlighting is not as simple as just feeding the body text to a highlighter and displaying the results without modification, because we have links embedded inside the body, namely the Noweb links (LinkTargetOid) and line labels (LinkTargetLineLabel). And we want to convert these links into HTML links, and so we cannot weave in the links before colorization; we cannot put them in naively after colorization because then we'd have to re-parse the text we get back and try to see where the links are.
So instead we do some preprocessing of the body text before we feed it to the colorization function. First we put the links aside, and replace each place we found the link with the NULL byte in f:getBlockContentsWithoutLinks. Because the NULL bytes are invisible to the syntax highlighting process, this works well enough. And then when we get back the colorized string from f:colorizeText, we go to each NULL byte and splice in the corresponding link (weaving it as needed). See f:spliceLinks for this last bit.
That's the high-level overview; now let's look at the details.
getBlockContentsWithoutLinks :: [BToken] ->Text
getBlockContentsWithoutLinks = T.concat .map bTokenToText
where bTokenToText ::BToken->Text
bTokenToText = \caseBTokenString text -> text
BTokenPadding->""-- 55BTokenLinkTarget (LinkTargetOid _) ->"\NUL"BTokenLinkTarget (LinkTargetLineLabel _) ->"\NUL"BTokenLinkTarget {} ->""
The first thing we do is to get the contents of the block with f:getBlockContentsWithoutLinks. This function is similar to f:getBlockContents from Lilac.Parse, but does not encode the link information directly into the text. Instead, links are replaced with the \NUL character. This is because the text is fed into f:colorizeText, and we don't want the links to be dealt with in any special way yet. The BTokenPadding token is ignored at 55 because BTokenPadding is only meant to be used for tangling.
colorizeText ::Text->Text->HtmlWeaver ()
colorizeText lang txt
|Just syntax <- Sky.lookupSyntax targetLang Sky.defaultSyntaxMap,
Justlines<-
foldMap' Just$ Sky.tokenize defaultTokeniserConfig syntax txt =
toHtmlRaw
. Text.Blaze.Html.Renderer.Text.renderHtml
$ Sky.formatHtmlInline Sky.defaultFormatOpts lines|otherwise= code_ $ toHtmlRaw txt
where
targetLang
| lang `elem` ["emacs-lisp", "elisp"] ="lisp"-- 56| lang `elem` ["gitconfig"] ="ini"-- 57|otherwise= lang
defaultTokeniserConfig =Sky.TokenizerConfig Sky.defaultSyntaxMap False
56 is there because Kate (as of 2026) does not recognize Emacs Lisp (a pity). So we have to make do with a fallback to Common Lisp, which Skylighting recognizes as "lisp".
57 maps Git configuration to the INI file format syntax.
Block colorization relies on the skylighting syntax highlighting library. This library ignores the \NUL character, which is what we want. Once we have the colorized text, we splice in the links with spliceLinks, which relies on f:weaveLink for constructing HTML for each link. The splicing works by splitting on the \NUL character and manually "zipping" up the colorized substrings with the woven link HTML fragments.
We use a modified variation of the kate theme from Skylighting (see f:colorizeText) which we call f:kateLilac We make some tweaks to make some colors easier on the eyes. In particular, the ExtensionTok by default is too bright, so we darken it a bit here 58.
We need to use both visibility59 and opacity60 to hide/unhide the controls. Visibility hides the element without changing the layout, while opacity hides it visually.
The rest of the block has three parts (these are evident from f:weaveBlock):
.code-block-meta
🔗 The name or tangle path. These attributes are so common (and important) that we always display them where possible.
.code-block-meta-raw
🔗 Other metadata such as the programming language syntax of the code block. These attributes are typically not as interesting, and so require the user to press a button (labeled M) in the .code-block-controls container to reveal them.
.code-block-body
🔗 Main body of the block, which contains the actual code that have been fenced by #+begin_... and #+end_....
Active figure captions are highlighted, just like for headings. Tables and figures use the same figcaption HTML element for the caption, so we only need to specify the CSS in one place, for both figures and Tables.
Math fragments come in two types: standalone and inline. Standalone fragments get linked to themselves, much like how we weave headings in f:weaveHeading.
Tables are similar to figures, because they have some content (a table instead of an image), and also have a caption. In this sense they are pretty simple. On the other hand, tables are necessarily verbose because of several reasons:
HTML tables have multiple parts (e.g., the head and body, as well as the caption). See f:weaveTableRows.
We want to add thicker lines between certain rows and columns, which means we have to add in some logic to give special meaning to certain parts of the table. This is done in f:weaveTableBody for the rows and f:weaveTableColumnGroups for the columns.
Org mode allows marking characters, which are not to be displayed in the HTML output (they should be invisible). See f:weaveTableColumnGroups72.
The table-wrapper65 is there to hide a tiny gap when the sticky table header 77 takes effect (when the user scrolls down a long table). Instead of adding a border to the <table> element, we add one to the table-wrapper div instead.
In the world of HTML tables, column groups let you apply certain properties to entire columns. For us, we use column groups to set vertical line styles, if the user specified a < character to mark the start of a column group. The < character is only read from a row containing the "do not export" marking character /. See tbl:Marking characters for an example of how this is used in practice.
We start with getColgroups67, which ultimately returns a list of column group sizes (each integer being how many columns are in that group). By default we assume that we only have 1 column group (which encompasses all columns in the table, via f:getMaxCol). However, if we can detect a special row which has a slash (/) in the first column and which has column group start markers (<) then we use that to generate the list of column group sizes.
This row is then fed into the helper function 68 which uses f:getTableCellsFromRow to get the (visible) cells from the row we're interested in (it drops the cell in the first column if it has a making character in it).
f:tableCellLacks just checks whether the column does not have any of the given marking characters. In 73, we use it to detect special marking characters like the slash / or column group start marker <.
Back to f:weaveTableColumnGroups, getColgroupSizes70 simply looks for the < character to see if we should create a new column group; otherwise it simply increments the column group size. We start out with [0] as the initial term and immediately compute it as [1]71 so that we can skip the first column (because the < is optional for the very first column group).
We draw both thick and thin vertical lines, depending on the situation. Thin lines separate columns, and thick lines separate column groups. We use the border on the left of a table cell to color in the vertical line 7980, which means we'll have to color in the first column of a column group to mark it as a thick vertical line. We use 1-based indexing because it feels slightly more intuitive when we have variables like numCols66 to encode the total number of columns.
The table body is more complicated, because of a few things:
checking whether the row is just a horizontal line 74,
checking whether the row is visible at all 76 (the marking characters in f:markingCharsInvisible render the entire row invisible), and
checking whether the table cell is mostly numeric in nature 75, in which case we add the table-cell-numeric class to make the text inside the cell right-aligned (so that the digits of the same unit (tens, hundreds, etc) line up vertically with other numeric cells in the same column).
f:getHeaderRowIndex gets the index of the header row by searching for the first row which lacks f:markingChars in the first column; it's not simply 1 (using 1-based indexing) because there could be invisible rows just above the header.
78 sets the starting position of the header cell (<th>) to be -1, not 0, because -1 makes it hide a gap when scrolling down (due to the position being sticky). This is hacky, but there doesn't seem to be an alternative way to do this in native CSS.
At 81 we try to add a link back to the place (referrerLCI) where we first made a reference to this footnote definition. The f:weaveFootnoteBacklink function tries to find the cell index in the current document, and if it is found, links to that cell.
We don't wrap everything inside a <dl> tag because it may be the case that many more glossary terms exist after this (current) cell being woven. In that case, we want to close off the <dl> only at the very end. That's what 83 and 84 are for. The isGlossaryTerm helper 82 checks if the given cell index points to a glossary term cell inside the current object being woven.
The dlStart and dlEnd bits don't really mean much if we have glossary terms inside list items, because each list item marker in between each item will break up the continuity of glossary terms. So we'll end up with multiple singleton description lists (each with just one glossary term in it) instead of just one description list tag with multiple terms. This is not the most elegant result, but it is what it is.
Sorting of glossary terms means sorting what's visible for the human user. So for links, it means only using the display name, not the target of the link which may have a long (random) UUID. This means using a custom comparing function for the prose 85, to remove all links so that we only use the display text of links. We can do this with f:stripLinks, just like we did for f:getLinkAnchor, in f:toNormalizedText.
f:toNormalizedText also puts everything in lowercase, because otherwise capital letters take precedence over lowercase ones, which can be confusing. It also calls f:stripStyles because otherwise those (useless) style characters like * and / will interfere with the sorting.
In f:weaveCollectedGlossaryTerm, because we only have the cell index, we first pull out the glossary cell to find the CGlossaryTerm we want to weave. We also insert a link back to the place where the glossary term was originally defined, to give the user additional context.
The branch of code at 86 weaves internal terms (terms that are defined in the current document). As such they have no need to disambiguate multiple identical terms (these are not allowed within a single document).
In contrast, 87 weaves external terms (terms that are defined in another document). These may need disambiguation, so we weave the externalOrgDocName88. Getting the externalOrgDocName (and the link to it) is handled by f:getRootRelativeOrgDocCellLink.
f:getLinkToGCI is a basic building block for linking out to cells (whether internal or external) based on their global cell index. It returns a tuple of (linkText, externalOrgDocName), where linkText is what we can put in the href (of the form #foo for internal cells and path/to/doc.html#foo for external cells). The externalOrgDocName is blank for internal cells.
For glossary terms, we want to place the definition (<dd>) on the same line as the term (<dt>). Because this is not the default behavior (by default the two are placed each on their own lines) we place a newline character (the string \a in CSS) after a <dd> at 89.
Convert the newtype:Prose contents of data:Citation into CslJson Text, while preserving the information of non-text things like links and math fragments by embedding them as CslLink for things that CslJson Text does not support.
Call into Citeproc.citeproc to get the transformed CslJson Text back out.
We can't simply convert newtype:Prose into HTML first before calling Citeproc.citeproc because then that would result in things like <a href=...> being converted into <a href=...> (entity conversion of the given text).
For the first step, we need a function to convert newtype:Prose into CslJson Text. For that we have f:proseToCslJsonText. Apart from using CslLink to encode links 90, this function uses the CslDiv constructor to encode information that cannot be represented natively in CslJson such as math fragments 91 and certain text styles 92.
f:proseToMaybeCslJsonText is a wrapper around f:proseToCslJsonText to convert it easily into a Maybe (CslJson Text). This is useful because the Citeproc library uses this type for constructing its citation objects.
The individual data:Cite items are converted into Citeproc's CitationItem type. Orgmode has fancy styling capabilities for individual citation items (this is sv1 in f:citeToCiteprocCitationItems below), but we only support the t and na style variants (we don't even look at the second element in the citation.styleVariation tuple 93). This is because we rely on CSL to do nearly all of the styling for us. CSL itself allows for just a handful of variations, namely:
suppress-author: do not show the author name
author-only: only show the author name
See ti:0400-citeproc-style-variations for examples. We achieve the semantic meaning of t (aka text) and na (aka noauthor) by using combinations of the above. The na maps 1:1 to author-only so that's an easy one. However the t style needs to inject the author name (as regular text) just before the cite. In order to achieve this, we create two citations, using a combination of suppress-author (CP.SuppressAuthor) and author-only (CP.AuthorOnly) to get what we want. Because we end up creating two citations in place of one, the result type of f:citeToCiteprocCitationItems is a list.
cslJsonTextToProse ::CslJsonText->Prose
cslJsonTextToProse cslJsonText = compressProse .Prose$ f [] cslJsonText
where f :: [Style] ->CslJsonText-> [PToken]
f s = \caseCslEmpty-> []
CslText t ->
[ PTokenStyledTextStyledText {body = t, style = styleMaskFrom s}
]
CslConcat a CslEmpty-> f s a
CslConcat (CslConcat a b) c -> f s (CslConcat a (CslConcat b c))
CslConcat a b -> f s a <> f s b
CslQuoted a -> f s a
CslItalic a -> f (Italic: s) a
CslNormal a -> f [] a
CslBold a -> f (Bold: s) a
CslUnderline a -> f (Underline: s) a
CslNoDecoration a -> f (filter (/=Underline) s) a
CslSmallCaps a -> f (Smallcaps: s) a
CslBaseline a -> f (filter (`notElem` [Subscript, Superscript]) s) a
CslSup a -> f (Superscript: s) a
CslSub a -> f (Subscript: s) a
CslNoCase a -> f s a -- 94CslDiv t a
| t =="LILAC_STYLE_CODE\NUL"-> f (Code: s) a
| t =="LILAC_STYLE_VERBATIM\NUL"-> f (Verbatim: s) a
|"LILAC_MATH_FRAGMENT\NUL"`T.isPrefixOf` t ->
(CP.fromText (T.drop 20 t) ::Prose).unProse
|otherwise-> [PTokenDivStart t] <> f s a <> [PTokenDivEnd]
CslLink t a
|"LILAC_LINK\NUL"`T.isPrefixOf` t ->
(CP.fromText (T.drop 11 t) ::Prose).unProse
|"#ref-"`T.isPrefixOf` t ->let uri =N.URI
{ N.uriScheme ="",
N.uriAuthority =Nothing,
N.uriPath = T.unpack t,
N.uriQuery ="",
N.uriFragment =""
}
target =LinkTargetURI uri
name = compressStyledText . getStyledText $ f s a
in [PTokenLinkLink {target = target, name = name}]
|otherwise->case (N.parseURI $ T.unpack t) ofNothing-> []
Just uri ->let target =LinkTargetURI uri
name = compressStyledText . getStyledText $ f s a
in [PTokenLinkLink {target = target, name = name}]
getStyledText [] = []
getStyledText (pToken : rest) =case pToken ofPTokenStyledText styledText -> styledText : getStyledText rest
PTokenLink {}
PTokenMathFragment {}
PTokenCitation {}
PTokenDivStart {}
PTokenDivEnd {} -> []
The CslNoCase is fine to ignore 94 because it's most likely used by Citeproc to handle case-changing transformations, and is probably not meant to be used directly by consumers.
The CslDiv is used to wrap bibliographical entries inside printed bibliography for formatting purposes. We ignore them for now, except for those LILAC_STYLE... bits that we've encoded in f:proseToCslJsonText.
Now we have all of the pieces needed to process citations. The idea is to process all citations found inside an Org document at once, just before the weaving begins in f:weaveObject, by calling Citeproc.citeproc which will get us a Result a which includes the formatted citations as well as the formatted bibliography. This part is handled by f:prepareCitations. Then during the weaving process we refer to these formatted citations (of type Prose), before finally converting them to HTML again.
The neededByCitations helper filters out the bibliographical entries so that we only print the bibliography that are actually used by the citations. See this upstream issue.
The #ref-... style is based on what Citeproc uses; for now we choose to use it as is. If we wanted to modify it, we would need to tweak how we handle the CslLink constructor in f:cslJsonTextToProse.
For tests around citation processing, we need a CSL style file (XML) and some bibliographical entries. For the CSL style files, we can refer to the styles we cloned as a submodule under csl-styles.
Here is a sample bibliography, taken from https://zbib.org and converted from BibTeX format into CSL JSON with Pandoc via pandoc citations.bib -t csljson -o citations.json.
[{"ISBN":"9780937073803 9780937073810","author":[{"family":"Knuth","given":"Donald Ervin"}],"collection-number":"no. 27","collection-title":"CSLI lecture notes","id":"a","issued":{"date-parts":[[1992]]},"keyword":"Computer programming","publisher":"Center for the Study of Language; Information","publisher-place":"Stanford, Calif.","title":"Literate programming","type":"book"},{"DOI":"10.1145/362929.362947","ISSN":"0001-0782, 1557-7317","URL":"https://dl.acm.org/doi/10.1145/362929.362947","accessed":{"date-parts":[[2025,8,30]]},"author":[{"family":"Dijkstra","given":"Edsger W."}],"container-title":"Communications of the ACM","id":"b","issue":"3","issued":{"date-parts":[[1968,3]]},"page":"147-148","title":"Letters to the editor: Go to statement considered harmful","title-short":"Letters to the editor","type":"article-journal","volume":"11"},{"DOI":"10.1112/plms/s2-42.1.230","ISSN":"00246115","URL":"http://doi.wiley.com/10.1112/plms/s2-42.1.230","accessed":{"date-parts":[[2025,8,30]]},"author":[{"family":"Turing","given":"A. M."}],"container-title":"Proceedings of the London Mathematical Society","id":"c","issue":"1","issued":{"date-parts":[[1937]]},"page":"230-265","title":"On Computable Numbers, with an Application to the Entscheidungsproblem","type":"article-journal","volume":"s2-42"},{"DOI":"10.1002/j.1538-7305.1948.tb01338.x","ISSN":"00058580","URL":"https://ieeexplore.ieee.org/document/6773024","accessed":{"date-parts":[[2025,8,30]]},"author":[{"family":"Shannon","given":"C. E."}],"container-title":"Bell System Technical Journal","id":"d","issue":"3","issued":{"date-parts":[[1948,7]]},"page":"379-423","title":"A Mathematical Theory of Communication","type":"article-journal","volume":"27"}]
Check what the result of calling Citeproc.citeproc looks like, after converting back to newtype:Prose using IEEE style.
Expected bibliography for 0400-citeproc-smallcaps ↑ ↑ ↑
__block_type = src
__src_language = haskell
name = Expected bibliography for 0400-citeproc-smallcaps
[ ( "a",
Prose
{ unProse =
[ ptext "Knuth, D.E." [Smallcaps],
ptext " 1992. " [],
ptext "Literate programming" [Italic],
ptext ". Center for the Study of Language; Information, Stanford, Calif." []
]
}
)
]
We should be able to handle citations written across multiple lines. Because this is only a formatting change, we expect to get the same citations and bibliography as we did in ti:0400-citeproc-smallcaps.
Citations can be in a multiline list item. They can also have links and inline math in them, thanks to f:proseToCslJsonText which preserves these prose elements in the conversion to CslJson Text.
#+bibliography: bib.json
#+cite_export: csl ../../../csl-styles/acm-siggraph.csl
- List item [cite: global \(1 + 1
= 2\) ;*bold* /italic/ [fn:@a]
@a and really
~amazing~;
some bit of
=@b verbatim=
for a @c thing
; global suffix with OID link
[[id:00000000-0000-0000-0000-000000000000::foo]]]
[fn:@a] Definition of @a.
Expected bibliography for 0400-citeproc-multiline-complex-list-item ↑ ↑
__block_type = src
__src_language = haskell
name = Expected bibliography for 0400-citeproc-multiline-complex-list-item
[ ( "a",
Prose
{ unProse =
[ ptext "Knuth, D.E." [Smallcaps],
ptext " 1992. " [],
ptext "Literate programming" [Italic],
ptext ". Center for the Study of Language; Information, Stanford, Calif." []
]
}
),
( "c",
Prose
{ unProse =
[ ptext "Turing, A.M." [Smallcaps],
ptext " 1937. " [],
PTokenLinkLink
{ name = [stext "On Computable Numbers, with an Application to the Entscheidungsproblem" []],
target =LinkTargetURIN.URI
{ N.uriScheme ="https:",
N.uriAuthority =JustN.URIAuth
{ N.uriUserInfo ="",
N.uriRegName ="doi.org",
N.uriPort =""
},
N.uriPath ="/10.1112/plms/s2-42.1.230",
N.uriQuery ="",
N.uriFragment =""
}
},
ptext ". " [],
ptext "Proceedings of the London Mathematical Society" [Italic],
ptext " " [],
ptext "s2-42" [Italic],
ptext ", 1, 230\8211\&265." []
]
}
)
]
We support the /t (text) and /na (no author) citation style variations. The /t makes it so that you can use the citation at the beginning of the sentence because the author name will be pulled out of the citation delimiters. The /na variation simply suppresses the author name from inside the citation.
#+bibliography: bib.json
#+cite_export: csl ../../../csl-styles/acm-siggraph.csl
- [cite/t:@a] discusses a great programming idea. On an unrelated note, Turing
was a pioneer in computer science. He wrote about "computable numbers"
before there were computers [cite/na:@c].
The Table of Contents or TOC is automatically generated for both the headings of the current document (which we call page contents for the user) and also the relationship of this document with others in the overall archive (which we call site contents for the user, or GTOC for "global" table of contents internally).
The page contents track the first three levels of headings.
We convert the flat heading cells into a nested structure (with bullet points), because in the future we may decide to make TOC headings collapsible, and in that scenario it would feel much more natural to have a nested structure. The spirit of this is very similar to how we computed the section numbers for each heading in f:headingParser. Every time we enter a nesting level, we'll have to create a new ordered list tag (<ol>), and every time we leave a level (and we may leave more than one level at once), we have to close those unordered list tags (with </ol>).
95 constructs artificial heading cells for the page metrics, so that we get entries for these bits which are generated automatically. It's admittedly a bit hacky, because we have to also manually craft heading ancestry entries at f:weaveHeadingAncestry.
All of the business around using LItemEnd and LOrderedEnd is needed because when we nest levels, we cannot close the current list item (because nested list items are technically children of the current list item). The call to closeHeadingLevels97 is needed because the f helper only works to close nesting levels when we encounter a heading that is less nested than the previous one. So this means the last heading will always remain "open". We close off the last heading level in 96.
For the Active headings TOC indicator we want to highlight all visible headings as well as all of their parent headings ("ancestor" headings) in the document. We need to generate a "heading ancestry" map where the keys are individual headings, and the values are lists of other headings which are considered its ancestors.
To be a bit more precise though, we need ancestry information for all cells, not just headings, because it could be the case that the viewport is not showing any headings, but only the paragraphs (or other non-heading cells, such as a code block or table) that belong to a heading. In these situations we would still want to highlight the most relevant heading (and its ancestors). What we can do is group a heading and all of its non-heading children into a single <div> wrapper. Then we can just check for whether this wrapper is visible. So the heading ancestry should be available for all heading wrappers in the document, where the keys are the headings (and their children that the wrappers have grouped up together) and the values are the headings we want to highlight in the TOC as being active.
In f:genHeadingAncestry we fold over all cells, but skip over any non-heading cell. For headings, we get the unique ID of it by calling f:getLinkAnchor. This is used as the map key. And whenever we detect Headings we update the list of ancestor headings (also using f:getLinkAnchor to get the name), and use this as the basis of the map value (with a minor tweak via ancestorsOf101).
We update the heading ancestry as follows:
If the list is empty, push the current heading into the list.
Otherwise, compare current heading level with heading level of the first element (most nested) of the list:
If current one is higher 98, pop off elements until we see one that's even higher and push this one.
If they're equal 100, then pop off one element and push this one.
If current one is lower 99, then just push into the list (no popping).
genHeadingAncestry :: [Cell] ->MapText [Text]
genHeadingAncestry =snd. foldl' f ([], mempty)
where f ::
([Heading], MapText [Text]) ->Cell->
([Heading], MapText [Text])
f ([], m) cell@(CHeading heading) =let updatedAncestry = [heading]
key = getLinkAnchor Nothing cell
val = [] :: [Text]
in (updatedAncestry, insert key val m)
f (ancestry@(a : as), m) cell@(CHeading heading) =let updatedAncestry
| heading.level < a.level -- 98=
heading
: (dropWhile (\a' -> heading.level <= a'.level) ancestry)
| heading.level > a.level = heading : ancestry -- 99|otherwise= heading : as -- 100
key = getLinkAnchor Nothing cell
val =map
(getLinkAnchor Nothing.CHeading)
$ ancestorsOf heading ancestry
in (updatedAncestry, insert key val m)
f acc _ = acc
ancestorsOf ::Heading-> [Heading] -> [Heading]
ancestorsOf leafHeading ancestry -- 101|null ancestry = []
|otherwise=dropWhile (\a' -> leafHeading.level <= a'.level) ancestry
The ancestorsOf function 101 is needed because we can't use either the existing ancestry (from the previous iteration) or the updatedAncestry as is. This is because ancestry is simply outdated (it may have an elaborate hierarchy that is not relevant to a new top-level heading), and updatedAncestry includes the current heading (and the current heading cannot be its own ancestor).
We then convert this map into a JSON object and embed it into the document directly in f:weaveHeadingAncestry, so that the frontend code can parse and reference it.
The main difference between the two is that a GTOC uses a tree data structure, whereas the TOC uses a flat list of cells.
The entrypoint is f:weaveGtoc, which uses f:weaveGtocForest to recursively weave the tree into a nested structure of unordered list items (<ul>...</ul>).
We could feed the entire tree to f:weaveGtocTree as is, but we first make the root index.org node a sibling to its children 106. This way, we avoid showing the root node on its own level (always by itself, without siblings, taking up valuable indentation space). That is, instead of
The root node gets rendered separately on its own, much like an "Introduction" chapter in a book. For this reason the root index should typically be titled as "Introduction".
weaveGtocTree ::TreeGPath->HtmlWeaver ()
weaveGtocTree (Node path children) =do
weaveConf <- ask
li_ [] $dolet (docName, docPath) =case getRootRelativeOrgDocLink weaveConf path ofNothing->
( "LILAC_ERROR_DOC_NAME_NOT_FOUND",
"LILAC_ERROR_DOC_LINK_NOT_FOUND"
)
Just res -> res
docLabel
|Just obj <-lookup path weaveConf.archive.objects,
Just title <-lookup"title" obj.orgDoc.meta =
title
|otherwise= docName
boldAttr =if path == weaveConf.originObjectPath
then [class_ "bold"]
else []
a_ ([href_ docPath] <> boldAttr) $do
toHtml @Text docLabel
let isAncestor = path `isGtocAncestor` weaveConf.originObjectPath
when (isAncestor && (not$null children)) $do
ul_ [] $ weaveGtocForest children -- 107
f:isGtocAncestor checks if the first path is an "ancestor" of the current object being woven. We use this to make sure we only render the part of the GTOC we care about, because we don't want to weave the entire GTOC tree because it may be too big.
t:0305-gtoc-ancestor encodes some expected behavior of f:isGtocAncestor. Perhaps the most interesting input is 108 where the given path deviates from the originObjectPath; this is the condition which will make sure we "prune" the tree so that we don't render the entire GTOC tree in f:weaveGtocTree.
One of the things in the navbar is the "breadcrumbs" area, which is the path that points to the current document. This path is constructed by starting with the project name, followed by the path to the object currently being woven, which is taken from the originObjectPath field from data:WeaveConf.
The middle part can be any number of intervening directory names. The project name comes from data:ProjectConf, and the Org document name comes from the originObjectPath from data:WeaveConf.
The middle breadcrumbs 113 are perhaps the most interesting. We get a list of all parent objects, which are all index.org objects in parent directories (including the root). We can get this list of parent index.org objects by getting all leading paths (with inits112) like
for an origin object at path a/b/c/foo.lobj. 111 is just for dropping the first and last elements, because they are not needed. Then we just make each of those leading substrings end with /index.lobj114, and then do lookups into the object map 115. If the path exists, then we render a link to it. If not, we leave it as a plaintext string.
The reliance on originObjectPath assumes that the archive has object paths that are relative to the repository root. It could be that this is not true, but we don't bother to handle that edge case here.
The navbar causes a problem for links that point to elements inside the page, because the browser will normally go to the target element by putting it at the top of the page, but the navbar will sit on top of it. So we have to adjust it so that the navbar never hides something underneath it.
The page metrics shows additional information about the current document. It sits on the bottom of the page (see css:Layout). It has the following parts:
Because these page metrics are generated, they won't have entries in the Page contents (because the user did not write these headings). So we have to inject artificial heading cells in f:weaveToc95, with page-metrics-cells.
To better support Literate Programming, we list all files that are tangled from here in f:weaveTangledBlocksList. This goes through all code blocks that are tangled from the current document, and sorts them based on the tangled path.
Backlinks are useful because they show additional context around the current page. We basically want to know all of the other Org docs that refer to the current one.
The groupedByTarget helper 121 groups edges by the target cell, so that we can see which cells are the most popular from external objects. So instead of [(1, 2), (5, 2), (4, 7)] we get [([1, 5], 2), ([4], 7)]. It uses sortBy122 to first sort by the number of external indices first, and then by the name of the internal cell being referenced.
For getInternalLinkAnchor, it's OK to have Nothing as the first argument 123 for f:getLinkAnchor because it's only used for paragraph cells to name them, and it's impossible to link to paragraphs and so we'll never end up with an internal cell which is a paragraph cell.
The named cells index displays a list of all named cells used in the current document, because if a cell was given a name it is probably of some interest to the reader.
The footer is the element at the very bottom of the page. This is not to be confused with Footnotes, which are a user-controlled thing (a page may or may not have footnotes, depending on usage).
The lists of tangled file, backlinks, etc do not need any additional customization because they are ultimately just links. The footer, however, needs additional tweaks.
The tests here use Org files as input. The output are HTML documents. For the test cases, we search for the presence of certain substrings within the HTML.
f:getWovenPrettyFromPath weaves the given Org file to a Text, by first compiling it to an object. Then in our test cases we can check for expected substrings inside it. Note how it uses renderHtml from the Text.Blaze.Html.Renderer.Pretty module, because we want to be able to write test cases where we write expected HTML substrings in a prettified way (for readability).
In practice we will only need multiple needles if we are dealing with a complicated haystack (with nested structures) and we only want to check some key structures within it.
f:weaveExpect takes a path to an Org file and the list of expected substrings inside the woven result.
f:expectCiteprocResult takes an Either String (CP.Result Prose). If it's a Right values, it checks the citations and bibliography for a given Org doc in the test/Lilac/WeaveSpec folder. If it's a Left value, it expects the operation to fail, and expects to find a substring.
↑ Main body of the block, which contains the actual code that have been fenced by #+begin_... and #+end_....
.code-block-meta
↑ The name or tangle path. These attributes are so common (and important) that we always display them where possible.
.code-block-meta-raw
↑ Other metadata such as the programming language syntax of the code block. These attributes are typically not as interesting, and so require the user to press a button (labeled M) in the .code-block-controls container to reveal them.