Parse
1 Overview
We use Megaparsec to parse an Org document. The parsing requires us to read in everything in a single Org file.
We use a flat list of cells1 3 for parsing, where each cell can be a
paragraph (prose text, made up of mainly Styled text and Links including Line labels, Footnotes, and Citations),
ordered or unordered list item (can be nested),
block (code block, quotation, example, etc),
figure (image),
LaTeX math fragment (inline or standalone),
footnote, or
This list-of-cells scheme is good for quickly making sense of the overall document structure in linear fashion, start to finish. It also simplifies the data structure --- we just parse one cell after the other. Though using a tree data structure for the headings could be useful for situations where tree-based manipulations are desirable, we will never need to do such manipulations for purposes of weaving the Org input into HTML (or any other format). Hence our preference for arranging the cells as a list instead of a tree.
In summary an Org document is made up of cells, as well as some metadata 2 (such as the title, author name, etc). The doc also has an OID 1, which allows other Org files to refer to it (even if the Org file does not have any headings, which are the only other things that can have OIDs).
1.1 OrgDoc data structure
The OrgDoc is important because it's what we end up with after we're done parsing a raw Org file.
type OrgDoc :: Type
data OrgDoc = OrgDoc
{ oid :: OID, -- 1
meta :: Map Text Text, -- 2
cells :: [Cell], -- 3
pseudoEdges :: [PseudoEdge], -- 4
footnoteRefs :: Map Text (Int, LCI), -- 5
glossary :: Map Text LCI, -- 6
lineLabels :: Map Text Int, -- 7
cslPath :: Maybe FilePath, -- 8
bibPaths :: [FilePath], -- 9
citations :: [Citation] -- 10
}
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON) -- 11As stated earlier, we use a list of cells 3 to represent most of the document content. The Int value we use to index into this list of cells is given its own type newtype:LCI to avoid using the Int primitive (using Int everywhere can get confusing).
type LCI :: Type
newtype LCI = LCI
{ unLCI :: Int
}
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, FromJSONKey, ToJSON)The oid 1 is a unique identifier for the document. Every Org file we parse is required to have a
:PROPERTIES:
:ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
:END:at the beginning of the document to uniquely identify it. It is parsed into an OID type, which just wraps around a UUID. This way we can be agnostic about the concrete details of a UUID in the rest of the program.
type OID :: Type
newtype OID = OID UUID
deriving stock (Eq, Ord, Generic)
deriving anyclass (FromJSON, FromJSONKey, ToJSON)There are some helpers in Org ID (OID) for this type, which plays a big role in Lilac because OIDs are also used for headings; OIDs for documents and headings form the building block of Org links between them.
The meta field 2 captures all document metadata, in a simple key/value map. One example is #+title: ... used to specify the human-friendly title of the document.
The deriving anyclass (ToJSON) 11 is needed to create the default implementation for JSON conversion automatically for us. It is the equivalent of writing a default instance like
instance ToJSON OrgDocbut it's shorter because we don't have to put it on its own line, and it's even more closely coupled with the declaration of the OrgDoc type itself. If we try to use deriving stock (Eq, Generic, Show, ToJSON) we get an error like
‘ToJSON’ is not a stock derivable class (Eq, Show, etc.)which is why we use the deriving anyclass variant. All child types also require the same deriving anyclass (ToJSON) declaration for this to work. At the end of it though, we can use the encode and encodeFile functions from Data.Aeson.
1.2 Auxiliary data structures
We also need several other auxiliary data structures to encode various bits of information beyond just what the cells have in them. Basically all of these auxiliary bits encode information about relationships between certain cells.
1.2.1 pseudoEdges
The pseudoEdges 4 are edges that are ultimately used to create a graph representation of links between cells (and ultimately they are finalized in f:finalizeEdges and used to generate backlinks at f:mkBacklinks). In that final representation, we want to use global cell index numbers to represent the universe of all cells in a data:Archive. However, inside the parser for a single Org document, we don't have access to this global cell index number, as that is computed separately in f:mkArchive (in other words, this computation depends on what other Org files (as objects) are included in the archive). Also, we don't know the target cell's number either, and the best information we have is just a data:LinkOid. Because of these issues, we call the edges here "pseudo" edges. To represent these edges, we use a newtype.
type PseudoEdge :: Type
newtype PseudoEdge = PseudoEdge (LCI, LinkOid)
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)The newtype:PseudoEdge represents a link between a local cell index and a target cell via the data:LinkOid. Both of these values are immediately available to us, so that's why we use them for the pseudo edge representation (there's no need to compute these values lazily after the fact when there is more information available).
1.2.2 footnoteRefs
Footnotes get their own separate data structure called footnoteRefs 5, because we have to remember the order of when these footnotes are referenced from other cells (anywhere we could have prose, such as paragraphs, headings, etc). This ordering is needed to determine which number to display as a superscript when creating the link to the footnote definition. This same ordering information is used in f:weaveObject which rearranges the footnote definitions so that they are woven in the order they are referenced, not the order in which they are defined (parsed). This is unlike every other cell where the order of parsing them (cell-by-cell) is preserved during weaving.
1.2.3 glossary
For the glossary 6, see Description lists, aka glossary terms. We save all glossary terms we find for this document, so that we can print them out later, much like citations.
1.2.4 lineLabels
The lineLabels 7 are for Line labels (named positions inside a code block, which you can link to from the prose text). We reset the numbering of these back to 1 every time we see a new code block, so we keep track of the assigned numbers here with a map.
1.2.5 Citations
The cslPath 8 is the path to the citation style file we want to use for generating citations (for all of the citations used in the OrgDoc).
The bibPaths 9 is the list of files that each have a list of bibliographical works that we want to cite. This is a list of files, and not just a single list, because we want to allow users to be able to use multiple files in case they want to reference a work already captured in a separate file somewhere else (essentially amounting to file reuse).
Finally the citations 10 are the stylized citations used by the OrgDoc. See Citations for details.
1.3 Top-level parser parseOrgDoc
The top-level parsing function is f:parseOrgDoc, which takes some raw text input and parses it into an OrgDoc (data:OrgDoc), using the f:orgDocParser as the main workhorse 12.
The f:orgDocParser is itself composed of other parser combinators. The hierarchy of the main parsers look like this:
All of these parsers live inside the type:OrgParser monadic type, which is defined as a state monad transformer (StateT) with data:OrgParserState, which wraps around the inner Parsec Void Text monad, where Void is the custom error component and Text is the type of the text being parsed. Using Void for the custom error component means that we'll be defaulting to the default error handling mechanism that ships with Megaparsec.
type OrgParser :: Type -> Type
type OrgParser = StateT OrgParserState (Concrete type for Megaparsec)The Concrete type for Megaparsec feeds the concrete Text type to use as the stream for Megaparsec. Megaparsec defines a polymorphic API and expects users to use concrete types like this for speed optimizations.
MP.ParsecT Void Text Identity1.4 OrgParserState
The data:OrgParserState is the custom state we want to use while parsing. It's a mutable variable which we keep updating as we make progress during parsing.
type OrgParserState :: Type
data OrgParserState = OrgParserState
{ docOid :: OID, -- 13
headingOidStack :: [Maybe OID], -- 14
headingLevel :: Int, -- 15
headingSectionNumber :: [Int], -- 16
insideHeading :: Bool, -- 17
cellNames :: Set Text, -- 18
cellNumber :: LCI, -- 19
pseudoEdges :: [PseudoEdge], -- 20
tanglePaths :: Set Text, -- 21
insideCodeBlock :: Bool, -- 22
anonBlockNumber :: Int, -- 23
lineLabelNumber :: Int, -- 24
lineLabels :: Map Text Int, -- 25
styleMask :: StyleMask, -- 26
insideLinkDisplayName :: Bool, -- 27
insideParagraph :: Bool, -- 28
listLevels :: [Level], -- 29
listMarkers :: [ListMarker], -- 30
continuationIndent :: Int, -- 31
footnoteNumber :: Int, -- 32
footnoteRefs :: Map Text (Int, LCI), -- 33
footnoteDefinitions :: Map Text Prose, -- 34
insideFootnoteDefinition :: Maybe Text, -- 35
prevCellIsFootnote :: Bool, -- 36
glossary :: Map Text LCI, -- 37
glossaryTermOffset :: Int, -- 38
cslPath :: Maybe FilePath, -- 39
bibPaths :: [FilePath], -- 40
insideCitation :: Bool, -- 41
insideCitePrefix :: Bool, -- 42
bracketBalance :: Int, -- 43
citations :: [Citation], -- 44
equationNumber :: Int, -- 45
anonEquationNumber :: Int, -- 46
figureNumber :: Int, -- 47
tableNumber :: Int, -- 48
orgParserConf :: OrgParserConf -- 49
}
deriving stock (Eq, Show)Running the orgDocParser 12 requires a default set of values for the OrgParserState, which we provide with f:defaultOrgParserState.
defaultOrgParserState :: OrgParserState
defaultOrgParserState =
OrgParserState
{ docOid = nilOid,
headingOidStack = [],
headingLevel = 0,
headingSectionNumber = [],
insideHeading = False,
cellNames = Set.empty,
cellNumber = LCI 0,
pseudoEdges = [],
tanglePaths = Set.empty,
insideCodeBlock = False,
anonBlockNumber = 0,
lineLabelNumber = 0,
lineLabels = M.empty,
styleMask = styleMaskFrom [],
insideLinkDisplayName = False,
insideParagraph = False,
listLevels = [],
listMarkers = [],
continuationIndent = 0,
footnoteNumber = 0,
footnoteRefs = M.empty,
footnoteDefinitions = M.empty,
insideFootnoteDefinition = Nothing,
prevCellIsFootnote = False,
glossary = M.empty,
glossaryTermOffset = 0,
cslPath = Nothing,
bibPaths = [],
insideCitation = False,
insideCitePrefix = False,
bracketBalance = 0,
citations = [],
equationNumber = 0,
anonEquationNumber = 0,
figureNumber = 0,
tableNumber = 0,
orgParserConf = defaultOrgParserConf
}Now let's examine all of the various fields of data:OrgParserState.
1.4.1 Fields for headings
There are a couple things to keep track of for Headings.
Headings may or may not have their own OID. Typically, OIDs for headings are only created when running org-store-link in Emacs, which will automatically create one for us when we want to refer to a named cell (you put your cursor over the cell and run org-store-link). All named cells within a heading use the closest OID when going up the heading hierarchy.
Headings also have a section number, starting with 1 and incrementing with subsequent headings at the same level. Each level has its own counter, separated with a dot. E.g., "1.2" and "2.13" are valid section numbers. We need to keep track of these numbers (and know how to increment them, etc), as we parse headings at different levels.
The fields below mostly deal with one of these two concerns.
The docOid 13 is what becomes the oid 1 of data:OrgDoc. If there are no headings, then this docOid is what is used as a unique prefix to identify named cells (any cell with a #+name: ..., such as a code block) from other Org files.
The headingOidStack 14 is a stack of OIDs, with the first one being the docOid and later ones being OIDs of headings encountered at deeper levels. This is used to keep track of the current OID to use for named cells; if we are at a deeply nested heading with its own OID, but then move out of it, we need to know which OID to use. Using headingOidStack, we just pop off those ones we don't need any more and refer to the OID of the closest heading up the hierarchy (ending at the docOid if needed as a fallback).
The headingLevel 15 is the last-seen heading level we parsed. See t:0004-headings-cannot-skip-level for how we use it. This is used for determining how many levels to pop off the headingOidStack when we exit a nested level. It's also used to calculate the headingSectionNumber.
The headingSectionNumber 16 is the last-generated heading section number. This is what we use to assign each new heading its section number (by incrementing the correct part of it).
The insideHeading 17 boolean is used to keep track of whether we are currently parsing things from inside a heading or not. This is used, for example, to disable parsing of citations from inside headings; i.e., citations are not recognized when they are placed inside headings (see f:proseParser).
1.4.2 Fields for cells
Cell names must be unique inside a single Org file, and this is tracked with cellNames 18.
All cells, even if they are not named, are sequentially numbered in cellNumber 19. This is used to encode the starting vertex of each edge in pseudoEdges 20 (see pseudoEdges).
1.4.3 Fields for code blocks
The tanglePaths 21 is used in f:assertUniqueTanglePath to check that no two blocks tangle to the same filesystem path.
The insideCodeBlock boolean 22 tracks whether we're inside a code block; see f:compressedTextParser for how this is used.
The anonBlockNumber 23 is used to keep track of all anonymous blocks. Anonymous blocks can link out to other named blocks, and for such links we have to refer back to the anonymous block. As these blocks are anonymous we have to give them an artificial name, which is the anonymous block number (a sequential counter). See f:blockParser for details.
1.4.4 Fields for line labels
The lineLabelNumber 24 is a sequential counter for Line labels which is reset at every code block.
The lineLabels map 25 associates the label with the label number, and this number is used to display the link in the text (source) as well as the code block (destination).
1.4.5 Fields for prose text
The styleMask 26 is discussed in the section on parsing styled strings, which are one of the main building blocks of Prose.
The insideLinkDisplayName 27 and insideParagraph 28 booleans are both used in f:textParser.
1.4.6 Fields for list items
The listLevels 29 and listMarkers 30 are discussed in List items.
Indentation is very important for knowing which things belong under which list items, and the continuationIndent 31 is used to keep track of how much indentation to expect inside a single (contiguous) piece of content; see f:paragraphParser and f:listHeadIndentLevelParser for a usage example.
1.4.7 Fields for footnotes
Footnotes are numbered (footnoteNumber 32), and also have an association with cell numbers (footnoteRefs 33). The footnote definitions are kept in a single map (footnoteDefinitions 34).
Like other booleans that tweak the behavior of the parser, footnotes also need to do this via insideFootnoteDefinition 35.
Footnote definitions must be kept in a single group of contiguous cells, and this is checked with prevCellIsFootnote 36 (also see f:footnoteDefinitionParser).
1.4.8 Fields for the glossary
Description lists, aka glossary terms are tracked with the glossary 37. The glossaryTermOffset 38 is used to improve error messages (see t:0015-glossary-duplicate-term).kj
1.4.9 Fields for citations
Citations require a number of fields. The cslPath 39, bibPaths 40, and citations 44 fields are the same in purpose as the ones discussed at Citations up above.
The insideCitation 41, insideCitePrefix 42, and bracketBalance 43 fields all deal with the lower level details of parsing these citations correctly (being strict about what we accept).
1.4.10 Fields for miscellaneous sequential counters
Mathematical equations, figures, and tables all need to get numbered. These are handled by equationNumber 45, anonEquationNumber 46, figureNumber 47, and tableNumber 48.
1.4.11 Fields for Lilac extensions
Lilac extensions can alter the behavior of the parser. These are encapsulated in the data:OrgParserConf, which we capture in orgParserConf 49.
2 Lilac.Parse module
The parsing mechanism is in the Lilac.Parse module. We discuss each parser and interleave some tests for the smaller parsers. If you are curious about the overall testing framework itself (where and how these interleaved tests are executed), take a look at the Testing framework.
{-# OPTIONS_GHC -fno-warn-orphans #-}
module Lilac.Parse
( compressProse,
compressStyledText,
defaultOrgDoc,
defaultOrgParserConf,
genSectionNumber,
genHeadingOidStack,
isLineLabel,
isNumericText,
maskToStyles,
orgDocParser,
parseOrgDoc,
styleMaskFrom,
tableHorizontalLine,
unparseLink,
unparseMathFragment,
validateOidText,
)
where
import Citeproc
( CiteprocOutput,
addDisplay,
addFontStyle,
addFontVariant,
addFontWeight,
addHyperlink,
addQuotes,
addTextCase,
addTextDecoration,
addVerticalAlign,
dropTextWhile,
dropTextWhileEnd,
fromText,
inNote,
localizeQuotes,
mapText,
movePunctuationInsideQuotes,
toText,
)
import Citeproc.Types
( DisplayStyle
( DisplayBlock,
DisplayIndent,
DisplayLeftMargin,
DisplayRightInline
),
FontStyle
( ItalicFont,
NormalFont,
ObliqueFont
),
FontVariant
( NormalVariant,
SmallCapsVariant
),
FontWeight
( BoldWeight,
LightWeight,
NormalWeight
),
TextCase
( CapitalizeAll,
CapitalizeFirst,
Lowercase,
PreserveCase,
SentenceCase,
TitleCase,
Uppercase
),
TextDecoration
( NoDecoration,
UnderlineDecoration
),
VerticalAlign
( BaselineAlign,
SubAlign,
SupAlign
),
)
import Data.Bits (setBit, testBit, (.^.))
import Data.Bits.Bitwise (toListLE)
import Data.Char
( GeneralCategory (CurrencySymbol),
generalCategory,
isAsciiLower,
isAsciiUpper,
isDigit,
isSpace,
)
import Data.List (groupBy)
import Data.Map.Strict qualified as M
import Data.Set qualified as Set
import Data.Text qualified as T
import Lilac.Types
( BToken (BTokenLinkTarget, BTokenPadding, BTokenString),
Block (Block, body, meta, parentOid),
Cell
( CBlock,
CCommand,
CFigure,
CFootnote,
CGlossaryTerm,
CHeading,
CListItem,
CMathFragment,
CParagraph,
CTable
),
Citation (Citation, cites, noteNumber, prefix, styleVariation, suffix),
Cite (Cite, id, locator, prefix, suffix),
Command (PrintBibliography, PrintGlossary),
Figure (Figure, caption, link, meta, parentOid),
Footnote (Footnote, body, name),
GlossaryTerm (GlossaryTerm, definition, term),
Heading (Heading, body, level, meta, section),
LCI (LCI, unLCI),
Level (Level, indent, number),
Link (Link, name, target),
LinkOid (LinkOid, name, oid),
LinkTarget
( LinkTargetFilePath,
LinkTargetFootnote,
LinkTargetLineLabel,
LinkTargetOid,
LinkTargetURI
),
ListMarker
( LItemEnd,
LItemStart,
LOrderedEnd,
LOrderedStart,
LUnorderedEnd,
LUnorderedStart
),
MathFragment (MathFragment, body, meta, parentOid),
OID,
OrgDoc
( OrgDoc,
bibPaths,
cells,
citations,
cslPath,
footnoteRefs,
glossary,
lineLabels,
meta,
oid,
pseudoEdges
),
OrgParser,
OrgParserConf (maxHeadingLevel, maxListLevel),
PToken
( PTokenCitation,
PTokenDivEnd,
PTokenDivStart,
PTokenLink,
PTokenMathFragment,
PTokenStyledText
),
Prose (Prose, unProse),
PseudoEdge (PseudoEdge),
Style
( Bold,
Code,
Italic,
Smallcaps,
Subscript,
Superscript,
Underline,
Verbatim
),
StyleMask (StyleMask),
StyledText (StyledText, body, style),
Table (Table, caption, grid, meta, parentOid),
defaultOrgParserConf,
defaultOrgParserState,
fromText,
markingChars,
nilOid,
protocolsEmacs,
protocolsWeb,
styleMaskFrom,
)
import Lilac.Types as OrgParserState
( OrgParserState
( anonBlockNumber,
anonEquationNumber,
bibPaths,
bracketBalance,
cellNames,
cellNumber,
citations,
continuationIndent,
cslPath,
docOid,
equationNumber,
figureNumber,
footnoteDefinitions,
footnoteNumber,
footnoteRefs,
glossary,
glossaryTermOffset,
headingLevel,
headingOidStack,
headingSectionNumber,
insideCitation,
insideCitePrefix,
insideCodeBlock,
insideFootnoteDefinition,
insideHeading,
insideLinkDisplayName,
insideParagraph,
lineLabelNumber,
lineLabels,
listLevels,
listMarkers,
orgParserConf,
prevCellIsFootnote,
pseudoEdges,
styleMask,
tableNumber,
tanglePaths
),
)
import Network.URI qualified as N
import Relude
( Bool (False, True),
Char,
Either (Left, Right),
FilePath,
Int,
Map,
Maybe (Just, Nothing),
MonadFail,
Ord,
String,
Text,
abs,
catMaybes,
concatMap,
drop,
elem,
error,
fail,
filter,
find,
flip,
fmap,
foldl',
forM_,
fromEnum,
fromList,
fromMaybe,
fst,
get,
id,
isJust,
isNothing,
length,
many,
map,
mapM_,
maybeToList,
modify,
not,
notElem,
null,
optional,
or,
otherwise,
pure,
put,
readMaybe,
repeat,
replicate,
reverse,
runStateT,
show,
sort,
take,
toEnum,
toList,
unwords,
void,
when,
zip,
($),
(&&),
(*),
(+),
(-),
(.),
(/=),
(<),
(<$>),
(<=),
(<>),
(<|>),
(=<<),
(==),
(>),
(>=),
(>>),
(||),
)
import Relude.Extra.Map
( delete,
insert,
keys,
lookup,
member,
notMember,
)
import Relude.Unsafe (fromJust)
import System.OsPath qualified as P
import Text.Megaparsec qualified as MP
import Text.Megaparsec.Char (char, string)
import Text.Megaparsec.Char.Lexer qualified as L
f:skipNewlinesAndComments
f:lexeme
f:parseOrgDoc
f:orgDocParser
f:defaultOrgDoc
f:docMetaEntryParser
f:docMetaParser
f:lilacExtensionsParser
f:maxHeadingLevelParser
f:maxListLevelParser
f:signedIntParser
f:cellParser
f:detectInvalidIndentation
f:headingParser
f:failIfHeadingNestsWithoutDirectParent
f:genSectionNumber
f:genHeadingOidStack
f:propertiesDrawerParser
f:propertiesParser
f:propertiesEntryParser
f:proseParser
f:proseSpaceConsumer
f:whitespaceCompressor
f:onlySpaces
f:styleMarkers
f:failIfLeftoverActiveStyles
f:styleMarkerParser
f:isStyleMarker
f:styledTextParser
f:textParser
f:escapedSymbolParser
f:verbatimCodeParser
f:slashSeparatedWords
f:maskToStyles
f:toStyleMask
f:styleMaskFlip
f:isVerbatim
f:isCode
f:isEmptyMask
f:compressProse
f:compressStyledText
f:assertUniqueCellName
f:assertUniqueLineLabel
f:linkParser
f:linkDisplayNameParser
f:failIfVerbatimOrCode
f:protocolWebLinkParser
f:fileLinkParser
f:rawLinkParser
f:mathFragmentInlineParser
f:paragraphParser
f:newlineAndIndentParser
f:cellBoundary
f:blockMetaEntryParser
f:assertUniqueTanglePath
f:blockParser
f:firstParentOid
f:blockStartParser
f:bTokenParser
f:zeroBTokensParser
f:someBTokensParser
f:nowebLinkTargetParser
f:lineLabelParser
f:oidLinkParser
f:isLineLabel
f:oidReferenceParser
f:compressedTextParser
f:ignoreSpecialCommas
f:compressBTokens
f:padConsecutiveNowebLinks
f:getLinkPaddingBools
f:nowebLinkAtEndOfLine
f:getBlockContents
f:zeroIndent
f:someIndent
f:getContinuationIndentForLevel
f:listItemsEndParser
f:getEndingListMarkers
f:listHeadIndentLevelParser
f:orderedHeadParser
f:unorderedHeadParser
f:listItemStartParser
f:listItemNester
f:listItemFitter
f:mathFragmentNumberedParser
f:trimTrailingWhitespace
f:mathFragmentUnnumberedParser
f:mathFragmentStandaloneParser
f:figureParser
f:metaParser
f:metaEntryParser
f:tableParser
f:tableFormulaParser
f:captionParser
f:tableLineParser
f:tableHorizontalLine
f:tableRowParser
f:tableCellParser
f:rowsToTuples
f:rowToTuples
f:footnoteLinkParser
f:footnoteReferenceParser
f:footnoteDefinitionParser
f:reconcileFootnotes
f:convertToGlossaryTerm
f:assertUniqueGlossaryTerm
f:isGlossaryTermDelimiter
f:glossaryLocationParser
f:validateOidText
f:validateCslPath
f:collectValsFor
f:bibliographyLocationParser
instance:CiteprocOutput Prose
f:unparseStyledText
f:unparseLink
f:unparseMathFragment
f:dropTextFromStyledText
f:citationParser
f:collectCitations
f:resetState
f:deleteRedundantKeys
f:failIfUnbalancedBrackets
f:citeParser
f:citeIdParser
f:citeLocatorParser
f:citeLocatorTermParser
f:citeLocatorCoordsParser
f:citeLocatorCoordParser
f:citeLocatorCoordRangeParser
f:citeAffixParser
f:isNumericText
f:isNumericTextParser
f:isPunctuation
f:isAlphaNumChar
f:isAlphaChar
f:isDigitChar
f:intraCellWhitespaceParser3 orgDocParser
At a high level, the f:orgDocParser delegates most of the work to the f:cellParser, which in turn delegates work again to the individual parsers that know how to parse different cells. We have to filter out cells that look like CListItem [] with meaningfulCell 52 because they are a by-product of parsing List items (this can happen when f:listItemsEndParser detects maximum indentation that matches the most-deeply-nested item (in which case no list items end)).
orgDocParser :: OrgParser OrgDoc
orgDocParser = do
_ <- optional skipNewlinesAndComments -- 50
(docOid, cslPath, bibPaths, meta) <- docMetaParser -- 51
_ <- lilacExtensionsParser
cells <- many cellParser
MP.eof
footnoteRefs <- reconcileFootnotes
citations <- collectCitations
pState <- get
resetState
let meaningfulCell :: Cell -> Bool
meaningfulCell = \case
CListItem [] -> False
CHeading {}
CListItem {}
CParagraph {}
CGlossaryTerm {}
CBlock {}
CMathFragment {}
CFigure {}
CTable {}
CFootnote {}
CCommand {} -> True
let meaningfulCells = filter meaningfulCell cells -- 52
when (pState.cellNumber.unLCI /= length meaningfulCells)
$ fail
( "mismatch: cellNumber: "
<> show pState.cellNumber
<> " len meaningfulCells: "
<> (show $ length meaningfulCells)
)
pure
$ OrgDoc
{ oid = docOid,
cells = meaningfulCells,
pseudoEdges = pState.pseudoEdges,
footnoteRefs = footnoteRefs,
glossary = pState.glossary,
lineLabels = pState.lineLabels,
cslPath = cslPath,
bibPaths = bibPaths,
citations = citations,
meta = deleteRedundantKeys meta ["cite_export"] -- 53
}We now look at some of the less obvious parts (other than f:cellParser for Cells) first:
Initial leading newlines and comments. The f:orgDocParser first starts by skipping any leading whitespace/comments 50. We have to do this first because all of the other parsers we use will follow the pattern of parsing the thing we want, followed by skipping over as much trailing whitespace (but not leading indentation) as possible. In other words, none of the parsers are able to deal with leading whitespace/comments, so we have to deal with it here separately as the very first thing we check for.
Document metadata. We need to parse any metadata associated with the document 51, such as the document ID, title and author. See f:docMetaParser for details.
Reconciliation of some data structures. Once we've finished parsing the entire document, we can then make assertions such as "all footnote references must have a corresponding definition" or "all referenced line labels must exist in code blocks". We call this process reconciliation and this is done for footnotes and line labels.
Resetting of some stateful fields. Some of our unit tests check for the state of the data:OrgParserState with the f:shouldParseWithState helper; more on this at
3.1 Whitespace and comments
Before writing our first parsers, we have to consider the problem of whitespace and comments.
Any syntax that is expected to be used by humans have some designation for whitespace to separate meaningful tokens from each other. One way to deal with (necessary) whitespace is to bake in whitespace parsers into each token we want to parse. This gets cumbersome so we want to instead be able to teach parsers to be able to skip over trailing whitespace. This way, these parsers will all know how to deal with whitespace in a uniform way. For example, this is what we do in f:cellParser where we wrap all of the major Cell parsers with a lexeme.
A slightly different problem is how to deal with comments. In a sense they are just like whitespace because they hold no meaningful information on their own. Luckily, Text.Megaparsec.Char.Lexer provides a space helper which can create a parser that skips over whitespace as well as comments. This is what we use below to define f:skipNewlinesAndComments. See the Megaparsec tutorial for more information.
skipNewlinesAndComments :: OrgParser ()
skipNewlinesAndComments =
L.space whiteSpaceParser singleLineCommentParser blockCommentParser -- 54
where
whiteSpaceParser :: OrgParser ()
whiteSpaceParser = do
_ <- MP.takeWhile1P (Just "newline") (== '\n') -- 55
pState <- get
_ <-
MP.optional
. MP.label "indentation for next thing"
$ MP.count pState.continuationIndent (char ' ')
pure ()
singleLineCommentParser :: OrgParser ()
singleLineCommentParser = L.skipLineComment "# " -- 56
blockCommentParser :: OrgParser ()
blockCommentParser =
L.skipBlockComment
"#+begin_comment\n"
"#+end_comment\n" -- 57The three arguments to L.space 54 (where L stands for Text.Megaparsec.Char.Lexer) are the parsers for whitespace, single-line comments, and block comments. L.space will try to repeatedly apply the three given parsers; it only stops parsing when faced with input where all three parsers fail.
We only parse newline characters in whiteSpaceParser 55. We don't parse spaces. This is because we do not want to skip over any indentation --- that is, space characters immediately following a newline. Such indentation could be an error (for example, for headings or paragraphs), or a necessary part of a cell (such as CListItem cells in f:listItemsEndParser), and so we leave that determination up to other parsers.
It's important that we use takeWhile1P for whiteSpaceParser (where the 1 denotes that it requires at least 1 newline character) because otherwise if it is allowed to accept 0 newlines (as with takeWhileP), it may loop forever upon seeing non-newline input.
Single-line comments use "# " 56, whereas multi-line comments expect an opening and closing delimiter 57. We include explicit newlines in blockCommentParser because we want to ensure that block comment markers are always on their own line (e.g., #+end_comment is valid, but #+end_comment foo is not).
Text.Megaparsec.Char.Lexer also defines a L.skipBlockCommentNested helper, which is able to understand if the ending marker of a block (#+end_comment\n for us) should be ignored because it's nested and instead skip past it to find the true (non-nested) ending marker. That's a little too complicated, so we use the non-nested L.skipBlockComment to keep the syntax simple.
Lastly we have f:lexeme, which wraps around any other parser to make it consume trailing newlines and comments. This is used in f:cellParser. We just reuse f:skipNewlinesAndComments as the main workhorse here.
lexeme :: OrgParser a -> OrgParser a
lexeme = L.lexeme skipNewlinesAndComments3.2 docMetaParser
An Org document's metadata is the first meaningful thing that could be parsed. Metadata can be created with a :PROPERTIES: drawer, or with a line that looks like #+key: value where key is a string and value is a series of tokens (it can have multiple words in it). The key and value syntax is trickier, so we look at that first with f:docMetaEntryParser below.
docMetaEntryParser :: OrgParser (Text, Text)
docMetaEntryParser = do
avoidParsingBlockMetadata -- 58
avoidParsingLilacExtensions -- 59
_ <- MP.notFollowedBy $ string "#+__" -- 60
_ <- string "#+"
key <- MP.takeWhile1P (Just "(doc meta) keyword letter") isKeywordLetter
_ <- MP.label "(doc meta) separator" $ string ": "
_ <- many $ char ' '
value <- MP.takeWhile1P (Just "(doc meta) value letter") (\c -> c /= '\n')
pure (key, value)
where
isKeywordLetter c = or $ map ($ c) [isAsciiLower, isAsciiUpper, (== '_')]
avoidParsingBlockMetadata :: OrgParser ()
avoidParsingBlockMetadata =
mapM_
MP.notFollowedBy
$ map
string
[ "#+begin_",
"#+name: ",
"#+header: ",
"#+print_bibliography:",
"#+print_glossary:"
]
avoidParsingLilacExtensions :: OrgParser ()
avoidParsingLilacExtensions = MP.notFollowedBy $ string "#+lilac_"We have to tiptoe around code blocks by checking that it is not looking at block-related metadata syntax with avoidParsingBlockMetadata 58. This is important because otherwise if an Org document begins with a code block as the first thing (before headings or anything else), we may consume that as part of docMetaParser and fail because we think we are expecting : followed by a value when we encounter #+begin_src ....
Similarly, we also do not parse any Lilac extensions 59 because they can only be parsed by the f:lilacExtensionsParser.
We avoid parsing keys with a double underscore 60, because that's reserved for internal use by Lilac. Having the restriction at 60 allows the failure condition at 113 in f:metaEntryParser to trigger.
The f:docMetaParser first checks for the top-level :PROPERTIES: drawer with f:propertiesDrawerParser. Then it just parses f:docMetaEntryParser multiple times, while consuming any trailing whitespace or comments with f:lexeme.
docMetaParser :: OrgParser (OID, Maybe FilePath, [FilePath], Map Text Text)
docMetaParser = do
topLevelDrawerProps <- propertiesDrawerParser
_ <- optional skipNewlinesAndComments
metaKVsList <- many $ lexeme docMetaEntryParser
let (bibPathsText, remainingKVs) = collectValsFor "bibliography" metaKVsList
metaKVs = fromList remainingKVs
bibPaths = map T.unpack bibPathsText
let conflictingKeys = keys $ M.intersection topLevelDrawerProps metaKVs
when (not $ null conflictingKeys)
$ fail
$ "conflicting keys in the top-level PROPERTIES drawer "
<> "and document metadata key/values: "
<> (show conflictingKeys)
case lookup "ID" topLevelDrawerProps of
Nothing -> fail $ "required document ID property not found"
Just oidText -> do
oid <- validateOidText oidText -- 61
cslPath <- case lookup "cite_export" metaKVs of
Just maybeCslPath -> validateCslPath maybeCslPath
Nothing -> pure Nothing
modify $ \s ->
s
{ OrgParserState.bibPaths = bibPaths,
OrgParserState.cslPath = cslPath,
docOid = oid
}
pure (oid, cslPath, bibPaths, topLevelDrawerProps <> metaKVs)OID validation in f:validateOidText 61 delegates to the f:fromText helper from Lilac.Types.
validateOidText :: (MonadFail m) => Text -> m OID
validateOidText oidText = case Lilac.Types.fromText oidText of
Nothing -> fail $ show oidText <> " is not a valid OID"
Just oid -> pure oidNote that using the properties drawer is the main way in which the ID field of a document is created, which is then used by Orgmode (see org-id-link-to-org-use-id, org-id-link-consider-parent-id, and org-id-locations-file). Using ID's for things (instead of the file name) allows us to create links that are more resilient against file name changes (or directory changes). See Drawers for more discussion about properties drawers and ti:0002-doc-metadata-properties-drawer.
The special key #+cite_export: ... is parsed separately as the cslPath, using the f:validateCslPath helper. This is needed for supporting Citations.
3.2.1 Lilac extensions
Lilac supports extensions to alter the behavior of the parser for the document being parsed. These extensions are all prefixed by lilac_ and are set like other document metadata; they must come after the other (typical) Org metadata.
Below is a list of supported extensions:
- lilac_maxHeadingLevel
- 🔗 f:maxHeadingLevelParser This takes an integer as a value. Using -1 means no headings are allowed at all. Using 0 means headings can have as many asterisks as desired. Using a positive integer means headings may only have that many (or less) asterisks; otherwise Lilac will fail the parse.
- lilac_maxListLevel
- 🔗 f:maxListLevelParser This has the same semantics as lilac_maxHeadingLevel but is for list items, not headings.
These extensions are encoded inside the data:OrgParserConf.
type OrgParserConf :: Type
data OrgParserConf = OrgParserConf
{ maxHeadingLevel :: Int,
maxListLevel :: Int
}
deriving stock (Eq, Show)The default preferences are saved in f:defaultOrgParserConf.
defaultOrgParserConf :: OrgParserConf
defaultOrgParserConf =
OrgParserConf
{ maxHeadingLevel = 3,
maxListLevel = 3
}If you want to deviate from the defaults, you have to specify these in your top-level document metadata, for f:lilacExtensionsParser to pick up.
lilacExtensionsParser :: OrgParser ()
lilacExtensionsParser = do
void
. many
$ MP.choice
[ maxHeadingLevelParser,
maxListLevelParser
]The f:maxHeadingLevelParser resembles the f:docMetaEntryParser, but the main difference is that it does not return anything (its type is OrgParser ()). Instead it just modifies the pState.
maxHeadingLevelParser :: OrgParser ()
maxHeadingLevelParser = do
pState <- get
_ <- string $ "#+lilac_maxHeadingLevel: "
_ <- many $ char ' '
val <- signedIntParser
let opc = pState.orgParserConf {maxHeadingLevel = val}
put pState {orgParserConf = opc}The f:signedIntParser knows how to parse a signed integer. The parserBetweenSignAndNumber is the parser for parsing the area between the sign (-) and the number. Here we define it as pure () meaning that nothing is allowed. The usual use case here is to use a whitespace parser, so as to allow a space between the sign and number, but we don't do this here for simplicity.
signedIntParser :: OrgParser Int
signedIntParser = L.signed parserBetweenSignAndNumber number
where
number :: OrgParser Int
number = L.lexeme skipNewlinesAndComments $ L.decimal
parserBetweenSignAndNumber :: OrgParser ()
parserBetweenSignAndNumber = pure ()f:maxListLevelParser closely resembles f:maxHeadingLevelParser, and similarly updates the orgParserConf field inside f:defaultOrgParserState.
maxListLevelParser :: OrgParser ()
maxListLevelParser = do
pState <- get
_ <- string $ "#+lilac_maxListLevel: "
_ <- many $ char ' '
val <- signedIntParser
let opc = pState.orgParserConf {maxListLevel = val}
put pState {orgParserConf = opc}3.3 Resetting of state
We call f:resetState in f:orgDocParser to clear some state from data:OrgParserState, because otherwise it gets too polluted during testing. Also, it's not like we're throwing away information because everything we need is still captured elsewhere (just check the fields that have the same name in both data:OrgParserState and data:OrgDoc).
resetState :: OrgParser ()
resetState =
modify $ \s ->
s
{ OrgParserState.bibPaths = [],
OrgParserState.citations = [],
OrgParserState.cslPath = Nothing,
OrgParserState.cellNumber = LCI 0,
OrgParserState.pseudoEdges = []
}In f:resetState we cannot simply refer to the field alone in the record update, like this
modify (\s -> s {citations = []})because it will result in an "Ambiguous record update" error, because both data:OrgDoc and data:OrgParserState have a field named citations. So we work around this by importing the latter through import Lilac.Types as OrgParserState in module:Lilac.Parse. See this discussion from 2024 for more information.
In a similar vein to f:resetState, f:deleteRedundantKeys 53 removes certain entries from the given map data structure. In the case of f:orgDocParser, it removes keys that won't be used because the information in those keys have already been parsed and saved elsewhere.
deleteRedundantKeys :: (Ord k) => Map k v -> [k] -> Map k v
deleteRedundantKeys = foldl' (flip M.delete)3.4 Cells
The cell is the main building block of an Org document --- it is roughly any region of text surrounded by blank lines.
type Cell :: Type
data Cell
= CHeading Heading
| CListItem [ListMarker]
| CParagraph Prose
| CBlock Block
| CMathFragment MathFragment
| CFigure Figure
| CTable Table
| CFootnote Footnote
| CGlossaryTerm GlossaryTerm
| CCommand Command
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)All of the different cell types describe some sort of content, except for one, CListItem. The CListItem is purely structural because it only describes itself and relies on other cells for displaying any useful information. See List items to see how this is used.
The data:Command encodes information about generated content, such as Printing bibliographies.
type Command :: Type
data Command
= PrintBibliography
| PrintGlossary Bool
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)The f:cellParser is responsible for parsing these different recognized values of data:Cell.
cellParser :: OrgParser Cell
cellParser = do
cell <-
lexeme
$ MP.choice
[ toggleFootnote False $ listItemStartParser,
toggleFootnote False $ listItemsEndParser,
toggleFootnote False $ resetIndent headingParser,
toggleFootnote False $ resetIndent blockParser,
toggleFootnote False $ resetIndent figureParser,
toggleFootnote False $ resetIndent tableParser,
toggleFootnote False $ resetIndent mathFragmentStandaloneParser,
toggleFootnote True $ resetIndent footnoteDefinitionParser,
toggleFootnote False $ resetIndent bibliographyLocationParser,
toggleFootnote False $ resetIndent glossaryLocationParser,
toggleFootnote False $ resetIndent paragraphParser,
detectInvalidIndentation
]
modify (\s -> s {bracketBalance = 0}) -- 62
when (cell /= CListItem [])
$ modify (\s -> s {cellNumber = LCI $ s.cellNumber.unLCI + 1})
pure cell
where
resetIndent :: OrgParser a -> OrgParser a -- 63
resetIndent p = do
a <- p
modify (\s -> s {continuationIndent = 0})
pure a
toggleFootnote :: Bool -> OrgParser a -> OrgParser a
toggleFootnote b p = do
a <- p
modify (\s -> s {prevCellIsFootnote = b})
pure aThe resetIndent wrapper 63 resets the continuationIndent attribute 31 in the data:OrgParserState back to 0; this information is useful for parsers that need to understand indentation precisely. See f:listItemsEndParser for how it's used by that function.
The f:listItemStartParser and f:listItemsEndParser are the only parsers that can consume leading indentation! The f:paragraphParser can consume leading indentation, but only on lines except the first one (so that it can ensure that paragraphs that are part of a list item can have all of its lines line up together). For example, the following paragraph is inside a list item
- Hello
world.and will be parsed as a string Hello world and not as Hello world (the leading indent for world will be ignored).
Any other indentation should be treated as being invalid. To enforce this, we put in a f:detectInvalidIndentation parser which technically isn't a parser but which will throw an error if it is able to successfully parse indentation. The idea is that if none of the other parsers in the list of parsers in f:cellParser are able to consume indentation, then this will allow detectInvalidIndentation to see such indentation and fail the overall parser.
We signal the error with invalid ... 65 so as to not shadow the typical unexpected ... language from Megaparsec's usual parse failures. Note also that the pure $ CListItem [] 66 is only there to satisfy the type checker --- we will always fail if the takeWhile1P 67 succeeds (the MP.fancyFailure 64 will not be executed if takeWhile1P fails).
The bracketBalance is set to 0 62 in f:cellParser after a cell is parsed, because we're not interested in cross-cell issues about unbalanced brackets. See f:failIfUnbalancedBrackets.
The rest of the top-level headings this document is mainly concerned with describing all of the different building blocks used for constructing cells.
3.5 Tests
These tests check for mostly empty documents, and also those with some metadata.
We should be able to parse empty input.
it "empty input" $ do
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
"empty input"
nilOidOrgDocPrefix
`shouldParse` (L.defaultOrgDoc nilOid [], defaultOrgParserState)The f:defaultOrgDoc is just a default (mostly blank) OrgDoc. It's useful for testing.
defaultOrgDoc :: OID -> [(Text, Text)] -> OrgDoc
defaultOrgDoc oid metas =
OrgDoc
{ oid = oid,
cells = [],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta = fromList metas'
}
where
metas' =
if oid == nilOid
then ("ID", "00000000-0000-0000-0000-000000000000") : metas
else metasWe should also be able to parse blank documents (not empty, but only has whitespace).
We have to put in two newlines above because during parsing we will lose one newline (it is used to parse the "\n#+end_src" string; see f:someBTokensParser). This leaves us with just one newline in test/Lilac/ParseSpec/0001-blankline.
shouldParseWith
"0001-blankline"
(L.defaultOrgDoc nilOid [])Aside: ti:0001-blankline in HTML looks like there is no content (this is intentional), because f:zeroBTokensParser parses it the same way it parses a block with zero bytes in it (see ti:0007-blocks-empty).
Parsing comments should also result in a blank document.
# This is a single-line comment.
#+begin_comment
This is inside
a block comment.
#+end_commentshouldParseWith
"0001-comments-only"
(L.defaultOrgDoc nilOid [])At the other end of the spectrum, Lilac should be able to parse itself.
before (LU.readFrom "" "index.org") $ do
it "self-parse" $ \input -> do
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
"index.org"
`shouldSucceedOn` input3.5.1 Document metadata
Below are some unit test cases for parsing metadata.
#+title: An Org Document
#+author: A. U. Thor
#+property: header-args+ :noweb no-export
#+auto_tangle: tshouldParseWith
"0002-doc-metadata"
$ L.defaultOrgDoc
nilOid
[ ("author", "A. U. Thor"),
("auto_tangle", "t"),
("property", "header-args+ :noweb no-export"),
("title", "An Org Document")
]Note that we currently don't do any intelligent processing of the metadata fields. For example, the value of the property key above is a mess because it itself contains a key and value (:noweb and no-export).
Below is the same as the test above, but with some whitespace and comments sprinkled in between.
#+title: An Org Document
# This is a comment line.
#+author: A. U. Thor
# This is another comment line.
# Note the whitespace above this line.
#+property: header-args+ :noweb no-export
# Below is a block comment.
#+begin_comment
Yet another comment.
#+end_comment
#+auto_tangle: t
# Last comment followed by some trailing newlines.
shouldParseWith
"0003-doc-metadata-with-comments"
$ L.defaultOrgDoc
nilOid
[ ("author", "A. U. Thor"),
("auto_tangle", "t"),
("property", "header-args+ :noweb no-export"),
("title", "An Org Document")
]Indentation is invalid for document metadata. Even single-line comments may not be indented.
# invalid comment (it's indented)shouldFailWithMessage
"0008-indentation-comment"
"invalid indentation"
1Metadata values can have leading whitespace. We ignore such whitespace.
#+title: Parse me without leading whitespaceshouldParseWith
"0002-doc-metadata-whitespace-before-value"
$ L.defaultOrgDoc
nilOid
[ ("title", "Parse me without leading whitespace")
]Metadata values are required (there must be some non-whitespace text for it).
#+title:shouldFailWithError
"0002-doc-metadata-missing-value"
( utoks ":\n"
<> elabel "(doc meta) separator"
<> elabel "(doc meta) keyword letter"
)
7Metadata can also come from top-level properties drawers.
:PROPERTIES:
:ID: 7af1aa3e-41ca-4810-a001-2fdf10768f2f
:END:shouldParseWithState
"0002-doc-metadata-properties-drawer"
( L.defaultOrgDoc
(fromWords 0x7af1aa3e41ca4810 0xa0012fdf10768f2f)
[ ("ID", "7af1aa3e-41ca-4810-a001-2fdf10768f2f")
]
)
defaultOrgParserState
{ docOid = fromWords 0x7af1aa3e41ca4810 0xa0012fdf10768f2f
}If the :ID: is missing, we assign a nil (all zero bits) OID. We reuse the same test case file (ti:0001-blankline) from t:0001-blankline.
shouldParseWithState
"0001-blankline"
(L.defaultOrgDoc nilOid [])
defaultOrgParserStateMetadata from the top-level properties drawer and also the key/value metadata lines get combined together into a single map.
:PROPERTIES:
:ID: 7af1aa3e-41ca-4810-a001-2fdf10768f2f
:END:
#+title: An Org Document
#+author: A. U. Thor
#+property: header-args+ :noweb no-exportshouldParseWithState
"0002-doc-metadata-properties-drawer-combine"
( L.defaultOrgDoc
(fromWords 0x7af1aa3e41ca4810 0xa0012fdf10768f2f)
[ ("ID", "7af1aa3e-41ca-4810-a001-2fdf10768f2f"),
("author", "A. U. Thor"),
("property", "header-args+ :noweb no-export"),
("title", "An Org Document")
]
)
defaultOrgParserState
{ docOid = fromWords 0x7af1aa3e41ca4810 0xa0012fdf10768f2f
}If there is a clash, error out instead of silently choosing precedence.
:PROPERTIES:
:ID: 00000000-0000-0000-0000-000000000000
:foo: bar
:END:
#+title: An Org Document
#+author: A. U. Thor
#+property: header-args+ :noweb no-export
#+foo: xyzbefore
( LU.readFrom
"test/Lilac/ParseSpec/"
"0002-doc-metadata-properties-drawer-clash"
)
$ do
it "0002-doc-metadata-properties-drawer-clash" $ \input -> do
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
"0002-doc-metadata-properties-drawer-clash"
input
`shouldFailWith` ( errFancy 181
. fancy
$ MP.ErrorFail
( "conflicting keys in the top-level PROPERTIES"
<> " drawer and document metadata"
<> " key/values: [\"foo\"]"
)
)4 Styled text
A data:StyledText is just some text that has one or more styles attached to it. Here are some examples: bold, italic, underline, verbatim, and code.
type StyledText :: Type
data StyledText = StyledText
{ body :: Text,
style :: StyleMask
}
deriving stock (Eq, Generic, Ord, Show)
deriving anyclass (FromJSON, ToJSON)The newtype:StyleMask is a low-level bit mask that can store multiple boolean values of each of the supported data:Style bits. If the string has no style, then it's not stylized at all and the mask will be 0 (no bits set). See StyleMask details for a discussion of how this data structure is used and transformed into a list of data:Style values. For now, we can just pretend that the StyleMask is like a list of data:Style values.
In order to save some keystrokes (especially in test cases), we use a couple functions, f:stext and f:ptext.
stext :: Text -> [Style] -> StyledText
stext body styles = StyledText {body = body, style = styleMaskFrom styles}ptext :: Text -> [Style] -> PToken
ptext body styles = PTokenStyledText $ stext body stylesThe data:Style type is straightforward enough. For now we don't support all possible styles supported by Orgmode (such as strike-through text), for both simplicity but also prescriptive reasons. For example, strike-through text should never be needed during exposition.
type Style :: Type
data Style
= Bold
| Italic
| Underline
| Verbatim
| Code
| Smallcaps
| Subscript
| Superscript
deriving stock (Enum, Eq, Generic, Ord, Show)
deriving anyclass (FromJSON, ToJSON)The f:styleMarkers keeps track of the characters that activate and deactivate each style. It has the syntactic characters we use to mark text with the a supported style.
We don't actually allow users to use the Smallcaps, Subscript, or Superscript styles themselves (this is why they're missing from f:styleMarkers); these are used inside f:cslJsonTextToProse and f:weaveStyledText to preserve these styles that CslJson might generate for citation processing.
styleMarkers :: Map Char Style
styleMarkers =
fromList
[ ('*', Bold),
('/', Italic),
('_', Underline),
('=', Verbatim),
('~', Code)
]Characters in the f:styleMarkers can be escaped with a backslash. For example, the strings 1\*2 and a\_b are interpreted as 1*2 and a_b, because they act to escape the style markers * and _, respectively. If you want to write a literal backslash, you must use \. This use of the backslash as an escape character deviates from Orgmode.
This rule also applies for slashes, such as 1\/2 in ti:0005-styled-strings-escape-marker. However, for slashes there is a simpler (and arguably more intuitive syntax) where slashes inside a word are not treated as style markers, so they don't need to be escaped. See f:slashSeparatedWords and ti:0005-styled-strings-slash-separated-words.
/foo and\/or 1\/2 bar \\ a\\b/shouldParseWith
"0005-styled-strings-escape-marker"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-escape-marker
]
[][ ptext "foo and/or 1/2 bar \\ a\\b" [Italic]
]4.1 Parsing
Parsing styled strings is somewhat tricky, because of two main problems:
the style markers use the same character for both activation and deactivation, and
the styles can be nested or interleaved.
You can have, for example, lines that are italicized like this but also have bold text inside of it.
We approach these problems with a simple idea: use a bit mask (newtype:StyleMask) to keep track of markers. Every time we see a starting marker, we turn on the bit in the mask to activate that style (if it's off), or turn it off (if it's on), by calling f:styleMaskFlip.
If the input is such that the markers are not balanced, such that by the time we finish parsing a paragraph (or end of a heading) and the mask still has active styles in it, we throw an error with f:failIfLeftoverActiveStyles. That is, we expect style starting and ending markers to always be balanced by the end of every paragraph or heading. This deviates from vanilla Org, which does not have this restriction.
failIfLeftoverActiveStyles :: OrgParser ()
failIfLeftoverActiveStyles = do
pState <- get
let (StyleMask mask) = pState.styleMask
when (mask /= 0) $ do
fail
$ "leftover active styles: "
<> (show $ maskToStyles $ StyleMask mask)We don't really care which styles are turned on or off at which point (the style markers can be used at the beginning, middle, or end of a word --- where a word is any sequence of non-whitespace characters). The only thing we check is that by the end of the cell (heading or paragraph), all styles must be turned off.
* *foo *bar baz a*b*c (*ok*) **yesshouldParseWithState
"0005-styled-strings-messy-boundaries"
( nilOrgDoc
[ CHeading
Heading
{ level = 1,
section = [1],
body =
Prose
Expected prose for 0005-styled-strings-messy-boundaries,
meta = fromList []
}
]
[]
)
defaultOrgParserState
{ headingOidStack = [Nothing],
headingLevel = 1,
headingSectionNumber = [1]
}[ ptext "foo " [Bold],
ptext "bar baz a" [],
ptext "b" [Bold],
ptext "c (" [],
ptext "ok" [Bold],
ptext ") yes" []
]In ti:0005-styled-strings-leftover-active-styles we keep parsing to the end of the heading, expecting to see the ending style marker for bold text, but never see it. So the failure from f:failIfLeftoverActiveStyles gets triggered.
* foo *bar bazshouldFailWithMessage
"0005-styled-strings-leftover-active-styles"
"leftover active styles: [Bold]"
15Note that as soon as we see a style marker we stylize the text at hand; the StyleMask is "strict" in this sense. This way, we can start parsing stylized text without needing to see a corresponding ending marker. This imperative style is easy to think about.
The algorithm for parsing data:StyledText requires parsing style markers (and only style markers), or parsing other text that will be affected by style markers. Let's first look at the simpler case, that of parsing style markers.
styleMarkerParser :: OrgParser StyledText
styleMarkerParser = do
pState <- get
let mask = pState.styleMask
c <- MP.satisfy $ isStyleMarker mask
styleMask <- toStyleMask c
let newMask = styleMaskFlip mask styleMask -- 68
put pState {styleMask = newMask}
pure
StyledText
{ body = "",
style = newMask
}All we do is parse a single style marker character, and flip the bit for it as either on or off, with f:styleMaskFlip 68. Ideally we wouldn't even bother returning anything at the end of f:styleMarkerParser (because we've updated the pState and that's the only essential thing we must do). But because of the way it is used in f:proseParser, we have to return a StyledText. We return an empty body field though, so that we can filter it out with the nonEmpty helper in f:proseParser.
We cannot parse multiple style markers in one go, because f:isStyleMarker changes behavior depending on the current active style mask. If we are in a verbatim or code style, we ignore all other style markers except for those that will end the current (verbatim or code) style. Otherwise, we won't be able to write stuff like * because we would think that the asterisk in there is starting a bold style (which it is not because it is surrounded by ~ to mark code).
isStyleMarker :: StyleMask -> Char -> Bool
isStyleMarker styleMask c
| isVerbatim styleMask = c == '='
| isCode styleMask = c == '~'
| otherwise = member c styleMarkersParsing actual styled text requires reading the state of the StyleMask, and then reading some text. This is done in f:styledTextParser. It delegates essentially all of the work to f:textParser.
styledTextParser :: OrgParser StyledText
styledTextParser = do
pState <- get
let mask = pState.styleMask
text <- textParser mask
pure
StyledText
{ body = text,
style = mask
}The f:textParser is careful not to step on other parsers' toes, because in a sense it is the most "greedy" parser, as it can parse many different types of input.
textParser :: StyleMask -> OrgParser Text
textParser styleMask
| isVerbatim styleMask || isCode styleMask = do
t <- verbatimCodeParser styleMask
pure $ T.map newlineToSpace t -- 69
| otherwise = do
o <- MP.getOffset
word <-
MP.choice
[ MP.try $ slashSeparatedWords,
MP.label "normal characters" normalChars,
escapedSymbolParser styleMask -- 70
]
pState <- get
let haveNotSeenGlossaryTermDelimiterYet =
pState.insideParagraph && pState.glossaryTermOffset == 0
wordIsGlossaryTermDelimiter = word == "::" && isEmptyMask styleMask
when
( haveNotSeenGlossaryTermDelimiterYet
&& wordIsGlossaryTermDelimiter
)
$ do
modify (\s -> s {glossaryTermOffset = o}) -- 71
pure word
where
newlineToSpace c = if c == '\n' then ' ' else c
normalChars :: OrgParser Text
normalChars = do
cs <- MP.some $ normalChar
pure $ T.pack cs
normalChar :: OrgParser Char
normalChar = do
pState <- get
let toAvoid =
[ (pState.insideCitation, string ";"),
(pState.insideCitePrefix, citeIdParser),
( pState.insideCitation
&& (pState.bracketBalance == 0),
string "]"
),
(pState.insideLinkDisplayName, string "]]"),
( not pState.insideCitation
&& not pState.insideCitePrefix
&& not pState.insideLinkDisplayName,
string "[["
)
]
forM_
toAvoid
$ \(condition, p) -> when condition $ MP.notFollowedBy p
c <- MP.satisfy isNormalChar
let bracketBalanceDiff
| c == '[' = 1
| c == ']' = -1
| otherwise = 0
modify (\s -> s {bracketBalance = s.bracketBalance + bracketBalanceDiff})
pure c
isNormalChar c =
notMember c styleMarkers
&& (not $ isSpace c)
&& (not $ c == '\\')For the verbatim and code styles, we replace newlines with spaces with calls to newlineToSpace 69, because in practice newlines can get inserted only because we're wrapping the source text (in Org mode) at 80 columns. If the user wants to put in a piece of multiline text with specific newlines, they should just use a source code block instead of an inline verbatim or code style.
The verbatim and code styles rely on f:verbatimCodeParser to parse text inside those styles. We support escaping the closing style marker with a backslash \, because this is the only way to allow using the styles with the closing markers. That is, if we just did a simple
MP.takeWhile1P (Just "code") (\c -> c /= '~')then there would be no way to write something like
~~~to have ~ (tilde) inside a code style. You'd have to write
=~=to enclose the tilde inside a verbatim style. However, because of 72, we can write
~\~~to stylize a single tilde with the code style. The disadvantage is that we have to always escape a backslash \ if we want just a single backslash without doing any escaping.
verbatimCodeParser :: StyleMask -> OrgParser Text
verbatimCodeParser styleMask = do
parts <-
MP.some
$ MP.choice
[ escapedSymbol, -- 72
MP.takeWhile1P Nothing isNotClosingOrEscape
]
pure $ T.concat parts
where
escapedSymbol :: OrgParser Text
escapedSymbol = do
_ <- char '\\'
c <- MP.satisfy (\c -> not $ isSpace c)
if
| isVerbatim styleMask && c `elem` ['=', '\\'] -> pure $ T.singleton c
| isCode styleMask && c `elem` ['~', '\\'] -> pure $ T.singleton c
| otherwise -> pure $ "\\" <> T.singleton c
isNotClosingOrEscape c
| isVerbatim styleMask = c /= '=' && c /= '\\'
| otherwise = c /= '~' && c /= '\\'Back to f:textParser. We take special care to escape style markers with f:escapedSymbolParser 70.
escapedSymbolParser :: StyleMask -> OrgParser Text
escapedSymbolParser styleMask = do
_ <- char '\\'
s <-
MP.choice
[ do
c <- MP.satisfy (\c -> isStyleMarker styleMask c || c == '\\')
pure $ T.singleton c,
string "n",
string "vert{}"
]
pure $ case s of
"vert{}" -> "|"
"n" -> "\n"
_ -> sf:escapedSymbolParser also lets users insert a newline character with \n. This way, you can write multiline text inside a table, even if the table is not taking up the full width of the HTML page (at which point the browser will insert soft newlines on its own).
The word \vert{} can be used to escape a pipe character inside table cells, because although our Megaparsec-powered parser is powerful enough to handle escaping pipe characters with \| (for consistency with \/ to escape the italic style marker), Orgmode is unable to deal with this in Emacs (so pressing TAB will mess up the alignment of table cells). For this reason, we use \vert{} instead. Of course, outside of tables the pipe character can be written raw like this "|" without issue.
There is one exception to the "style markers must always be balanced" rule, and it's what we call "slash-separated words". These are cases where we see one or more slash characters inside an alphanumeric word, such as and/or, or a plain (non-math) fraction like 1/2, or even something like 1/2/3. In these cases, we don't want the slashes to be treated as italic style markers.
slashSeparatedWords :: OrgParser Text
slashSeparatedWords = do
let wordParser :: OrgParser Text
wordParser = do
_ <- cannotStartWithClosingPunc
MP.takeWhile1P (Just "slash-separated words") isWordChar
firstWord <- wordParser
remainingWords <-
MP.some $ do
_ <- MP.notFollowedBy $ MP.choice [string "/ ", string "/\n"]
_ <- char '/'
word <- wordParser
pure $ "/" <> word
when (lastWordIsMeaningless $ reverse remainingWords)
$ fail
"""
last word is not meaningful; unlikely to be
intended as a slash-separated-word
"""
let allWords = firstWord : remainingWords
when (hasLinkDelimiter allWords)
$ fail
"""
has link delimiter \"[[\"; unlikely to be
intended as a slash-separated-word
"""
pure $ T.concat allWords
where
cannotStartWithClosingPunc :: OrgParser ()
cannotStartWithClosingPunc =
MP.notFollowedBy
$ MP.choice
[ string "]",
string "}",
string ")",
string "]]"
]
isWordChar c = not (isSpace c) && c `notElem` (keys styleMarkers <> "\\")
lastWordIsMeaningless :: [Text] -> Bool
lastWordIsMeaningless = \case
[] -> True
(lastWord : [])
| lastWord `elem` ["/.", "/,", "/:"] -> True
| otherwise -> False
_ -> False
hasLinkDelimiter :: [Text] -> Bool
hasLinkDelimiter = or . map ("[[" `T.isInfixOf`)The rules inside f:slashSeparatedWords are a "best effort" attempt, not something that is always correct. For the cases where we need more flexibility, you can use escaped slashes like in ti:0005-styled-strings-escape-marker.
See ti:0005-styled-strings-slash-separated-words for some real-world examples.
4.2 StyleMask details
The previous section explained how we parse styled strings without getting into the implementation behind the newtype:StyleMask. We now get into these implementation details.
The StyleMask is an 8-bit word, which can hold boolean values for the 6 different Style data constructors, e.g. Bold, Italic, etc. It's a space-efficient type to replace an equivalent map or list of [Style] values. We use a newtype around a Word8 (an 8-bit word) so as to keep it distinct from Word8 itself.
type StyleMask :: Type
newtype StyleMask = StyleMask Word8
deriving stock (Eq, Generic, Ord)
deriving anyclass (FromJSON, ToJSON)We have a custom Show instance because the default Show would just treat the Word8 as a number. Instead we want to show what the binary bits (turned on inside the Word8) would represent in terms of actual styles. We have to import Prelude because of the way Relude redefines show (and we need to use the show that Prelude uses; see this related discussion for the Basic Prelude library, a Relude alternative, for reference).
instance Show StyleMask where
show (StyleMask a) = "styleMaskFrom " <> x
where
x =
Prelude.show
. catMaybes
. map
( \(i, b) ->
if b
then (Just $ (toEnum i :: Style))
else Nothing
)
. zip [0 ..]
$ toListLE aThe shown text string is prefixed with f:styleMaskFrom because that function is used as a constructor function to create a StyleMask easily from scratch. This follows the tradition used in Data.Map where they also use a constructor function (fromList) when show-ing a Map data structure.
styleMaskFrom :: [Style] -> StyleMask
styleMaskFrom ss = StyleMask $ foldl' f (0 :: Word8) ss
where
f :: Word8 -> Style -> Word8
f acc s = setBit acc (fromEnum s)Similarly, we sometimes want to go the other direction --- that is, deconstruct a StyleMask into [Style]. So we have f:maskToStyles.
maskToStyles :: StyleMask -> [Style]
maskToStyles (StyleMask mask) = foldl' f [] $ zip [0 ..] $ toListLE mask
where
f :: [Style] -> (Int, Bool) -> [Style]
f acc (idx, b) =
if b
then (toEnum idx) : acc
else accWe have a helper function f:toStyleMask to convert Char (a style marker character) into a StyleMask. This is used to construct a style mask from a style marker in the input Org doc. The toStyleMask function runs inside a MonadFail monad, because the Char type could hold a value that is not a valid style marker. The use of MonadFail is seamless inside ParsecT (the monad we use inside OrgParser), because ParsecT has a MonadFail instance.
toStyleMask :: (MonadFail m) => Char -> m StyleMask
toStyleMask cs = pure =<< styleToStyleMask =<< charToStyle cs
charToStyle :: (MonadFail m) => Char -> m Style
charToStyle c = case lookup c styleMarkers of
Just style -> pure style
Nothing -> fail $ "programmer error: unrecognized style marker " <> (show c)
styleToStyleMask :: (MonadFail m) => Style -> m StyleMask
styleToStyleMask s = pure $ StyleMask (setBit 0 $ fromEnum s)f:styleMaskFlip is a helper to perform a binary XOR operation on two StyleMask objects, so that callers don't have to perform the deconstruction and reconstruction of the underlying Word8 primitive themselves. This operation is used to either turn on bits, or turn off bits if they're already on.
styleMaskFlip :: StyleMask -> StyleMask -> StyleMask
styleMaskFlip (StyleMask a) (StyleMask b) = StyleMask $ a .^. bWe also have some helpers to check whether particular styles are active.
isVerbatim :: StyleMask -> Bool
isVerbatim (StyleMask a) = testBit a (fromEnum Verbatim)isCode :: StyleMask -> Bool
isCode (StyleMask a) = testBit a (fromEnum Code)f:isEmptyMask checks whether any styles are active.
isEmptyMask :: StyleMask -> Bool
isEmptyMask (StyleMask a) = a == 0We stated earlier around t:0005-styled-strings-leftover-active-styles that we don't have to wait to see the ending style marker to start parsing styled text. While this is true, the downside is that by the time we construct a StyledText, we have no knowledge of future StyledText objects that have not yet been parsed. This can lead to a proliferation of StyledText objects that all share the same style, but are still parsed into separate PToken objects. That is, instead of just one PToken with a single StyledText object with a sizable body field, we could end up with [PToken] where each of the elements has a small body, but with the same overall style.
Obviously, having a list of PToken objects when we can just have a single one is problematic, if not wasteful. We want to join up sequences of styled text that share the same style, and make them use a single StyledText object. For this purpose, there is f:compressProse.
compressProse :: Prose -> Prose
compressProse prose =
Prose
. reverse
. dropUnnecessary -- 73
$ foldl' f [] prose.unProse
where
f :: [PToken] -> PToken -> [PToken]
f [] token = token : []
f
((t1@(PTokenStyledText (StyledText body1 style1))) : rest)
t2@(PTokenStyledText (StyledText body2 style2))
| style1 == style2 =
(PTokenStyledText $ StyledText (T.concat [body1, body2]) style1)
: rest
| otherwise = t2 : t1 : rest
f acc token = token : acc
dropUnnecessary [] = []
dropUnnecessary (((PTokenStyledText (StyledText " " _))) : rest) = rest
dropUnnecessary (((PTokenStyledText (StyledText "" _))) : rest) = rest
dropUnnecessary tokens = tokensThis function works by looking at two consecutive PTokenStyledText elements at a time (one that's already in the accumulator (acc) from the previous iteration and one from the current token at hand), joining them if they share the same style.
f:compressProse also eliminates trailing whitespace, with dropUnnecessary 73. Trailing whitespace can occur if the last word of a paragraph at the end of the file ends with an active StyleMask followed by a newline (trailing whitespace between paragraphs can't occur because the lexeme in f:cellParser will skip past them). For example, if the input text looks like
_foo
bar_then the last underscore can be followed by a newline (assuming the editor used to edit this file is UNIX-friendly and appends a trailing newline), in which case that newline would get parsed by the spaceConsumer in f:paragraphParser as its own separate StyledText object as a single space character. We want to eliminate these objects because they don't add any useful information for Lilac itself, and so we use dropUnnecessary above to filter them out.
dropUnnecessary also removes tokens with an empty string as the StyledText object. This can occur when we parse trailing newlines at the end of the line even without style markers, from f:whitespaceCompressor.
Here we test how we minimize the number of PToken objects; although the individual letters are spaced out with multiple spaces, we only use two PToken objects (one for the plain text and another for the bold text) and each of these objects have just single spaces between the letters.
a b c d e *f g h i j*shouldParseWith
"0005-styled-strings-compression"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-compression
]
[][ ptext "a b c d e " [],
ptext "f g h i j" [Bold]
]f:compressStyledText is like f:compressProse, but only for data:StyledText. It's used in f:cslJsonTextToProse to cut down on the verbosity of StyledText items generated by the conversion of CslJson to Prose.
compressStyledText :: [StyledText] -> [StyledText]
compressStyledText = reverse . dropUnnecessary . foldl' f []
where
f :: [StyledText] -> StyledText -> [StyledText]
f [] styledText = styledText : []
f
((t1@(StyledText body1 style1)) : rest)
t2@(StyledText body2 style2)
| style1 == style2 =
(StyledText (T.concat [body1, body2]) style1) : rest
| otherwise = t2 : t1 : rest
dropUnnecessary [] = []
dropUnnecessary ((StyledText " " _) : rest) = rest
dropUnnecessary ((StyledText "" _) : rest) = rest
dropUnnecessary styledTexts = styledTexts4.3 Additional tests
The simple case with no style markers.
fooshouldParseWith
"0005-styled-strings-no-style"
$ nilOrgDoc
[ plainParagraph "foo"
]
[]The next case tests a simple case where one word is in bold.
foo *bar* bazshouldParseWith
"0005-styled-strings-bold"
$ nilOrgDoc
[ CParagraph
$ Prose
[ ptext "foo " [],
ptext "bar" [Bold],
ptext " baz" []
]
]
[]Below we mix things up a bit more, using more styles.
*foo /bar _underlined bold italic_/ baz* =verbatim text= ~x y z~ *bold*shouldParseWith
"0005-styled-strings-complex"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-complex
]
[][ ptext "foo " [Bold],
ptext "bar " [Bold, Italic],
ptext "underlined bold italic" [Bold, Italic, Underline],
ptext " baz" [Bold],
ptext " " [],
ptext "verbatim text" [Verbatim],
ptext " " [],
ptext "x y z" [Code],
ptext " " [],
ptext "bold" [Bold]
]Here's another test, which checks some edge cases when dealing with punctuation.
(foo =bar=). What's up? and *that!* /(italic)/. It's #1!shouldParseWith
"0005-styled-strings-complex-punctuation"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-complex-punctuation
]
[]Note how we preserve the information of having no space character between the bar and the closing parenthesis ")".
[ ptext "(foo " [],
ptext "bar" [Verbatim],
ptext "). What's up? and " [],
ptext "that!" [Bold],
ptext " " [],
ptext "(italic)" [Italic],
ptext ". It's #1!" []
]Check that we can parse slashes inside words as slashes, not as italic style markers.
/italic fraction 1/2/ foo and/or bar /A+/B-/ +1/+2. key/valueshouldParseWith
"0005-styled-strings-slash-separated-words"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-slash-separated-words
]
[][ ptext "italic fraction 1/2" [Italic],
ptext " foo and/or bar " [],
ptext "A+/B-" [Italic],
ptext " +1/+2. key/value" []
]This is another test, involving the Verbatim style. It checks the case where we have a style marker inside a word (not at the usual beginning or end of a word).
(=import ./.=) helloshouldParseWith
"0005-styled-strings-verbatim"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-verbatim
]
[][ ptext "(" [],
ptext "import ./." [Verbatim],
ptext ") hello" []
]This is another test to check that the bold style marker (asterisk) is ignored if inside a verbatim or code marker.
=*a= =b*= ~*c~ ~d*~shouldParseWith
"0005-styled-strings-verbatim-code-ignore-immediate-bold"
$ nilOrgDoc
[ CParagraph
$ Prose
Expected prose for 0005-styled-strings-verbatim-code-ignore-immediate-bold
]
[][ ptext "*a" [Verbatim],
ptext " " [],
ptext "b*" [Verbatim],
ptext " " [],
ptext "*c" [Code],
ptext " " [],
ptext "d*" [Code]
]Verbatim and code styles take precedence over links. So even if you have text that looks like a link, they will not be parsed as such if surrounded by verbatim or code styles. The same applies for inline math, whose parser should not apply if we're inside a verbatim (or code) region.
=[[foo=
[[hello]] there
=\(1 + 1 = ~\(2 * 2\)~
~[[bar]]~shouldParseWith
"0005-styled-strings-verbatim-precedence"
$ nilOrgDoc
Expected prose for 0005-styled-strings-verbatim-precedence
[PseudoEdge (LCI 1, LinkOid {oid = fromWords 0 0, name = "hello"})][ CParagraph
$ Prose
[ ptext "[[foo" [Verbatim]
],
CParagraph
$ Prose
[ PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "hello"
},
name =
[ stext "hello" []
]
},
ptext " there" []
],
CParagraph
$ Prose
[ ptext "\\(1 + 1 " [Verbatim],
ptext " " [],
ptext "\\(2 * 2\\)" [Code]
],
CParagraph
$ Prose
[ ptext "[[bar]]" [Code]
]
]ti:0005-styled-strings-escaping checks how we handle symbol escaping inside verbatim and code styles (compare against f:verbatimCodeParser).
=\\= =\== =\(= ~\\~ ~\~~ ~\(~shouldParseWith
"0005-styled-strings-escaping"
$ nilOrgDoc
Expected prose for 0005-styled-strings-escaping
[][ CParagraph
$ Prose
[ ptext "\\" [Verbatim],
ptext " " [],
ptext "=" [Verbatim],
ptext " " [],
ptext "\\(" [Verbatim],
ptext " " [],
ptext "\\" [Code],
ptext " " [],
ptext "~" [Code],
ptext " " [],
ptext "\\(" [Code]
]
]5 Links
A link has
- a name
- 🔗 what's displayed to the reader, and
- a target
- 🔗 where we want to take the reader when they click on the link.
type Link :: Type
data Link = Link
{ name :: [StyledText],
target :: LinkTarget
}
deriving stock (Eq, Generic, Ord, Show)
deriving anyclass (FromJSON, ToJSON)The data:LinkTarget is a rich definition of the kind of target we are linking to. For now, this data structure only supports the following cases:
data:LinkOid, which links to either another part of the current Org document or another (external) Org document. "External" doesn't mean any Org document; it must be compiled inside the same data:Archive.
These are also used for Noweb reference inside another code block (because code blocks are just cells). They can be used to also link to figures, which are also cells. Example
When inside code blocks, these use angle brackets (e.g., <<foo>>). This can reference code blocks in other files by the Org ID (OID), where the syntax is <<id:...::foo>> where ... is the OID and foo is the name of the code block at that OID (either using the top-level OID of the document or the OID of the closest enclosing heading). We have to use an OID-to-file map data structure (which is parsed here).
When used outside of code blocks, square brackets are used (e.g., [[target]] or [[target][name]]) where target is just a plain text string (such as a URL or some other entity in the document), and name is what is displayed to the user when the link is rendered.
A line label, which is a link that points to a reference inside a code block. This is useful for referring to different areas inside a code block, without having to break up the code block into multiple separate parts.
A footnote definition. These also use data:LinkOid internally, and always refer to footnote definitions within the same Org document.
URI (a typical web link, most common example being https://...).
File path. These links are only useful because they let you go open that file on disk locally in Emacs. On weaving these links are checked if they refer to a path inside the current Git repo. If they are inside, then we calculate what the link should target (based on the weaving configuration), assuming that the entire Git repo is also being served somewhere on the internet.
If the file path is pointing to a place outside the Git repo, then upon weaving we display it as a path description (and not as a link), because the link will be broken anyway (again, assuming that only the content inside the Git repo is accessible by a link).
There are also Raw links which is prose text that looks like a URL. For example, https://www.haskell.org is an example of a raw link. This is a convenience facility because [[https://www.haskell.org]] behaves the same way as https://www.haskell.org.
The data:LinkOid is made up of the Org ID and the name. This is used pervasively as it is the only way to refer to headings (within the same doc or in an external Org doc). It is also required for constructing Noweb links that refer to code blocks from external documents. See f:oidReferenceParser to see the expected format of these links.
We have a custom Show instance for data:LinkOid so that it's much more concise (especially when we're marshaling and unmarshaling from JSON).
instance Show LinkOid where
show (LinkOid (OID uuid) name)
| T.null name = "id:" <> show uuid
| otherwise = "id:" <> show uuid <> "::" <> T.unpack name5.1 OIDs and Emacs
Lilac relies on Org files that are written in Emacs (as it is the best environment for writing Org files). When the org-id-locations-file and org-id-track-globally variables are set, Emacs remembers which files have which Org IDs (where the "ID" is the OID generated by org-id-get-create and org-id-store-link). Emacs then saves this information to the value of org-id-locations-file.
Historically Lilac used to parse these files generated by Emacs, but it has moved on to a compilation pipeline that doesn't need to rely on these files coming from Emacs. If Lilac wants to bring back tangling and weaving based purely on Org files without the compilation process, it will be forced to read these OID location files again. Below is a brief discussion of the format of these files to serve as a historical reference.
The syntax is of a list of list of strings, in Emacs lisp. The inner lists have the relative file path of the Org file as the first element, followed by the OIDs. Each Org ID is just a UUIDv4 string. The OIDs are in reverse order (probably because Emacs grows the list right-to-left, just like Haskell's : (aka "cons") function).
Below is an example, with newlines inserted for better presentation (Emacs does not use newlines here, and so in the wild everything is in one line):
(("note/raw/recreation.org" "a1e6d0d5-4540-423b-864e-b96ad06c56eb")
("../prog/lilac/index.org" "3c576a02-418b-431b-ab96-e4936e977fb7"
"ce9af5ef-6559-4ffb-82c1-a439214eda6d"
"8f5752c0-6162-4de6-9ceb-f78ba584722c"
"b449a097-d702-4fd6-98de-cb81f7e49177"))The information stored in this file is global, because Emacs will update it whenever it encounters Org files.
5.2 Standalone link parser
The standalone link parser f:linkParser can parse links that can be embedded inside prose text. In practice this is probably the most important link parser because it knows how to parse links found in prose text (outside of code blocks).
linkParser :: OrgParser Link
linkParser = do
failIfVerbatimOrCode
MP.choice
[ MP.try oidLinkParser,
MP.try protocolWebLinkParser,
MP.try fileLinkParser,
MP.try footnoteLinkParser,
MP.try rawLinkParser
]The call to f:failIfVerbatimOrCode fails the parser if either the verbatim or code style is active. This is because the verbatim and code styles take precedence over parsing links. See, for example t:0005-styled-strings-verbatim-precedence of the sort of behavior we expect.
failIfVerbatimOrCode :: OrgParser ()
failIfVerbatimOrCode = do
pState <- get
let mask = pState.styleMask
when (isVerbatim mask || isCode mask)
$ fail "aborting because verbatim or code style is active"By conditionally failing f:linkParser with f:failIfVerbatimOrCode, we also allow other parsers (such as f:styledTextParser) to succeed (see f:proseParser for the interaction of this parser with these other parsers).
Note that in f:linkParser we don't parse Noweb links here because they can only exist inside code blocks.
5.3 OID links
The f:oidLinkParser closely resembles the f:nowebLinkTargetParser, because they both encode a data:LinkOid, the only difference being where they are used (Noweb links are used exclusively inside code blocks to refer to other code blocks).
oidLinkParser :: OrgParser Link
oidLinkParser = do
_ <- string "[["
avoidParsingKnownProtocols
(oid, name) <- oidReferenceParser "[]" "]]" -- 76
pState <- get
let mask = pState.styleMask
implicitDisplayName
| T.null name = "id:" <> T.show oid
| otherwise = name
implicitDisplayNameStyledText
| isLineLabel name = []
| otherwise =
[ StyledText
{ body = implicitDisplayName,
style = mask
}
]
_ <- string "]"
displayName <-
MP.choice
[ MP.try $ string "]" >> pure implicitDisplayNameStyledText,
MP.try linkDisplayNameParser -- 77
]
let linkOid =
LinkOid
{ oid = oid,
name = name
}
linkTargetConstructor =
if isLineLabel name
then LinkTargetLineLabel -- 78
else LinkTargetOid
edge = PseudoEdge (pState.cellNumber, linkOid) -- 79
modify (\s -> s {OrgParserState.pseudoEdges = edge : s.pseudoEdges})
pure
$ Link
{ target = linkTargetConstructor linkOid,
name = displayName
}
where
avoidParsingKnownProtocols :: OrgParser ()
avoidParsingKnownProtocols =
MP.notFollowedBy
. MP.choice
. map (MP.try . string)
$ protocolsWeb
<> protocolsEmacsThere are three main pieces involved here:
f:oidReferenceParser 76 for parsing the location of the linked item (anything that can be named, such as a code block, line label, figure, or table);
f:linkDisplayNameParser 77 for parsing the display name of the link (if any); and
so-called "pseudo" edges 79.
We keep the opening and closing parentheses from the name of a line label because even though the LinkTargetLineLabel value constructor at 78 captures whether the outer LinkOid belongs to a line label, it's still useful to differentiate it against another LinkOid with an identical name. For example, if a document has a cell named foo and a line label like (ref:foo), we would like to store these as LinkOid names foo and (foo) respectively. See ti:0401-broken-links-line-label for a case where we have a line label link, but no corresponding line label reference in a code block.
So in practice although we can just pattern match on this constructor to know whether the data:LinkOid contained within it is a line label or not, we still have to inspect the name (with f:isLineLabel) if we're only dealing with the inner LinkOid type.
Pseudo-edges are needed for backlink generation. Now let's look at the first two pieces in more detail.
5.3.1 OID references
The OID reference uniquely identifies the linked item. Below are some examples:
[[foo]]
[[id:62a7d69a-7aa7-4607-b93e-7ed382a5f256][link display name]]
[[id:62a7d69a-7aa7-4607-b93e-7ed382a5f256::target name][link display name]]The first one, [[foo]], is called an implicit link; it links to some other cell (block, table, figure, etc) within the current document, without (explicitly) specifying the OID of the current document. All documents must have an OID, and it is parsed in f:docMetaParser. Implicit links only specify the target (in this example, foo.
The second one, [[id:62a7d69a-7aa7-4607-b93e-7ed382a5f256][link display name]], refers to an OID explicitly. The OID could be either a document itself (the 1) or the OID of a heading (the latter is much more common in practice). Note that these OIDs can be any OID, not just from the current document. This example also has a link display name as the text to display to the user when we weave this to HTML. These link display names are parsed separately in f:linkDisplayNameParser (Link display names).
The third example has a target name in it. This name may or may not have newlines in it.
For all three examples, the part in the first set of brackets (foo, id:62a7d69a-7aa7-4607-b93e-7ed382a5f256, and id:62a7d69a-7aa7-4607-b93e-7ed382a5f256::name of target item) are all parsed by f:oidReferenceParser.
oidReferenceParser :: [Char] -> [Char] -> OrgParser (OID, Text)
oidReferenceParser disallowedChars ending =
MP.choice
[ MP.try explicitOIDandTargetName,
MP.try explicitOID,
MP.try implicitOIDandExplicitTargetName
]
where
implicitOIDandExplicitTargetName :: OrgParser (OID, Text)
implicitOIDandExplicitTargetName = do
pState <- get
name <- targetName
pure (pState.docOid, name) -- 80
explicitOID = do
_ <- string "id:" -- 81
oidText <-
MP.some $ do
_ <- MP.notFollowedBy . string $ T.pack ending
MP.noneOf $ ':' : disallowedChars
oid <- validateOidText $ T.pack oidText
pure (oid, "")
explicitOIDandTargetName = do
(oid, _) <- explicitOID
_ <- string "::"
name <- targetName
pure (oid, name)
targetName :: OrgParser Text
targetName = compressedTextParser disallowedChars ending -- 82The implicit OID links get their OID assigned for the current document through the pState.docOid 80, and use targetName 82 to parse the name of the target. We cannot use the OID of the closest heading (like block.parentOid for blocks 120) here, because for an internal Noweb link we would almost always be referring to another different heading in the doc which would most likely have a different OID than the parentOid. So the best one to use is docOid. See f:buildGraph for how the docOid (aka document's OID) is used for generating a graph for cycle detection.
For links with an explicit OID, the OID is prefixed with id: 81.
The parsing of the target name is a little bit tricky, because sometimes the user may want to break long lines with whitespace. This is why f:compressedTextParser is more lenient about how whitespace is interpreted, so that if the user wants to refer to a target that has spaces in it, they can write that target over multiple lines. That is, a user may refer to a target name like foo bar baz in the following ways:
[[foo bar
baz]]
[[foo
bar
baz]]
[[foo
bar baz]]All of the above links refer to the same foo bar baz named target in the current document, even though they are written slightly differently.
compressedTextParser :: [Char] -> [Char] -> OrgParser Text
compressedTextParser disallowedChars ending = do
pState <- get
tn <-
proseParser'
$ proseSpaceConsumer'
pState.insideCodeBlock
pure $ compressWhitespace tn
where
compressWhitespace = T.unwords . T.words
proseSpaceConsumer' :: Bool -> OrgParser Text -- 83
proseSpaceConsumer' insideCodeBlock
| insideCodeBlock = onlySpaces
| otherwise =
MP.choice
[ string "\n",
onlySpaces
]
proseParser' :: OrgParser Text -> OrgParser Text
proseParser' spaceConsumer = do
a <- MP.choice baseParsers
b <- MP.many $ MP.choice (spaceParser : baseParsers)
pure . T.concat $ a : b
where
spaceParser = whitespaceCompressor spaceConsumer
baseParsers =
[ textParser'
]
textParser' :: OrgParser Text
textParser' = do
ws <-
MP.some
$ MP.choice
[ MP.label "normal characters" normalChars
]
pure $ T.concat ws
where
normalChars :: OrgParser Text
normalChars = do
cs <- MP.some $ normalChar
pure $ T.pack cs
normalChar :: OrgParser Char
normalChar = do
_ <- MP.notFollowedBy . string $ T.pack ending
MP.satisfy isNormalChar
isNormalChar c =
notElem c disallowedChars
&& (not $ isSpace c)Unlike f:proseSpaceConsumer, the proseSpaceConsumer' 83 does not check for the correctness of the indentation following a newline (it does not take an expectedIndentLen parameter). This is because we want to be more generous for the case where OID links are found in prose, because sometimes indentation can get silently added when using fill-paragraph in Emacs (if the OID link target gets unlucky and ends up getting hard wrapped by fill-paragraph). Otherwise, this parser will fail, which would then make the OID link become wrongly parsed.
Now we could punish the user for having erroneous indentation, but because org-toggle-link-display can hide this error caused by fill-paragraph (and because the link target contents will be hidden), we take the approach of being a bit lenient here. We still disallow newlines inside the link target inside code blocks, which is why insideCodeBlock is still a parameter for proseSpaceConsumer'. See ti:0006-links-oids-lenient-whitespace.
5.3.2 Link display names
f:linkDisplayNameParser just gets the name of the text to display. It is reused in f:protocolWebLinkParser. It uses the proseParser' 84 helper which is very similar to the f:proseParser, except that instead of returning data:PToken values, it returns the finer-grained data:StyledText. We want to be careful about how we treat whitespace (especially across indentation levels), so we reuse the same f:proseSpaceConsumer that f:paragraphParser uses.
linkDisplayNameParser :: OrgParser [StyledText]
linkDisplayNameParser = do
pState <- get
modify (\s -> s {insideLinkDisplayName = True})
linkDisplayName <-
MP.between
(string "[")
(string "]]")
(proseParser' $ proseSpaceConsumer pState.continuationIndent)
modify (\s -> s {insideLinkDisplayName = False})
pure linkDisplayName
where
proseParser' :: OrgParser Text -> OrgParser [StyledText] -- 84
proseParser' spaceConsumer = do
a <- MP.choice baseParsers
b <- MP.many $ MP.choice (spaceParser : baseParsers)
pure . filter nonEmpty $ a : b
where
spaceParser = do
s <- whitespaceCompressor spaceConsumer
pure StyledText {body = s, style = styleMaskFrom []}
baseParsers =
[ styleMarkerParser,
styledTextParser
]
nonEmpty :: StyledText -> Bool
nonEmpty styledText = not $ T.null styledText.body5.3.3 Tests
Here are some basic links.
foo [[link-target-only]]
* [[link-target][link-name]]
[[a]] bshouldParseWithState
"0006-links"
( nilOrgDoc
[ CParagraph
$ Prose
[ ptext "foo " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "link-target-only"
},
name = [stext "link-target-only" []]
}
],
CHeading
Heading
{ level = 1,
section = [1],
body =
Prose
[ PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "link-target"
},
name = [stext "link-name" []]
}
],
meta = fromList []
},
CParagraph
$ Prose
[ PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "a"
},
name = [stext "a" []]
},
ptext " b" []
]
]
[ PseudoEdge (LCI 2, LinkOid {oid = fromWords 0x0000000000000000 0x0000000000000000, name = "a"}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0 0, name = "link-target-only"})
]
)
defaultOrgParserState
{ headingOidStack = [Nothing],
headingLevel = 1,
headingSectionNumber = [1]
}Sometimes we have text that look like links initially but turn out to be non-links.
[hi]shouldParseWith
"0006-links-nonlink"
$ nilOrgDoc
[ CParagraph $ Prose [ptext "[hi]" []]
]
[]Bracketed links can be preceded by ( which means f:linkParser won't be able to parse it directly; however, in f:textParser (specifically normalChar) we take care to ensure to stop consuming input if we see a double left bracket [[. This way, the input will look like [[... which will then allow f:linkParser to parse it. See ti:0006-links-inside-parentheses for an example.
ti:0006-links-oids has some more complex input.
[[id:442c807f-c9d9-45a7-88bf-ffffffffffff]]
[[id:442c807f-c9d9-45a7-88bf-ffffffffffff][bar]]
[[id:442c807f-c9d9-45a7-88bf-ffffffffffff][bar =[]=]]
[[id:442c807f-c9d9-45a7-88bf-ffffffffffff][bar
ok]]
[[id:442c807f-c9d9-45a7-88bf-ffffffffffff::foo][abc]]
[[foo][abc]]
- link name across list indentation [[foo][/italic/
def ghi]
!]]
- link target with spaces [[foo bar
baz]]
- oid with spaces [[id:442c807f-c9d9-45a7-88bf-ffffffffffff::a b
c]]shouldParseWith
"0006-links-oids"
( nilOrgDoc
[ CParagraph
$ Prose
[ PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = ""
},
name =
[ stext "id:442c807f-c9d9-45a7-88bf-ffffffffffff" []
]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = ""
},
name =
[ stext "bar" []
]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = ""
},
name =
[ stext "bar" [],
stext " " [],
stext "[]" [Verbatim]
]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = ""
},
name =
[ stext "bar" [],
stext " " [],
stext "ok" []
]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = "foo"
},
name = [stext "abc" []]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "foo"
},
name = [stext "abc" []]
}
],
CListItem
[ LUnorderedStart,
LItemStart
],
CParagraph
$ Prose
[ ptext "link name across list indentation " [],
PTokenLink
Link
{ name =
[ stext "italic" [Italic],
stext " " [],
stext "def" [],
stext " " [],
stext "ghi]" [],
stext " " [],
stext "!" []
],
target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "foo"
}
}
],
CListItem
[ LItemEnd,
LItemStart
],
CParagraph
$ Prose
[ ptext "link target with spaces " [],
PTokenLink
Link
{ name = [stext "foo bar baz" []],
target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "foo bar baz"
}
}
],
CListItem
[ LItemEnd,
LItemStart
],
CParagraph
$ Prose
[ ptext "oid with spaces " [],
PTokenLink
Link
{ name = [stext "a b c" []],
target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = "a b c"
}
}
],
CListItem
[ LItemEnd,
LUnorderedEnd
]
]
[ PseudoEdge (LCI 6, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = "a b c"}),
PseudoEdge (LCI 4, LinkOid {oid = fromWords 0x0000000000000000 0x0000000000000000, name = "foo bar baz"}),
PseudoEdge (LCI 2, LinkOid {oid = fromWords 0x0000000000000000 0x0000000000000000, name = "foo"}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x0000000000000000 0x0000000000000000, name = "foo"}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = "foo"}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = ""}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = ""}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = ""}),
PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = ""})
]
)If we encounter a link in the middle of a paragraph (where the OID portion is broken up incorrectly such that there is additional whitespace at the beginning of a line), we should still be able to parse the \n as a single space character because that's almost always the intent. This situation can happen when invoking fill-paragraph in Emacs with org-toggle-link-display.
So in ti:0006-links-oids-lenient-whitespace, even though the string (if we parse it naively) should look like with\n newline, that \n will get compressed so that we just get with newline instead.
Hello [[id:442c807f-c9d9-45a7-88bf-ffffffffffff::target name with
newline and space is still treated as space][foo]]shouldParseWith
"0006-links-oids-lenient-whitespace"
( nilOrgDoc
[ CParagraph
$ Prose
[ ptext "Hello " [],
PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff,
name = "target name with newline and space is still treated as space"
},
name = [stext "foo" []]
}
]
]
[PseudoEdge (LCI 0, LinkOid {oid = fromWords 0x442c807fc9d945a7 0x88bfffffffffffff, name = "target name with newline and space is still treated as space"})]
)A link to a line label will only have a link display name if the user defines it explicitly. That is, [[(foo)]] won't have a name 85, but [[(foo)][explicit name]] will have one 86.
#+begin_src txt
(ref:foo)
#+end_src
[[(foo)]] [[(foo)][explicit name]]shouldParseWithState
"0006-links-oids-line-labels"
( OrgDoc
{ oid = nilOid,
cells =
[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "1"),
("__block_type", "src"),
("__src_language", "txt")
],
body =
[ BTokenLinkTarget
$ LinkTargetLineLabel
LinkOid {oid = nilOid, name = "(foo)"}
]
},
CParagraph
$ Prose
[ PTokenLink
Link
{ name = [], -- 85
target =
LinkTargetLineLabel
LinkOid {oid = nilOid, name = "(foo)"}
},
ptext " " [],
PTokenLink
Link
{ name =
[ stext "explicit" [], -- 86
stext " " [],
stext "name" []
],
target =
LinkTargetLineLabel
LinkOid {oid = nilOid, name = "(foo)"}
}
]
],
pseudoEdges =
[ PseudoEdge
( LCI 1,
LinkOid
{ oid = nilOid,
name = "(foo)"
}
),
PseudoEdge
( LCI 1,
LinkOid
{ oid = nilOid,
name = "(foo)"
}
)
],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = fromList [("foo", 1)],
cslPath = Nothing,
bibPaths = [],
citations = [],
meta = fromList [("ID", "00000000-0000-0000-0000-000000000000")]
}
)
defaultOrgParserState
{ anonBlockNumber = 1,
lineLabelNumber = 1,
lineLabels = fromList [("foo", 1)]
}5.4 Links to recognized web protocols
Links to web protocols like https:// should be parsed as a LinkTargetURI, but not just for Raw links but also those links found inside square brackets, like [[http://abc.xyz]] and [[http://abc.xyz][foo]].
protocolWebLinkParser :: OrgParser Link
protocolWebLinkParser = do
maybeUri <-
MP.between
(string "[[")
(string "]")
uriParser
pState <- get
let mask = pState.styleMask
implicitName = maybeUri
implicitNameStyledText =
StyledText
{ body = implicitName,
style = mask
}
displayName <-
MP.choice
[ MP.try $ string "]" >> pure [implicitNameStyledText],
MP.try linkDisplayNameParser
]
case N.parseURI (T.unpack maybeUri) of
Nothing ->
fail $ "(protocol web link) not a valid URI: " <> (T.unpack maybeUri)
Just uri ->
pure
$ Link
{ target = LinkTargetURI uri,
name = displayName
}
where
uriParser :: OrgParser Text
uriParser = do
protocol <- MP.choice $ map (MP.try . string) protocolsWeb
rest <- MP.takeWhile1P (Just "name character") (/= ']')
pure $ protocol <> rest[[http://foo.org]]
[[https://foo.org]]
[[https://foo.org][hello]]
[[mailto:x@foo.org][email]]shouldParseWith
"0006-links-web-protocols"
( nilOrgDoc
[ CParagraph
$ Prose
[ PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "http:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "foo.org",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "http://foo.org" []]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "foo.org",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "https://foo.org" []]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "foo.org",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "hello" []]
},
ptext " " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "mailto:",
N.uriAuthority = Nothing,
N.uriPath = "x@foo.org",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "email" []]
}
]
]
[]
)5.5 Filesystem links
Orgmode lets you create links like
[[file:foo]]where foo is a path on the local filesystem (you can select the path using completion from Emacs). These links are useful for linking out to a file which you can open up using Emacs.
Practically speaking, in only makes sense to use these links to point to files that are already tracked by the Git repo that we're in (if we're using Lilac as a static site generator), because otherwise we might risk creating a broken link (assuming that the published site only includes files that are tracked in Git).
fileLinkParser :: OrgParser Link
fileLinkParser = do
maybeFilePath <-
MP.between
(string "[[")
(string "]")
filePathParser
pState <- get
let mask = pState.styleMask
implicitName = maybeFilePath
implicitNameStyledText =
StyledText
{ body = implicitName,
style = mask
}
filePath = T.unpack maybeFilePath
displayName <-
MP.choice
[ MP.try $ string "]" >> pure [implicitNameStyledText],
MP.try linkDisplayNameParser
]
if
| Just osPath <- P.encodeUtf filePath,
P.isValid osPath ->
pure
$ Link
{ target = LinkTargetFilePath filePath,
name = displayName
}
| otherwise -> fail $ "not a valid file path: " <> filePath
where
filePathParser :: OrgParser Text
filePathParser = do
_ <- string "file:"
fp <- MP.takeWhile1P (Just "name character") (/= ']')
pure fp([[file:foo/bar]])shouldParseWith
"0006-links-inside-parentheses"
$ nilOrgDoc
[ CParagraph
$ Prose
[ ptext "(" [],
PTokenLink
Link
{ target = LinkTargetFilePath "foo/bar",
name = [stext "foo/bar" []]
},
ptext ")" []
]
]
[]5.6 Raw links
The f:rawLinkParser parses text that look like URLs into LinkTargetURI. These raw links do not use the double bracket syntax like [[ ... ]] but instead just have the raw URI as is directly in the prose. Generally speaking, you should always use the double bracket syntax if you can, and only use raw links if they are short enough to fit on a single line.
rawLinkParser :: OrgParser Link
rawLinkParser = do
_ <- MP.try . MP.lookAhead $ do
_ <- MP.takeWhile1P (Just "(raw link) protocol") isAlphaChar
_ <- string "://"
pure ()
candidate <- MP.some $ MP.try remainderChar
case N.parseURI candidate of
Nothing -> fail $ "(raw link) not a valid URI: " <> candidate
Just uri ->
pure
Link
{ target = LinkTargetURI uri,
name =
[ StyledText
{ body = T.pack candidate,
style = styleMaskFrom []
}
]
}
where
remainderChar :: OrgParser Char -- 87
remainderChar = do
c <- MP.satisfy (\c -> N.isAllowedInURI c && c `notElem` [')'])
when (isPunctuation c && c /= '/') $ do
MP.notFollowedBy (char ' ')
MP.notFollowedBy (char '\n')
pure cThe tricky part is in the remainderChar 87. Essentially, if we see a raw link followed by some punctuation, like https://funloop.org, then we want the punctuation plus the whitespace immediately after it to be excluded from the link. The punctuation may or may not be part of the allowed characters in N.isAllowedInURI. We fix this problem by running some additional checks if we ever encounter punctuation, with f:isPunctuation, where the f:isAlphaNumChar helper is taken from network-uri.
However, the above rule about excluding punctuation doesn't apply for the / (slash) character because sometimes a URL may end with a slash. In such cases, we would still want to include the slash as part of the URL.
The f:isPunctuation parser (and its parts) are generic and useful enough that we expose them as top-level functions.
isPunctuation :: Char -> Bool
isPunctuation = not . isAlphaNumCharisAlphaNumChar :: Char -> Bool
isAlphaNumChar c = isAlphaChar c || isDigitChar cisAlphaChar :: Char -> Bool
isAlphaChar c = (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z')isDigitChar :: Char -> Bool
isDigitChar c = c >= '0' && c <= '9'See https://foo.org, or https://bar.net.
Try http://localhost:3000?
Go to ftp://10.0.0.1!
https://a.com/path/to/something/shouldParseWith
"0006-links-raw"
$ nilOrgDoc
[ CParagraph
$ Prose
[ ptext "See " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "foo.org",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "https://foo.org" []]
},
ptext ", or " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "bar.net",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "https://bar.net" []]
},
ptext ". Try " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "http:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "localhost",
N.uriPort = ":3000"
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "http://localhost:3000" []]
},
ptext "? Go to " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "ftp:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "10.0.0.1",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "ftp://10.0.0.1" []]
},
ptext "! " [],
PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "a.com",
N.uriPort = ""
},
N.uriPath = "/path/to/something/",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "https://a.com/path/to/something/" []]
}
]
]
[]5.7 Noweb links
Noweb links, or references, are LinkOid links to other code blocks. We've been using Noweb references already in this document, but here's an isolated example.
echo some messageThe block above refers to a Noweb reference named some message. We define it below:
hello worldNote that if no such Noweb reference is defined, Lilac will error out. See f:nowebLinkTargetParser for parsing details (it is discussed along with Blocks because that's the only place where these links may be defined).
Noweb links are powered by data:LinkOid which can reference blocks in the current Org file or any other Org file (as long as those other Org files are compiled and archived together).
The ability for Noweb links to reference blocks in other files is not part of Orgmode and is Lilac's own extension. This allows source code blocks to be defined across multiple Org files if needed. One simple example is version information that we might want to define once in a single place, but which might be referenced in other places, across multiple Org files. This is exactly what we do ourselves with GHC version, which is referenced in nix:ghcVersion for example.
5.8 Line labels
Line labels are embedded inside a block, and can be linked to from outside the block. Below is an example; we give two code blocks, the first with the line labels disabled and the second with the same content but with line labels enabled, so that you can see the full syntax if you're reading this from HTML:
(save-excursion ; (ref:a)
(goto-char (point-min)) ; (ref:foo)The (ref:a) and (ref:foo) will become targets that can be linked to via [[(a)]] and [[(foo)]]. So now you can write 88 and 89 to refer to specific lines in the example above.
5.8.1 Parsing of labels inside code blocks
The line label must start with a (ref: and end with a closing parenthesis ); this is the syntax that Orgmode uses, and which we also adopt in f:lineLabelParser:
lineLabelParser :: OrgParser LinkTarget
lineLabelParser = do
_ <- string "(ref:"
o <- MP.getOffset
lineLabel <- do
let disallowedChars = "()[]<>" :: [Char]
ending = ">>" :: [Char]
MP.label "line label" $ compressedTextParser disallowedChars ending
_ <- string ")"
assertUniqueLineLabel lineLabel o
pState <- get
pure
$ LinkTargetLineLabel
LinkOid
{ oid = pState.docOid,
name = "(" <> lineLabel <> ")"
}This uses f:compressedTextParser to let the user refer to line labels more freely, without having to be exact about whitespace.
The line labels must be unique inside a document; otherwise you'll run into a parse error, per f:assertUniqueLineLabel. f:assertUniqueLineLabel is virtually the same as f:assertUniqueCellName.
assertUniqueLineLabel :: Text -> Int -> OrgParser ()
assertUniqueLineLabel lineLabel offset = do
pState <- get
when (member lineLabel pState.lineLabels) $ do
MP.setOffset offset
fail
$ "line label "
<> show lineLabel
<> " is not unique"
let num = pState.lineLabelNumber + 1
put
pState
{ lineLabels =
insert
lineLabel
num
pState.lineLabels,
lineLabelNumber = num
}#+begin_src sh
a (ref:x)
#+end_src
#+begin_src sh
b (ref:x)
#+end_srcshouldFailWithMessage
"0006-unique-line-labels"
"line label \"x\" is not unique"
58The f:lineLabelParser is used in f:bTokenParser, if the block being parsed does not have #+header: :line-label off.
5.8.2 Parsing of links to line labels
The syntax for linking to a line label (ref:foo) is [[(foo)]]. Note the parentheses surround the line label, to distinguish it from other cells that can be named (via #+name: ..., such as other blocks). A cell's name may not begin with an opening parenthesis, so that [[(foo)]] always links to the line label named foo, not a cell named (foo); this prohibition of an opening parenthesis is codified in f:metaEntryParser at 112.
Parsing of line label links is handled by f:oidLinkParser, which uses f:isLineLabel to inspect the name of the LinkOid to see whether we are dealing with a line label or not.
isLineLabel :: Text -> Bool
isLineLabel s = T.take 1 s == "(" && T.takeEnd 1 s == ")"Unlike other links that use the double bracket syntax, these line label links cannot be given a custom name for how they are displayed inside code blocks; instead they are always named numerically when woven (see f:weaveLink), although when referring to a line label in an external document we also include the name of the code block for additional context f:weaveLinkDisplayName → 34.
During weaving, line labels are numbered starting at 1, and increment upwards. The lineLabels and lineLabelNumber fields in data:OrgParserState keep track of this numbering.
reconcileLineLabels :: OrgParser (Map Text Int)
reconcileLineLabels = do
pState <- get
let referenced = pState.lineLabelReferences
defined = M.keysSet pState.lineLabels
brokenLinks = Set.difference referenced defined
unusedDefs = Set.difference defined referenced
hasBrokenLinks = length brokenLinks /= 0
hasUnusedDefs = length unusedDefs /= 0
brokenLinksMsg = T.unpack . unwords $ toList brokenLinks
unusedDefsMsg = T.unpack . unwords $ toList unusedDefs
if
| hasBrokenLinks -> fail $ "broken line label links: " <> brokenLinksMsg
| hasUnusedDefs -> fail $ "unused line labels: " <> unusedDefsMsg
| otherwise -> pure pState.lineLabels5.8.3 Tangling behavior
By default, line labels are stripped in the tangling process in f:expandBlock. However, sometimes you'd want to tangle these line labels in the source code. One such use case is if you are using a linter for the source code, that doesn't like trailing comment characters.
You can do that with
#+header: :line-label tanglefor the source code block in question. See nix/release.nix for an example.
5.9 Footnotes
Footnotes are essentially named paragraphs. They look like this in practice:
Hello world [fn:foo].
...
* Footnotes
[fn:foo] A paragraph that defines this footnote.So a footnote is made up of both the link to the footnote from the main text, and a definition of the footnote at the bottom. This requirement of needing to define them somewhere in the document is akin to Line labels.
Both the link and the definition share the same footnote reference, which is parsed with the f:footnoteReferenceParser.
footnoteReferenceParser :: OrgParser Text
footnoteReferenceParser = do
_ <- string "[fn:"
name <- MP.takeWhile1P (Just "footnote name character") (/= ']')
_ <- string "]"
pure nameWhen parsing the link to a footnote, we parse it with the LinkTargetFootnote constructor for data:LinkTarget to distinguish it from LinkTargetOid which are generic links to any other named cell. Then at weaving time, we know how to style it (with a numeric superscript).
footnoteLinkParser :: OrgParser Link
footnoteLinkParser = do
name <- footnoteReferenceParser
pState <- get
displayName <- case lookup name pState.footnoteRefs of
Nothing -> do
let newFootnoteNumber = pState.footnoteNumber + 1
newFootnoteRefs =
insert
name
(newFootnoteNumber, pState.cellNumber)
pState.footnoteRefs
modify
( \s ->
s
{ footnoteNumber = newFootnoteNumber,
footnoteRefs = newFootnoteRefs
}
)
pure $ T.show newFootnoteNumber
Just (n, _) -> do
pure $ T.show n
pure
$ Link
{ target = LinkTargetFootnote name,
name = [StyledText {body = displayName, style = styleMaskFrom []}]
}As for the footnote definitions, they are expected to be placed on their own, as separate paragraphs. The definition may only be made up of 1 paragraph, because good footnotes are short and to the point.
type Footnote :: Type
data Footnote = Footnote
{ name :: Text,
body :: Prose
}
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)Parsing footnote definitions is a little different than other cells, because the order in which the footnotes are defined may not necessarily be the order in which the footnotes are referenced from the rest of the document.
For simplicity, Lilac doesn't care about the order in which footnote definitions appear. However, it does expect two conditions:
Footnote definitions must appear together, just once, in the entire document. That is, you may not define footnotes in multiple places 90. For example, see ti:0013-footnotes-multiple-groups.
Inline footnotes (where footnotes are defined in the same area where they are referenced) are not allowed.
During weaving, the footnote definitions are sorted according to the order in which they are referenced (via footnote links). See f:weaveObject for details.
The footnotes are not turned into cells; instead they are used to build up the footnoteDefinitions map 34 in data:OrgParserState, which are then dumped into data:OrgDoc. This is because there is no point in preserving information about the order in which these footnote definitions are defined; in other words, it's useless to store footnote definitions as individual cells in the order they're defined, if during weave time we are going to sort them anyway in a (possibly) different order.
footnoteDefinitionParser :: OrgParser Cell
footnoteDefinitionParser = do
name <- footnoteReferenceParser
_ <- onlySpaces
modify (\s -> s {insideFootnoteDefinition = Just name})
(CParagraph prose) <- paragraphParser
pState <- get
o <- MP.getOffset
when
( (not pState.prevCellIsFootnote)
&& (pState.footnoteDefinitions /= M.empty)
)
$ do
MP.setOffset o
fail "cannot have more than one group of footnote definitions" -- 90
modify
( \s ->
s
{ footnoteDefinitions =
insert name prose pState.footnoteDefinitions,
insideFootnoteDefinition = Nothing
}
)
pure $ CFootnote Footnote {name = name, body = prose}When we have finished parsing the document, it's expected that all footnote links and definitions line up with each other (that there is an exact 1:1 match). If there are either links that don't have an associated reference or a definition without an associated link, it's considered an error.
reconcileFootnotes :: OrgParser (Map Text (Int, LCI))
reconcileFootnotes = do
pState <- get
let referenced = M.keysSet pState.footnoteRefs
defined = M.keysSet pState.footnoteDefinitions
brokenLinks = Set.difference referenced defined
unusedDefs = Set.difference defined referenced
hasBrokenLinks = length brokenLinks /= 0
hasUnusedDefs = length unusedDefs /= 0
brokenLinksMsg = T.unpack . unwords $ toList brokenLinks
unusedDefsMsg = T.unpack . unwords $ toList unusedDefs
if
| hasBrokenLinks -> fail $ "broken footnote links: " <> brokenLinksMsg
| hasUnusedDefs -> fail $ "unused footnote definitions: " <> unusedDefsMsg
| otherwise -> pure pState.footnoteRefs5.9.1 Tests
Check for broken links.
Hello [fn:hello] world.shouldFailWithMessage
"0013-footnotes-broken-link"
"broken footnote links: hello"
24Check for unused definitions.
Hello world.
[fn:hello] Foo.shouldFailWithMessage
"0013-footnotes-unused-def"
"unused footnote definitions: hello"
30Basic happy path.
Hello [fn:hello] world.
[fn:hello] Foo.[ CParagraph
$ Prose
[ ptext "Hello" [],
PTokenLink
Link
{ target = LinkTargetFootnote "hello",
name = [stext "1" []]
},
ptext " world." []
],
CFootnote
Footnote
{ name = "hello",
body = Prose [ptext "Foo." []]
}
]shouldParseWithState
"0013-footnotes-simple"
OrgDoc
{ oid = nilOid,
cells =
Expected cells for 0013-footnotes-simple,
pseudoEdges = [],
footnoteRefs =
fromList
[ ("hello", (1, LCI 0))
],
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta = fromList [("ID", "00000000-0000-0000-0000-000000000000")]
}
defaultOrgParserState
{ footnoteNumber = 1,
footnoteRefs =
fromList
[ ("hello", (1, LCI 0))
],
footnoteDefinitions =
fromList
[ ( "hello",
Prose [ptext "Foo." []]
)
],
prevCellIsFootnote = True,
glossary = M.empty,
lineLabels = M.empty
}Complex example.
Hello [fn:a] world. [fn:b]
[fn:b] Definition of b.
[fn:a] Definition of a.[ CParagraph
$ Prose
[ ptext "Hello" [],
PTokenLink
Link
{ target = LinkTargetFootnote "a",
name = [stext "1" []]
},
ptext " world." [],
PTokenLink
Link
{ target = LinkTargetFootnote "b",
name = [stext "2" []]
}
],
CFootnote
Footnote
{ name = "b",
body = Prose [ptext "Definition of b." []]
},
CFootnote
Footnote
{ name = "a",
body = Prose [ptext "Definition of a." []]
}
]shouldParseWithState
"0013-footnotes-complex"
OrgDoc
{ oid = nilOid,
cells =
Expected cells for 0013-footnotes-complex,
pseudoEdges = [],
footnoteRefs =
fromList
[ ("a", (1, LCI 0)),
("b", (2, LCI 0))
],
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta = fromList [("ID", "00000000-0000-0000-0000-000000000000")]
}
defaultOrgParserState
{ footnoteNumber = 2,
footnoteRefs =
fromList
[ ("a", (1, LCI 0)),
("b", (2, LCI 0))
],
footnoteDefinitions =
fromList
[ ("a", Prose [ptext "Definition of a." []]),
("b", Prose [ptext "Definition of b." []])
],
prevCellIsFootnote = True,
glossary = M.empty,
lineLabels = M.empty
}ti:0013-footnotes-multiple-groups is like ti:0013-footnotes-complex, but we have a non-footnote cell that separates the two footnote definitions. This is not allowed.
Hello [fn:a] world. [fn:b]
[fn:b] Definition of b.
Some other text.
[fn:a] Definition of a.shouldFailWithMessage
"0013-footnotes-multiple-groups"
"cannot have more than one group of footnote definitions"
956 Citations
Citations are special kinds of links, but due to their complexity, we discuss them separately here in their own dedicated section.
The CSL specification says "citations consist of one or more cites to individual items". A cite is a single (unique) pointer to a bibliographical work. The description of the bibliographical work itself is defined separately from a cite, and the collection of all bibliographical works cited in a document is called the bibliography.
Citations are similar to footnotes, but there are some major differences:
A special Citation Style Language (CSL) file is needed to be able to weave both the bibliography and citations. These CSL files are just XML files. See this repository for an example.
The equivalent of a "footnote definition" is the global description of the bibliographical work, which can be a book, journal, article, web page, etc. In a sense, these descriptions are a subset of the universe of possible footnote definitions.
These descriptions are expected to be stored in CSL JSON format, separate from the Org file itself.
For weaving the bibliography, we have to say #+print_bibliography: in the place where we want to have them woven. From the reader's point of view, this is the equivalent of footnote definitions for footnotes.
A single cite may also use a locator, which refers to a more specific section in the work (such as via page numbers). It can also be flanked by affixes (a prefix or suffix), which just means text that can come before or after the cite to describe it more in natural language (such as styled text or even inline math fragments).
So whereas a footnote link may only need a numerical superscript for presentation when woven, citations need much more work. This translates to additional work required during the parsing stage.
The following sections take a closer look at each of these areas.
6.1 CSL style file (XML)
The CSL file is an XML file specifying various styling properties for rendering both the citations themselves and also the bibliography. This file can be specified as part of the top-level document metadata in f:docMetaParser with the #+cite_export key, whose value is checked by f:validateCslPath.
validateCslPath :: (MonadFail m) => Text -> m (Maybe FilePath)
validateCslPath maybeCslPath
| T.isPrefixOf cslPrefix maybeCslPath =
pure $ T.unpack <$> T.stripPrefix cslPrefix maybeCslPath
| otherwise =
fail
$ show cslPrefix
<> " prefix required for specifying CSL file"
where
cslPrefix :: Text
cslPrefix = "csl "The value must be prefixed with csl . This is just Org convention that we've carried over into Lilac.
#+cite_export: csl foo.cslshouldParseWithState
"0014-citations-csl-path"
OrgDoc
{ oid = nilOid,
cells = [],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Just "foo.csl",
bibPaths = [],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState6.2 CSL JSON (bibliographical entries)
The CSL JSON defines a list of bibliographical works. An Org document can refer to one of these, and may cite one or more of the works therein.
A document may refer to multiple CSL JSON files. All are consulted during weaving in order to generate bibliographies. The tricky part here is that you can define them more than once. However, they all use the same #+bibliography: key. So we cannot simply combine them into a Map because the similar keys will override each other.
Instead, we can collect all the values that share the same bibliography key with f:collectValsFor. See how this is used in f:docMetaParser.
collectValsFor :: Text -> [(Text, Text)] -> ([Text], [(Text, Text)])
collectValsFor key kvs =
let otherKVs = filter (\(k, _) -> k /= key) kvs
vals = reverse $ foldl' f [] kvs
in (vals, otherKVs)
where
f :: [Text] -> (Text, Text) -> [Text]
f acc (k, v)
| k == key = v : acc
| otherwise = acc#+bibliography: a.json
#+bibliography: b.jsonshouldParseWithState
"0014-citations-csl-json-paths"
OrgDoc
{ oid = nilOid,
cells = [],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths =
[ "a.json",
"b.json"
],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState6.2.1 Default CSL JSON
You can also define a global bibliography file in the ProjectConf (data:ProjectConf → 7). Doing so will essentially create an automatic #+bibliography: ... line in all Org documents in the archive.
If you choose to do this, then in Emacs you should set the org-cite-global-bibliography variable to the same equivalent path, so that the interactive org-cite-insert command will know where to read bibliographies.
6.3 Printing bibliographies
The document must specify where to print the list of bibliographical references that have been cited (otherwise, if we were to omit this, the citations become somewhat meaningless). This is done by having a line that just says #+print_bibliography: (note the trailing colon :).
Orgmode supports passing in additional options here to the CSL citation export processor, but for simplicity Lilac doesn't support any additional options here.
bibliographyLocationParser :: OrgParser Cell
bibliographyLocationParser = do
_ <- string "#+print_bibliography:"
pure $ CCommand PrintBibliography#+cite_export: csl foo.csl
#+bibliography: a.json
#+print_bibliography:shouldParseWithState
"0014-citations-bibliography-location"
OrgDoc
{ oid = nilOid,
cells = [CCommand PrintBibliography],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Just "foo.csl",
bibPaths = ["a.json"],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState6.4 Citation data structures
With the boilerplate in the preceding sections out of the way, we can focus on encoding all of the various parts of citations.
A data:Citation is a collection of one or more cites. The idea is that you can have multiple cites in a single span of text.
type Citation :: Type
data Citation = Citation
{ styleVariation :: (Maybe Text, Maybe Text),
cites :: [Cite],
noteNumber :: Maybe Int,
prefix :: Prose,
suffix :: Prose
}
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)If you just want to see what citations look like in practice, skip to the Tests below.
The data:Citation also specifies the style to be used for rendering the key in data:Cite, via the styleVariation. This is just an additional way of tweaking the style to use on top of the overall CSL style in CSL file.
Each data:Cite also has its own prefix and suffix for describing things using regular prose. Contrast this with the prefix and suffix at the data:Citation level, which are for the entire citation.
The most important field for data:Cite is the id 91, which refers to a unique bibliographical work. It's expected that this ID is found in one of the CSL JSON files. Orgmode calls this the key, but we prefer id because there is no immediate hashmap data structure here where we use it as a key.
The f:citationParser parses citations in the text.
citationParser :: OrgParser Int
citationParser = do
failIfVerbatimOrCode
pState <- get
_ <- string "[cite"
modify (\s -> s {insideCitation = True})
sv1 <- MP.optional $ char '/' >> MP.label "style variation 1" svParser
sv2 <- MP.optional $ char '/' >> MP.label "style variation 2" svParser
_ <- char ':'
_ <- MP.optional intraCellWhitespaceParser
citationPrefix <-
MP.optional $ do
_ <- MP.notFollowedBy pseudoCiteParser
p <- citeAffixParser True
_ <- MP.optional intraCellWhitespaceParser
_ <- char ';'
pure p
cites <-
MP.sepEndBy1
citeParser
(char ';' >> MP.optional intraCellWhitespaceParser)
citationSuffix <-
MP.optional $ do
_ <- MP.notFollowedBy pseudoCiteParser
p <- citeAffixParser True
_ <- MP.optional intraCellWhitespaceParser
pure p
_ <- MP.optional intraCellWhitespaceParser
_ <- char ']'
let maybeNoteNumber = do
footnoteName <- pState.insideFootnoteDefinition
fmap fst $ lookup footnoteName pState.footnoteRefs
c =
Citation
{ styleVariation = (T.pack <$> sv1, T.pack <$> sv2),
noteNumber = maybeNoteNumber,
cites = cites,
prefix = fromMaybe (Prose []) citationPrefix,
suffix = fromMaybe (Prose []) citationSuffix
}
modify
( \s ->
s
{ insideCitation = False,
citations = c : s.citations -- 92
}
)
pure $ length pState.citations -- 93
where
svParser :: OrgParser [Char]
svParser =
MP.some
$ MP.satisfy (\c -> isAlphaNumChar c || c == '_' || c == '-')
pseudoCiteParser = do
prefix <-
MP.optional $ do
p <- citeAffixParser True
_ <- intraCellWhitespaceParser
pure p
citeId <- citeIdParser
pure (prefix, citeId)The f:intraCellWhitespaceParser parses spaces and newlines, but does not cross cell boundaries (no double newlines). In other words, a citation may not have a double newline in it.
intraCellWhitespaceParser :: OrgParser [Char]
intraCellWhitespaceParser =
MP.label "intra-cell whitespace"
$ MP.some
$ MP.choice
[ MP.satisfy (\c -> c == ' '),
do
MP.notFollowedBy cellBoundary
char '\n'
]Anyway, the big picture is that every citation looks like [cite...], and that the cites within the citation are parsed separately with f:citeParser. Also, note that we return just a single integer 93. This is simply the index of the list of citations we build up in 92. This list is finalized in f:collectCitations, which is used by f:orgDocParser.
collectCitations :: OrgParser [Citation]
collectCitations = do
pState <- get
pure $ reverse pState.citationsWe have to build up a single list of all citations because when we call Citeproc.citeproc, we have to provide all citations used in the document; this is the only way to get numbering correct with a style like IEEE. For example, if we were to call Citeproc.citeproc once for every citation on its own, those citations would all get numbered as [1] because Citeproc will only see a single citation at a time.
6.4.1 Parsing a single cite
The f:citeParser parses a single data:Cite. A cite is composed of four pieces:
(Optional) The prefix, which is some explanatory prose just before the id.
The id (Orgmode calls this the key), which refers to a unique bibliographical entry.
(Optional) The locator, which identifies some subsection of the cited bibliographical entry, such as page or chapter numbers.
(Optional) The suffix, which is some explanatory prose that follows the cite id and optional locator.
citeParser :: OrgParser Cite
citeParser = do
_ <-
MP.try
. MP.lookAhead -- 94
$ MP.optional citePrefixParser
>> citeIdParser
citePrefix <- MP.optional citePrefixParser
citeId <- citeIdParser
citeLocator <-
MP.optional . MP.try $ do
_ <- intraCellWhitespaceParser
citeLocatorParser -- 95
citeSuffix <-
MP.optional $ do
_ <- intraCellWhitespaceParser
citeAffixParser False
pure
Cite
{ id = citeId,
locator = citeLocator,
prefix = fromMaybe (Prose []) citePrefix,
suffix = fromMaybe (Prose []) citeSuffix
}
where
citePrefixParser = do
p <- citeAffixParser True
_ <- intraCellWhitespaceParser
pure pWe have to use MP.lookAhead 94 because otherwise there is no way to tell apart a cite prefix (which precedes a cite id like @foo) from a citation prefix.
The Orgmode syntax description says this about the allowed characters for a cite key (what we call an id):
A string made of any word-constituent character, -, ., :, ?, !, `, ', /, *, @,
+, |, (, ), {, }, <, >, &, _, ^, $, #, %, or ~.And so that's why we allow such a broad range of characters in f:citeIdParser.
citeIdParser :: OrgParser Text
citeIdParser = do
_ <- char '@'
citeId <-
MP.takeWhile1P
(Just "citeId")
(\c -> isAlphaNumChar c || c `T.elem` "-.:?!`'/*@+|(){}<>&_^$#%~")
pure citeIdThe f:citeLocatorParser 95 parses the locator, which is made up of two parts. The first part is the term, which describes the location inside the work, such as "chapter" or "page". See f:citeLocatorTermParser for all supported terms. If omitted, this defaults to "page".
The second part is a CSV of coordinates, which can be a number (e.g., 35), a section (e.g., 1.2.3), or a range of numbers (e.g., 35-50) or sections (e.g., 1.2-4). We don't bother with roman numerals. See t:0014-citations-locators for more examples.
citeLocatorParser :: OrgParser (Text, Text)
citeLocatorParser =
MP.choice
[ explicitLocator,
implicitLocator
]
where
explicitLocator :: OrgParser (Text, Text)
explicitLocator = do
term <- citeLocatorTermParser
_ <- intraCellWhitespaceParser
coords <- citeLocatorCoordsParser
pure (term, coords)
implicitLocator :: OrgParser (Text, Text)
implicitLocator = do
coords <- citeLocatorCoordsParser
pure ("page", coords)The terms in f:citeLocatorTermParser are mostly lifted from org-cite-csl--label-alist from Orgmode. Unlike Orgmode, we do not allow special characters like ¶ or §.
citeLocatorTermParser :: OrgParser Text
citeLocatorTermParser =
MP.choice $ map (MP.try . locatorTermParser) locatorTerms
where
locatorTermParser :: (Text, Text) -> OrgParser Text
locatorTermParser (humanFriendlyTerm, term) = do
_ <- string humanFriendlyTerm
pure term
locatorTerms :: [(Text, Text)]
locatorTerms =
[ ("books", "book"),
("book", "book"),
("bks.", "book"),
("bk.", "book"),
("chapters", "chapter"),
("chapter", "chapter"),
("chaps.", "chapter"),
("chap.", "chapter"),
("ch.", "chapter"),
("columns", "column"),
("column", "column"),
("cols.", "column"),
("col.", "column"),
("figures", "figure"),
("figure", "figure"),
("figs.", "figure"),
("fig.", "figure"),
("folios", "folio"),
("folio", "folio"),
("fols.", "folio"),
("fol.", "folio"),
("numbers", "number"),
("number", "number"),
("nos.", "number"),
("no.", "number"),
("lines", "line"),
("line", "line"),
("ll.", "line"),
("l.", "line"),
("notes", "note"),
("note", "note"),
("nn.", "note"),
("n.", "note"),
("opus", "opus"),
("opp.", "opus"),
("op.", "opus"),
("pages", "page"),
("page", "page"),
("pp.", "page"),
("p.", "page"),
("paragraphs", "paragraph"),
("paragraph", "paragraph"),
("paras.", "paragraph"),
("para.", "paragraph"),
("parts", "part"),
("part", "part"),
("pts.", "part"),
("pt.", "part"),
("sections", "section"),
("section", "section"),
("secs.", "section"),
("sec.", "section"),
("sub verbo", "sub verbo"),
("s.vv.", "sub verbo"),
("s.v.", "sub verbo"),
("verses", "verse"),
("verse", "verse"),
("vv.", "verse"),
("v.", "verse"),
("volumes", "volume"),
("volume", "volume"),
("vols.", "volume"),
("vol.", "volume")
]The f:citeLocatorCoordsParser parses the coordinates, which is a comma-separated value.
citeLocatorCoordsParser :: OrgParser Text
citeLocatorCoordsParser = do
coords <-
MP.sepBy1
( MP.choice
[ MP.try citeLocatorCoordRangeParser,
citeLocatorCoordParser
]
)
(char ',' >> intraCellWhitespaceParser)
pure $ T.intercalate ", " coordsThe f:citeLocatorCoordParser can parse a single number or a section (numbers separated by periods). The use of MP.sepBy1 guarantees that we don't get multiple periods in a row.
citeLocatorCoordParser :: OrgParser Text
citeLocatorCoordParser = do
numCoord <- MP.sepBy1 numParser $ char '.'
pure $ T.intercalate "." numCoord
where
numParser :: OrgParser Text
numParser = MP.takeWhile1P (Just "locatorCoordNum") isDigitCharThe f:citeLocatorCoordRangeParser just parses a range of numbers or sections.
citeLocatorCoordRangeParser :: OrgParser Text
citeLocatorCoordRangeParser = do
n1 <- citeLocatorCoordParser
_ <- char '-'
n2 <- citeLocatorCoordParser
pure $ T.concat [n1, "-", n2]The f:citeAffixParser can parse anything, except for semicolons and the @ symbol if parsing the prefix (because otherwise this will end up swallowing the cite id such as @foo).
citeAffixParser :: Bool -> OrgParser Prose
citeAffixParser isPrefix = do
modify (\s -> s {insideCitePrefix = isPrefix})
pState <- get
prose <-
proseParser
. whitespaceCompressor
$ proseSpaceConsumer pState.continuationIndent
modify (\s -> s {insideCitePrefix = False})
failIfLeftoverActiveStyles
failIfUnbalancedBrackets -- 96
pure $ compressProse proseThe f:failIfUnbalancedBrackets helper 96 fails if it detects if square brackets have not been balanced by the end of the citation.
failIfUnbalancedBrackets :: OrgParser ()
failIfUnbalancedBrackets = do
pState <- get
when (pState.bracketBalance /= 0) $ do
fail "unbalanced brackets"6.4.2 Tests
Simple citation, without any locators or affixes.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite/v1/v2: @foo123-.:?!`'/*@+|(){}<>&_^$#%~]shouldParseWithState
"0014-citations-single-cite"
OrgDoc
{ oid = nilOid,
cells =
Expected prose for 0014-citations-single-cite,
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Just "foo.csl",
bibPaths = ["a.json"],
citations =
[ Citation
{ styleVariation = (Just "v1", Just "v2"),
noteNumber = Nothing,
cites =
[ Cite
{ id = "foo123-.:?!`'/*@+|(){}<>&_^$#%~",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
}
],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState[ CParagraph
$ Prose
[ PTokenCitation 0
]
]Cite with various (typical) locators.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite: @a 15]
[cite: @b 15-30]
[cite: @c chapters 15-30, 35]
[cite: @d secs. 1.5-10]
[cite: @e sec. 1.22.3]
[cite:@f]shouldParseWithState
"0014-citations-locators"
OrgDoc
{ oid = nilOid,
cells =
Expected prose for 0014-citations-locators,
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Just "foo.csl",
bibPaths = ["a.json"],
citations =
[ Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "a",
locator = Just ("page", "15"),
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "b",
locator = Just ("page", "15-30"),
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "c",
locator = Just ("chapter", "15-30, 35"),
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "d",
locator = Just ("section", "1.5-10"),
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "e",
locator = Just ("section", "1.22.3"),
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "f",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
}
],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState[ CParagraph
$ Prose
[ PTokenCitation 0,
ptext " " [],
PTokenCitation 1,
ptext " " [],
PTokenCitation 2,
ptext " " [],
PTokenCitation 3,
ptext " " [],
PTokenCitation 4,
ptext " " [],
PTokenCitation 5
]
]Citations with locators and affixes.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite:prefix @a1 15 suffix]
[cite: @b1 15-30; a really long prefix, maybe? @b2 chap. 4 more suffix]
[cite: @c1 chapters 15-30, 35 suffix with balanced [[brackets] and @sign]]
[cite: @d1 secs. 1.5-10; prefix @d2;@d3]
[cite:@ /see also/ @e1 sec. 1.22.3]
[cite:@f1;*foo* =@fake1= ~@fake2~ @f2 vols. 3-4]shouldParseWithState
"0014-citations-locators-affixes"
OrgDoc
{ oid = nilOid,
cells =
Expected prose for 0014-citations-locators-affixes,
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Just "foo.csl",
bibPaths = ["a.json"],
citations =
[ Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "a1",
locator = Just ("page", "15"),
prefix = Prose [ptext "prefix" []],
suffix = Prose [ptext "suffix" []]
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "b1",
locator = Just ("page", "15-30"),
prefix = Prose [],
suffix = Prose []
},
Cite
{ id = "b2",
locator = Just ("chapter", "4"),
prefix = Prose [ptext "a really long prefix, maybe?" []],
suffix = Prose [ptext "more suffix" []]
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "c1",
locator = Just ("chapter", "15-30, 35"),
prefix = Prose [],
suffix =
Prose
[ ptext "suffix with balanced [[brackets] and @sign]" []
]
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "d1",
locator = Just ("section", "1.5-10"),
prefix = Prose [],
suffix = Prose []
},
Cite
{ id = "d2",
locator = Nothing,
prefix = Prose [ptext "prefix" []],
suffix = Prose []
},
Cite
{ id = "d3",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "e1",
locator = Just ("section", "1.22.3"),
prefix =
Prose
[ ptext "@ " [],
ptext "see also" [Italic]
],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
},
Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "f1",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
},
Cite
{ id = "f2",
locator = Just ("volume", "3-4"),
prefix =
Prose
[ ptext "foo" [Bold],
ptext " " [],
ptext "@fake1" [Verbatim],
ptext " " [],
ptext "@fake2" [Code]
],
suffix = Prose []
}
],
prefix = Prose [],
suffix = Prose []
}
],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState[ CParagraph
$ Prose
[ PTokenCitation 0,
ptext " " [],
PTokenCitation 1,
ptext " " [],
PTokenCitation 2,
ptext " " [],
PTokenCitation 3,
ptext " " [],
PTokenCitation 4,
ptext " " [],
PTokenCitation 5
]
]Check for citation-level (aka common or global) affixes, with some examples taken from the Org community. These citation-level affixes are useful because it may be the case that the CSL processor decides to reorder the cites.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite:*global* prefix;@a1;global =suffix=]
[cite/t: see;@source1;@source2;by Smith /et al./]
[cite/t:see;@foo p. 7;@bar pp. 4;by foo]
[cite/a/f:c.f.;the very important @@atkey @ once;the crucial @baz vol. 3]shouldParseWithState
"0014-citations-worg"
OrgDoc
{ oid = nilOid,
cells =
Expected prose for 0014-citations-worg,
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Just "foo.csl",
bibPaths = ["a.json"],
citations =
[ Citation
{ styleVariation = (Nothing, Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "a1",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
}
],
prefix =
Prose
[ ptext "global" [Bold],
ptext " prefix" []
],
suffix =
Prose
[ ptext "global " [],
ptext "suffix" [Verbatim]
]
},
Citation
{ styleVariation = (Just "t", Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "source1",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
},
Cite
{ id = "source2",
locator = Nothing,
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [ptext "see" []],
suffix =
Prose
[ ptext "by Smith " [],
ptext "et al." [Italic]
]
},
Citation
{ styleVariation = (Just "t", Nothing),
noteNumber = Nothing,
cites =
[ Cite
{ id = "foo",
locator = Just ("page", "7"),
prefix = Prose [],
suffix = Prose []
},
Cite
{ id = "bar",
locator = Just ("page", "4"),
prefix = Prose [],
suffix = Prose []
}
],
prefix = Prose [ptext "see" []],
suffix = Prose [ptext "by foo" []]
},
Citation
{ styleVariation = (Just "a", Just "f"),
noteNumber = Nothing,
cites =
[ Cite
{ id = "@atkey",
locator = Nothing,
prefix = Prose [ptext "the very important" []],
suffix = Prose [ptext "@ once" []]
},
Cite
{ id = "baz",
locator = Just ("volume", "3"),
prefix = Prose [ptext "the crucial" []],
suffix = Prose []
}
],
prefix = Prose [ptext "c.f." []],
suffix = Prose []
}
],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState[ CParagraph
$ Prose
[ PTokenCitation 0,
ptext " " [],
PTokenCitation 1,
ptext " " [],
PTokenCitation 2,
ptext " " [],
PTokenCitation 3
]
]Unbalanced brackets are rejected from the cite prefix.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite: [ @foo]shouldFailWithMessage
"0014-citations-unbalanced-brackets-prefix-opening"
"unbalanced brackets"
58#+cite_export: csl foo.csl
#+bibliography: a.json
[cite: ] @foo]shouldFailWithError
"0014-citations-unbalanced-brackets-prefix-closing"
( utoks "]"
<> etoks "[["
<> etoks "[fn:"
<> etoks "\\("
<> etoks "@"
<> etoks "\\"
<> elabel "(raw link) protocol"
<> etoks "\n"
<> elabel "normal characters"
)
57Similarly, unbalanced brackets should fail for the cite suffix.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite: @foo []shouldFail "0014-citations-unbalanced-brackets-suffix-opening"#+cite_export: csl foo.csl
#+bibliography: a.json
[cite: @foo ]]shouldFailWithError
"0014-citations-unbalanced-brackets-suffix-closing"
( utoks "]]\n"
<> etoks "[["
<> etoks "[fn:"
<> etoks "\\("
<> etoks "\\"
<> elabel "(raw link) protocol"
<> etoks "\n"
<> elabel "normal characters"
)
62Semicolons must be meaningful; extraneous semicolons are not allowed.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite:;@foo]shouldFailWithError
"0014-citations-extra-semicolons-basic"
( utoks ";"
<> etoks "@"
<> elabel "intra-cell whitespace"
)
56#+cite_export: csl foo.csl
#+bibliography: a.json
[orgmode-citation-breakage-workaround@foo;;]Aside: sadly, using [cite:... directly above seems to break Orgmode's parsing, such that opening this file in Emacs results in an error. So "hide" it from Orgmode with orgmode-citation-breakage-workaround.
cite:shouldFailWithError
"0014-citations-extra-semicolons-consecutive"
( utoks ";"
<> etoks "@"
<> etoks "]"
<> elabel "intra-cell whitespace"
)
61Citations cannot be nested. Because we don't parse for nested citations in f:proseParser, in t:0014-citations-nested we end up parsing the magic string [cite: as prose text. So then this triggers the balanced brackets checker.
#+cite_export: csl foo.csl
#+bibliography: a.json
[cite:@foo;[cite: @bar]]shouldFailWithMessage
"0014-citations-nested"
"unbalanced brackets"
67Citations inside verbatim or code style markers are ignored.
~[cite:@foo]~
=[cite:@bar]=shouldParseWithState
"0014-citations-forbidden-in-verbatim-code"
OrgDoc
{ oid = nilOid,
cells =
Expected prose for 0014-citations-forbidden-in-verbatim-code,
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState[ CParagraph
$ Prose
[ ptext "[cite:@foo]" [Code],
ptext " " [],
ptext "[cite:@bar]" [Verbatim]
]
]For simplicity, citations are not allowed in headings. This is controlled by the insideHeading boolean in f:headingParser, and respected by f:proseParser.
* heading [cite:@foo]shouldParseWithState
"0014-citations-forbidden-in-headings"
OrgDoc
{ oid = nilOid,
cells =
[ CHeading
Heading
{ level = 1,
section = [1],
body = Prose [ptext "heading [cite:@foo]" []],
meta = fromList []
}
],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState
{ headingOidStack = [Nothing],
headingLevel = 1,
headingSectionNumber = [1]
}7 Prose
Prose is a series of contiguous prose text, which can have some styling (bold, italic, etc) as well as links, citations, footnote links, etc, or tokens. So it's just a list of these prose tokens (data:PToken).
type Prose :: Type
newtype Prose = Prose
{ unProse :: [PToken]
}
deriving stock (Eq, Generic, Ord, Show)
deriving anyclass (FromJSON, ToJSON)A PToken can be a styled string, link, citation, or math fragment. For the citation, we only store the index of it (starting from 0), to act as a "pointer" to the list of all citations in the entire document, collected in f:collectCitations.
type PToken :: Type
data PToken
= PTokenStyledText StyledText
| PTokenLink Link
| PTokenCitation Int
| PTokenMathFragment MathFragment
| PTokenDivStart Text
| PTokenDivEnd
deriving stock (Eq, Generic, Ord, Show)
deriving anyclass (FromJSON, ToJSON)The PTokenDivStart and PTokenDivEnd markers are similar to the data:ListMarker, in that they encode start and end points (in HTML). Specifically, they provide equivalent functionality of the CslDiv constructor for the CslJson type in Citeproc. This is needed in order to render citations (e.g., see Expected bibliography for 0400-citeproc-basic). Arguably the CslDiv is more elegant because it encodes a recursive list structure (by way of CslConcat and CslEmpty), and also because it only needs a single constructor CslDiv whereas we require two (PTokenDivStart and PTokenDivEnd). However this is the price we have to pay for not being recursive. It also makes parsing these things difficult, so we avoid parsing them altogether from raw text (notice how f:proseParser does not use them).
The f:proseParser can parse a series of data:PToken values. It is used by both f:headingParser and f:paragraphParser, because they both allow prose text in them. In particular, it avoids parsing Citations inside headings and also if we're already inside another citation (no nesting allowed).
proseParser :: OrgParser Text -> OrgParser Prose
proseParser spaceConsumer = do
pState <- get
let avoidCitations =
pState.insideHeading || pState.insideCitation
a <- MP.choice $ baseParsers avoidCitations -- 97
b <-
MP.many
$ MP.choice (spacePTokenParser : baseParsers avoidCitations) -- 98
pure . Prose . filter nonEmpty $ a : b
where
spacePTokenParser = do
s <- spaceConsumer
st <- get
pure
$ PTokenStyledText
StyledText {body = s, style = st.styleMask}
baseParsers avoidCitations
| avoidCitations =
[ PTokenLink <$> linkParser,
PTokenMathFragment <$> mathFragmentInlineParser,
PTokenStyledText <$> styleMarkerParser,
PTokenStyledText <$> styledTextParser
]
| otherwise =
[ PTokenCitation <$> citationParser,
PTokenLink <$> linkParser,
PTokenMathFragment <$> mathFragmentInlineParser,
PTokenStyledText <$> styleMarkerParser,
PTokenStyledText <$> styledTextParser
]
nonEmpty :: PToken -> Bool
nonEmpty = \case
PTokenStyledText styledText -> not $ T.null styledText.body
PTokenLink {}
PTokenMathFragment {}
PTokenCitation {}
PTokenDivStart {}
PTokenDivEnd {} -> TrueWe first parse among the non-whitespace parsers in baseParsers 97, then parse additional tokens that may include whitespace 98. This way, we force the prose to initially start out without any indentation (because the preceding cell, if any, would have swallowed up trailing whitespace automatically via lexeme in f:cellParser); otherwise, f:detectInvalidIndentation would not get a chance to detect this (invalid) indentation.
It may appear that we could alternatively skip past spaces inside prose text by making a lexeme-like wrapper which skips newlines and spaces for each parser in baseParsers. However, that approach has a problem. Consider the case of parsing something like
=foo=...where the ... will get parsed separately from the foo. We'll end up with two PTokenStyledText objects, one for foo and another for .... This is a problem because we'll also parse
=foo= ...exactly the same way as two PTokenStyledText objects. This is because the space between the foo and ... will get eaten up by the wrapper and discarded. And now we're in the position where we have to, on export, guess (due to information loss) if we need to insert whitespace between the foo and ....
Instead, explicitly parsing the whitespace with f:whitespaceCompressor lets us parse the two cases above differently, because we save all whitespace we parsed as their own PTokenStyledText objects. We call it a "compressor" because no matter how many spaces the spaceConsumer 99 consumes, we construct only 0 or 1 space characters in the data:StyledText object.
This parser only succeeds if the given spaceConsumer succeeds. And then it decides to report 0 or 1 space characters depending on what follows (MP.lookAhead). Note that we also do a second level of "compression" in f:compressProse, where we will drop StyledText objects with an empty string as the body.
Whitespace compression also happens when we detect footnote references 100. Specifically, if there is a prose token, some whitespace, and then a footnote reference, we ignore the whitespace. For example, if we have
foo [[fn:bar]]then the space between foo and [[fn:bar]] is ignored. This way, when we weave this we can get the footnote number superscript "attached" to the foo instead of having it float around by itself which will look odd.
7.1 Compatibility with CiteprocOutput
Citeproc will read the CSL style file (XML) to understand how to make formatting transformations (adding emphasis, changing cases, etc) to citations based on the specific style. The CiteprocOutput typeclass instance describes all of these transformations:
-- | CSL styles require certain formatting transformations to
-- be defined. These are defined in the 'CiteprocOutput' class.
-- The library may be used with any structured format that defines
-- these operations. See the 'Citeproc.CslJson' module for an instance
-- that corresponds to the markup allowed in CSL JSON. See
-- the 'Citeproc.Pandoc' module for an instance for Pandoc 'Inlines'.
class (Semigroup a, Monoid a, Show a, Eq a, Ord a) => CiteprocOutput a where
toText :: a -> Text
fromText :: Text -> a
dropTextWhile :: (Char -> Bool) -> a -> a
dropTextWhileEnd :: (Char -> Bool) -> a -> a
addFontVariant :: FontVariant -> a -> a
addFontStyle :: FontStyle -> a -> a
addFontWeight :: FontWeight -> a -> a
addTextDecoration :: TextDecoration -> a -> a
addVerticalAlign :: VerticalAlign -> a -> a
addTextCase :: Maybe Lang -> TextCase -> a -> a
addDisplay :: DisplayStyle -> a -> a
addQuotes :: a -> a
movePunctuationInsideQuotes :: a -> a
inNote :: a -> a
mapText :: (Text -> Text) -> a -> a
addHyperlink :: Text -> a -> a
localizeQuotes :: Locale -> a -> aAny type that has an instance of the CiteprocOutput typeclass will be able to be transformed by Citeproc according to the CSL style file. For us, this type is newtype:Prose, and if we add support for the CiteprocOutput typeclass for it, we'll be able to format both the data:Citation values as well as the CSL JSON (bibliographical entries) with newtype:Prose (so that users could use the familiar syntax for newtype:Prose in the CSL JSON directly, for example).
Unfortunately, adding in full support for all of the above functions in CiteprocOutput typeclass is quite tedious and actually not trivial. For example you have to know how to move around punctuation (periods and commas) from outside of quotes (".) to inside (."). The CslJson type is recursive and encodes quotes as a separate constructor, making these transformations easier.
Because of the difficulty, we don't support a full, proper implementation of CiteprocOutput for newtype:Prose; we take shortcuts and avoid doing a proper implementation in some places (e.g., instance:CiteprocOutput Prose:localizeQuotes). So instead of using newtype:Prose, we use the CslJson type (see f:proseToCslJsonText) and convert this back into newtype:Prose (see f:cslJsonTextToProse).
Now you may be thinking, why bother keeping the (unfinished) implementation of CiteprocOutput for newtype:Prose? It turns out that some of the functions are actually quite useful, in particular instance:CiteprocOutput Prose:fromText for implementing f:cslJsonTextToProse. And also, we may decide to avoid this conversion to CslJson in the future; in that scenario, having around this unfinished implementation would provide a solid starting point.
7.1.1 CiteprocOutput typeclass implementation for newtype:Prose
CiteprocOutput requires the type variable to also be an instance of Semigroup and Monoid. Because a newtype:Prose is just a wrapper around a list of data:PToken values, we don't have to actually define anything ourselves here (as the operations have already been defined for the list ([]) type)2.
instance Semigroup Prose where
x <> y = Prose $ x.unProse <> y.unProseinstance Monoid Prose where
mempty = Prose []We define the CiteprocOutput instance for newtype:Prose in instance:CiteprocOutput Prose below.
instance CiteprocOutput Prose where
instance:CiteprocOutput Prose:toText
instance:CiteprocOutput Prose:fromText
dropTextWhile = dropTextFromStyledText True
dropTextWhileEnd = dropTextFromStyledText False
instance:CiteprocOutput Prose:addFontVariant
instance:CiteprocOutput Prose:addFontStyle
instance:CiteprocOutput Prose:addFontWeight
instance:CiteprocOutput Prose:addTextDecoration
instance:CiteprocOutput Prose:addVerticalAlign
instance:CiteprocOutput Prose:addTextCase
instance:CiteprocOutput Prose:addDisplay
instance:CiteprocOutput Prose:addQuotes
instance:CiteprocOutput Prose:movePunctuationInsideQuotes
instance:CiteprocOutput Prose:inNote
instance:CiteprocOutput Prose:mapText
instance:CiteprocOutput Prose:addHyperlink
instance:CiteprocOutput Prose:localizeQuotesThe functions toText and fromText are for converting the value (in this case newtype:Prose) to and from a textual representation. For us this means, converting text to newtype:Prose (parsing) and back to text again. Let's look at the parsing first.
fromText input = case MP.runParser
( runStateT
customProseParser -- 101
defaultOrgParserState
)
""
input of
Left err -> error . T.pack $ MP.errorBundlePretty err
Right (prose, _) -> compressProse prose
where
customProseParser = do
leadingWhitespace <- MP.optional spaceConsumer
rest <- proseParser spaceConsumer
case leadingWhitespace of
Nothing -> pure rest
Just s ->
let w =
PTokenStyledText
StyledText
{ body = s,
style = styleMaskFrom []
}
in pure . Prose $ w : rest.unProse
spaceConsumer :: OrgParser Text
spaceConsumer = T.pack <$> intraCellWhitespaceParserThis definition is mostly a wrapper around f:proseParser. The tricky part is the leading optional whitespace. f:proseParser by default does not parse leading whitespace, because in regular Org documents we expect indentation to have semantic meaning for list items. However for citations, we don't expect to have to deal with list items at all, so we can still parse indentation. The only caveat is that we have to do it manually by creating a custom parser. That's what we do with the customProseParser 101 helper above.
The definition of toText is a bit more involved, though perhaps mechanical. The key is to convert the data:PToken values back into the Orgmode syntax. That way we can guarantee no loss of information going from Text to Prose and back again to Text.
toText prose = foldl' f T.empty prose.unProse
where
f :: Text -> PToken -> Text
f acc = \case
PTokenCitation {}
PTokenDivStart {}
PTokenDivEnd {} -> acc
PTokenStyledText styledText
| T.null styledText.body -> acc
| otherwise -> acc <> unparseStyledText styledText
PTokenLink link -> acc <> unparseLink link
PTokenMathFragment mathFragment -> acc <> unparseMathFragment mathFragmentWe don't support converting data:Citation back into Text, because that would suggest a nested citation (where a citation has another citation inside it), which is strictly forbidden, as seen in t:0014-citations-nested.
We also do not support converting PTokenDivStart and PTokenDivEnd to Text, because we never expect them to be written by humans; they only serve to convert CslJson in Citeproc back into newtype:Prose. And also, even if we supported it here, if we wanted to round trip back from Text into newtype:Prose we would fail because f:proseParser does not support parsing these markers.
Converting a data:StyledText back into a Text is straightforward. We simply surround the bold text with the appropriate style markers.
unparseStyledText :: StyledText -> Text
unparseStyledText styledText =
getStyleMarkers styledText.style
<> escapeStyleMarkers styledText.body
<> getStyleMarkers styledText.style
where
getStyleMarkers styleMask =
foldl' accumulateStyle T.empty . sort $ maskToStyles styleMask
accumulateStyle :: Text -> Style -> Text
accumulateStyle acc = \case
Bold -> acc <> "*"
Italic -> acc <> "/"
Underline -> acc <> "_"
Verbatim -> acc <> "="
Code -> acc <> "~"
Subscript
Superscript
Smallcaps -> acc
escapeStyleMarkers = T.foldl' escape T.empty
escape acc c
| mustEscape && c `elem` ("*/_=~" :: String) = acc <> T.pack ['\\', c]
| otherwise = acc <> T.singleton c
mustEscape =
not (isCode styledText.style || isVerbatim styledText.style)Converting a data:Link to Text is straightforward. We try to avoid duplicating information (e.g., if the display name is the same as the link itself, we avoid spelling it out).
unparseLink :: Link -> Text
unparseLink link = case link.target of
LinkTargetOid linkOid -> unparseLinkOid linkOid
LinkTargetLineLabel linkOid -> unparseLinkOid linkOid
LinkTargetFootnote name -> "[fn:" <> name <> "]"
LinkTargetURI uri ->
let linkTargetText = T.pack $ N.uriToString id uri ""
in if linkTargetText == displayName
then linkTargetText
else "[[" <> linkTargetText <> "][" <> displayName <> "]]"
LinkTargetFilePath filePath ->
let linkTargetText = T.pack filePath
in if linkTargetText == displayName
then "[[file:" <> linkTargetText <> "]]"
else "[[file:" <> linkTargetText <> "][" <> displayName <> "]]"
where
displayName
| hasNoStyle link.name = T.concat $ map (\s -> s.body) link.name
| otherwise = T.concat $ map unparseStyledText link.name
hasNoStyle :: [StyledText] -> Bool
hasNoStyle = null . foldl' f []
f :: [Style] -> StyledText -> [Style]
f acc s = acc <> maskToStyles s.style
unparseLinkOid linkOid =
let linkTargetText = show linkOid
in if linkTargetText == displayName
then "[[" <> linkTargetText <> "]]"
else "[[" <> linkTargetText <> "][" <> displayName <> "]]"For data:MathFragment, we have to disambiguate the three different kinds of declarations of math fragments --- numbered standalone (f:mathFragmentNumberedParser), unnumbered standalone (f:mathFragmentUnnumberedParser), and inline math (f:mathFragmentInlineParser). In terms of conversion to and out of data:PToken values, however, we will only ever use inline math, because the other ones require the math to be in its own cell as a CMathFragment at the data:Cell level, outside of a newtype:Prose. Still, we check for all three types of math fragments for completeness.
unparseMathFragment :: MathFragment -> Text
unparseMathFragment mathFragment = mathStart <> mathFragment.body <> mathEnd
where
(mathStart, mathEnd)
| Just _ <- lookup "__equation_number" mathFragment.meta =
("\\begin{equation}", "\\end{equation}")
| Just _ <- lookup "__equation_anon_number" mathFragment.meta =
("\\[", "\\]")
| otherwise = ("\\(", "\\)")Now we come to dropTextWhile and dropTextWhileEnd functions for the typeclass. These functions are for deleting Text, and correspond to T.dropWhile and T.dropWhileEnd, respectively.
Let's consider an example of how these functions might be implemented. If you look at the definitions used for the CslJson type in Citeproc, you can see that they assume the encoded value is just some Text values (inside the CslText constructor). This makes sense for the CslJson type because it only encodes a Text value, surrounded by any number of other constructors that encode things like emphasis (e.g., CslBold).
Our newtype:Prose is a bit different because it has more semantic information encoded in it beyond just raw Text (see data:PToken). The only really equivalent value is the PTokenStyledText constructor which only deals with Text with some styling on top. The other things like PTokenLink and PTokenMathFragment are not really Text values (we ignore PTokenCitation because citations cannot be nested). So running T.dropWhile won't make much sense if we're looking at a PTokenMathFragment.
Because of the above issue, we compromise by skipping over any non-PTokenStyledText value.
dropTextFromStyledText :: Bool -> (Char -> Bool) -> Prose -> Prose
dropTextFromStyledText dropFromStart f prose =
Prose
. (if dropFromStart then reverse else id)
. fst
. foldl' g ([], False)
$ if dropFromStart then prose.unProse else reverse prose.unProse
where
g :: ([PToken], Bool) -> PToken -> ([PToken], Bool)
g (acc, pastFirst) pToken
| pastFirst = (pToken : acc, pastFirst)
| otherwise = case pToken of
PTokenStyledText styledText
| T.null styledText.body -> (pToken : acc, pastFirst)
| otherwise ->
let modifiedBody = dropper f styledText.body
modified =
PTokenStyledText
StyledText
{ body = modifiedBody,
style = styledText.style
}
in if T.null modifiedBody
then (acc, True)
else (modified : acc, True)
PTokenLink {}
PTokenMathFragment {}
PTokenCitation {}
PTokenDivStart {}
PTokenDivEnd {} -> (pToken : acc, pastFirst)
dropper = if dropFromStart then T.dropWhile else T.dropWhileEndAdding font variations is not supported, because we have no notion of small caps as a stylistic option.
addFontVariant fontVariant prose = case fontVariant of
NormalVariant -> prose
SmallCapsVariant -> proseFor font styles, we treat ObliqueFont the same as ItalicFont. For NormalFont though, we simply strip all styling.
addFontStyle fontStyle prose = case fontStyle of
NormalFont -> Prose $ map noStyle prose.unProse
ItalicFont -> Prose $ map italicize prose.unProse
ObliqueFont -> Prose $ map italicize prose.unProse
where
noStyle pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = styledText.body,
style = styleMaskFrom []
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pToken
italicize pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = styledText.body,
style = styleMaskFrom $ Italic : maskToStyles styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pTokenFor the font weight, we only have one weight, which is Bold.
addFontWeight fontWeight prose = case fontWeight of
NormalWeight -> Prose $ map unBold prose.unProse
BoldWeight -> Prose $ map bold prose.unProse
LightWeight -> Prose $ map unBold prose.unProse
where
bold pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = styledText.body,
style = styleMaskFrom $ Bold : maskToStyles styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pToken
unBold pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = styledText.body,
style =
styleMaskFrom
. filter (/= Bold)
$ maskToStyles styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pTokenThe addTextDecoration definition is straightforward enough, because there is no information loss (it's either having an underline or not having one).
addTextDecoration textDecoration prose = case textDecoration of
NoDecoration -> Prose $ map noUnderline prose.unProse
UnderlineDecoration -> Prose $ map underline prose.unProse
where
underline pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = styledText.body,
style = styleMaskFrom $ Underline : maskToStyles styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pToken
noUnderline pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = styledText.body,
style =
styleMaskFrom
. filter (/= Underline)
$ maskToStyles styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pTokenWe don't support adding vertical alignment, so we don't do anything here.
addVerticalAlign verticalAlign prose = case verticalAlign of
BaselineAlign
SupAlign
SubAlign -> proseWe don't support case transformations (converting to uppercase, lowercase, etc). The Citeproc library has a Citeproc.CaseTransform module which provides much of the work for this. However, we don't bother with this for now to avoid additional complexity.
addTextCase _ textCase prose = case textCase of
Lowercase
Uppercase
CapitalizeFirst
CapitalizeAll
SentenceCase
TitleCase
PreserveCase -> proseWe don't support display transformations (whether to display as a block, indented, etc). This is presumably for the bibliography list only.
addDisplay displayStyle prose = case displayStyle of
DisplayBlock
DisplayLeftMargin
DisplayRightInline
DisplayIndent -> proseAdding quotes means just surrounding the prose with quote characters.
addQuotes prose = Prose $ [quote] <> prose.unProse <> [quote]
where
quote =
PTokenStyledText
StyledText
{ body = "\"",
style = styleMaskFrom []
}The movePunctuationInsideQuotes function moves all commas and periods to come just before the quote, instead of after it. For example, "a". would be converted into "b.". Because of the way styles are handled though, a data:PToken can only see text that have the same style; if there are two of them, and they each have a piece of the punctuation combination we are concerned about, this won't work. However, such cases should be rare, and even if we did support that case, it wouldn't make sense to move a bold period inside of a non-bold quote character, for example. So working inside of a single data:PToken should be reasonable enough.
movePunctuationInsideQuotes prose =
Prose $ map f prose.unProse
where
f pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = g styledText.body,
style = styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pToken
g s =
let attempts = map (tryRearrange s) punctCombos
in case find (\(mightHaveMore, _) -> mightHaveMore) attempts of
Just (_, s') -> g s'
Nothing -> s
tryRearrange s punctCombo
| (prefix, remainder) <- T.breakOn punctCombo s,
not (T.null remainder) =
( True,
prefix
<> T.reverse punctCombo
<> T.drop (T.length punctCombo) remainder
)
| otherwise = (False, s)
punctCombos :: [Text]
punctCombos =
[ "\".",
"\",",
"'.",
"',"
]We don't support inNote. CslJson doesn't support it either.
inNote = idmapText just applies an arbitrary function to all Text values.
mapText f prose = Prose $ map g prose.unProse
where
g :: PToken -> PToken
g pToken = case pToken of
PTokenStyledText styledText ->
PTokenStyledText
StyledText
{ body = f styledText.body,
style = styledText.style
}
PTokenLink {}
PTokenCitation {}
PTokenMathFragment {}
PTokenDivStart {}
PTokenDivEnd {} -> pTokenaddHyperlink just adds a URL; we have a rich data:Link type for this, so that's what we use.
addHyperlink url prose = Prose $ maybeToList link <> prose.unProse
where
link = case N.parseURI $ T.unpack url of
Nothing -> Nothing
Just uri ->
Just
$ PTokenLink
Link
{ name =
[ StyledText
{ body = url,
style = styleMaskFrom []
}
],
target = LinkTargetURI uri
}We don't support localization of quotes.
localizeQuotes _ prose = prose7.2 Tests
We need to check that there is no information loss going the full round trip from Text to Prose and back to Text again.
checkTextToProseRoundTrip :: String -> [(Text, Prose)] -> Spec
checkTextToProseRoundTrip msg testCases = do
it msg $ do
forM_ testCases $ \(text, prose) -> do
CP.fromText text `shouldBe` prose
CP.toText prose `shouldBe` textdescribe "Text to Prose round trip" $ do
checkTextToProseRoundTrip
"can handle StyledText"
[ ( "no style",
Prose [ptext "no style" []]
),
( "*bold*",
Prose [ptext "bold" [Bold]]
),
( "*bo\\*ld (escaped)*",
Prose [ptext "bo*ld (escaped)" [Bold]]
),
( "/italic/",
Prose [ptext "italic" [Italic]]
),
( "/it\\/alic (escaped)/",
Prose [ptext "it/alic (escaped)" [Italic]]
),
( "_underline_",
Prose [ptext "underline" [Underline]]
),
( "_under\\_line (escaped)_",
Prose [ptext "under_line (escaped)" [Underline]]
),
( "=*/_~verbatim=",
Prose [ptext "*/_~verbatim" [Verbatim]]
),
( "~*/_=code~",
Prose [ptext "*/_=code" [Code]]
)
]
checkTextToProseRoundTrip
"can handle OID links"
[ ( "[[id:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa1]]",
Prose
[ PTokenLink
Link
{ name = [stext "id:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa1" []],
target =
LinkTargetOid
LinkOid
{ oid = fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaa1,
name = ""
}
}
]
),
( "[[id:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa2][*foo*]]",
Prose
[ PTokenLink
Link
{ name = [stext "foo" [Bold]],
target =
LinkTargetOid
LinkOid
{ oid = fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaa2,
name = ""
}
}
]
),
( "[[id:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa3::abc][/bar/]]",
Prose
[ PTokenLink
Link
{ name = [stext "bar" [Italic]],
target =
LinkTargetOid
LinkOid
{ oid = fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaa3,
name = "abc"
}
}
]
)
]
checkTextToProseRoundTrip
"can handle footnote links"
[ ( "[fn:abc]",
Prose
[ PTokenLink
Link
{ name = [stext "1" []],
target = LinkTargetFootnote "abc"
}
]
),
( "[fn:abc][fn:xyz][fn:abc]",
Prose
[ PTokenLink
Link
{ name = [stext "1" []],
target = LinkTargetFootnote "abc"
},
PTokenLink
Link
{ name = [stext "2" []],
target = LinkTargetFootnote "xyz"
},
PTokenLink
Link
{ name = [stext "1" []],
target = LinkTargetFootnote "abc"
}
]
)
]
checkTextToProseRoundTrip
"can handle web links"
[ ( "[[https://foo.org][bar]]",
Prose
[ PTokenLink
Link
{ name = [stext "bar" []],
target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "foo.org",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
}
}
]
),
( "https://foo2.org",
Prose
[ PTokenLink
Link
{ name = [stext "https://foo2.org" []],
target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "foo2.org",
N.uriPort = ""
},
N.uriPath = "",
N.uriQuery = "",
N.uriFragment = ""
}
}
]
)
]
checkTextToProseRoundTrip
"can handle file links"
[ ( "[[file:foo/bar]]",
Prose
[ PTokenLink
Link
{ name = [stext "foo/bar" []],
target = LinkTargetFilePath "foo/bar"
}
]
),
( "[[file:foo/bar][baz]]",
Prose
[ PTokenLink
Link
{ name = [stext "baz" []],
target = LinkTargetFilePath "foo/bar"
}
]
)
]
checkTextToProseRoundTrip
"can handle inline math fragments"
[ ( "\\(4 = 2 * 2\\)",
Prose
[ PTokenMathFragment
MathFragment
{ parentOid = nilOid,
body = "4 = 2 * 2",
meta = fromList []
}
]
)
]We can't check for non-inline math fragments in the test above, because f:proseParser only allows for inline math fragments.
Check we can drop text characters from either end of newtype:Prose.
describe "Drop text chars from Prose" $ do
it "can handle plain text from the front" $ do
CP.dropTextWhile (== 'a') (CP.fromText "aa b *cc*" :: Prose)
`shouldBe` (CP.fromText " b *cc*")
it "can handle plain text from the back" $ do
CP.dropTextWhileEnd (== 'c') (CP.fromText "aa b *cc*" :: Prose)
`shouldBe` (CP.fromText "aa b ")
it "can handle styled (italic) text from the front" $ do
CP.dropTextWhile (== 'a') (CP.fromText "/aa/ b *cc*" :: Prose)
`shouldBe` (CP.fromText " b *cc*")
it "can handle styled (italic) text from the back" $ do
CP.dropTextWhileEnd (== 'c') (CP.fromText "/aa/ b *cc*" :: Prose)
`shouldBe` (CP.fromText "/aa/ b ")
it "skips over non-text elements from the front" $ do
CP.dropTextWhile (== 'a') (CP.fromText "[[foo]]/aa/ b *cc*" :: Prose)
`shouldBe` (CP.fromText "[[foo]] b *cc*")
it "skips over non-text elements from the back" $ do
CP.dropTextWhileEnd (== 'c') (CP.fromText "/aa/ b *cc*[[foo]]" :: Prose)
`shouldBe` (CP.fromText "/aa/ b [[foo]]")8 Paragraphs
The f:paragraphParser is similar to f:headingParser because it's also powered mostly by f:proseParser. However a key difference is that it is aware of the nuances between newlines, indentation, and spaces.
The main concern is to understand where the paragraph starts. Paragraphs can start in three ways:
without indentation,
after a list head (which is essentially indentation), or
with indentation (as part of a list).
In all three cases, we would like to check whether a paragraph's inner lines have consistent indentation. This way, a paragraph's lines all line up vertically on the same column, and is visually pleasing. Doing this check means understanding what is the expected indentation (if at all) for lines inside the paragraph.
paragraphParser :: OrgParser Cell
paragraphParser = do
pState <- get
modify (\s -> s {insideParagraph = True, glossaryTermOffset = 0})
prose <-
proseParser
. whitespaceCompressor
$ proseSpaceConsumer pState.continuationIndent -- 102
failIfLeftoverActiveStyles
pState' <- get
modify (\s -> s {insideParagraph = False})
if pState'.glossaryTermOffset > 0
then convertToGlossaryTerm prose pState'.glossaryTermOffset -- 103
else pure . CParagraph $ compressProse proseThe check at 103 is for seeing if the current paragraph qualifies as a glossary term. If not, we parse it as a normal paragraph.
The f:proseSpaceConsumer 102 knows how to parse spaces across lines (while taking into account the existing level of indentation in the current line, if any). This is a top-level function because we reuse it in f:linkDisplayNameParser.
proseSpaceConsumer :: Int -> OrgParser Text
proseSpaceConsumer expectedIndentLen = do
_ <- beLessGreedy
spaceConsumer
where
spaceConsumer :: OrgParser Text
spaceConsumer =
MP.choice
[ newlineAndIndentParser expectedIndentLen,
onlySpaces
]
beLessGreedy :: OrgParser ()
beLessGreedy = do
pState <- get
let toAvoid =
[ (pState.insideCitePrefix, spaceConsumer >> citeIdParser)
]
forM_
toAvoid
$ \(condition, p) -> when condition $ MP.notFollowedBy pThere are a couple tricky bits here. The first tricky part is with the beLessGreedy helper. It's important that the spaceConsumer does not steal either the space or newline just preceding the f:citeIdParser; otherwise, the citePrefixParser in f:citeParser will never succeed because it requires some whitespace, only after which we parse the citeId. This is why beLessGreedy tries to abort the parser if it detects a cite key like @foo just after the whitespace (which will be parsed by spaceConsumer).
The second tricky part is with spaceConsumer. It chooses either a newline followed by an indent of length expectedIndentLen (in f:newlineAndIndentParser), or just spaces. The f:newlineAndIndentParser is what ensures we have consistent indentation for lines of a paragraph.
newlineAndIndentParser :: Int -> OrgParser Text
newlineAndIndentParser expectedIndentLen = do
MP.notFollowedBy cellBoundary -- 104
_ <- string "\n"
maybeIndent -- 105
pure " "
where
maybeIndent =
if (expectedIndentLen == 0)
then nonSpace
else MP.eof <|> exactIndent
nonSpace :: OrgParser ()
nonSpace = MP.notFollowedBy (char ' ')
exactIndent :: OrgParser ()
exactIndent = do
_ <-
MP.label "indentation for paragraph continuation"
$ MP.count expectedIndentLen (char ' ')
MP.notFollowedBy (char ' ')
MP.notFollowedBy (char '\n')
pure ()The MP.notFollowedBy cellBoundary bit 104 is required to stay clear of cell boundaries (paragraph cells are separated from each other by a blank line, or in other words a sequence of two consecutive newlines). If we don't do this, our parser will see a newline and immediately think that there should follow some spaces for the indent portion, which may not be true if the newline was part of a cell boundary.
cellBoundary :: OrgParser Text
cellBoundary = string "\n\n"The maybeIndent 105 handles the case where there is no indentation (with nonSpace), or where there is indentation (with MP.eof <|> exactIndent). The first case does not need MP.eof <|> because it's redundant --- MP.notFollowedBy (char ' ') will succeed for MP.eof as well, as t:0005-notFollowedBy-behavior-for-empty-input demonstrates.
it "accepts empty input for \"notFollowedBy (char ' ')\"" $ do
MP.parse
(MP.notFollowedBy (char ' ') :: Parser ())
"empty input"
`shouldSucceedOn` ""8.1 Tests
Below we check several types of paragraphs without regard to indentation.
a b c d e *f g h i j*
x y
z foo
bar
_multiple
lines
with
underscore_shouldParseWith
"0005-prose-paragraphs"
$ nilOrgDoc
Expected prose for 0005-prose-paragraphs
[][ CParagraph
$ Prose
[ ptext "a b c d e " [],
ptext "f g h i j" [Bold]
],
CParagraph $ Prose [ptext "x y z foo bar" []],
CParagraph $ Prose [ptext "multiple lines with underscore" [Underline]]
]The paragraphs in the test case above all start without any indentation. This is because indentation is invalid for standalone paragraphs that are not part of a list in some way.
foo barshouldFailWithMessage
"0008-indentation-paragraph"
"invalid indentation"
1For tests involving paragraphs and lists, see those prefixed with 0005-paragraph-indentation-*, such as ti:0005-paragraph-indentation-mismatch-over.
9 Headings
Headings start with asterisks without any indentation, followed by some prose text (see the discussion of prose text).
* space before star is not allowed for headingsshouldFailWithMessage
"0008-indentation-headings"
"invalid indentation"
1The number of asterisks determine the nesting level. By default there can be only 3 levels (maximum of 3 asterisks for a heading), although you can raise this number with Lilac extensions). Still, there should be probably no more than 6 asterisks for a heading because HTML heading tags <h1>, <h2>, etc only go up to <h6>). This Org document itself only uses 3 sections.
The heading may also have some metadata associated with it. In practice this will be keys and values from a :PROPERTIES: drawer, although other drawers are possible. Drawers are discussed separately here.
We store drawer information inside the meta hashmap 106, with the drawer name acting as a key prefix (this way we can store multiple drawers, if there are any, in a single hashmap).
Other than the level, each heading is made up of Prose text.
The section field is generated during the parse, and is the section number like 1.1.2 or 5.3, represented as a list of integers. The numbering scheme matches the behavior of org-num-mode, so it is highly recommended to turn that on in Emacs. The value at the head of the list represents the deepest level (it's reversed); for example, section 5.3 is represented as [3, 5].
The parser for a heading is straightforward as the strict requirements make it easy to have simple rules for it.
headingParser :: OrgParser Cell
headingParser = do
pState <- get
_ <- MP.try $ MP.lookAhead (starsParser >> char ' ')
let opc = pState.orgParserConf
stars <- starsParser
when (opc.maxHeadingLevel < 0) $ do
fail
$ "headings prohibited because lilac_maxHeadingLevel < 0"
let lenStars = T.length stars
currentHeadingLevel = pState.headingLevel
when (opc.maxHeadingLevel > 0 && lenStars > opc.maxHeadingLevel) $ do
fail
$ "found "
<> show lenStars
<> " asterisks found for this heading, but "
<> "lilac_maxHeadingLevel = "
<> show opc.maxHeadingLevel
failIfHeadingNestsWithoutDirectParent currentHeadingLevel lenStars
_ <- char ' '
modify (\s -> s {insideHeading = True})
prose <- proseParser $ whitespaceCompressor onlySpaces
_ <- char '\n'
failIfLeftoverActiveStyles
props <- propertiesDrawerParser
let newHeadingSection = genSectionNumber lenStars pState.headingSectionNumber
newHeadingOidStack =
genHeadingOidStack
(lookup "ID" props)
lenStars
pState.headingOidStack
put
pState
{ headingLevel = lenStars,
headingSectionNumber = newHeadingSection,
headingOidStack = newHeadingOidStack,
insideHeading = False
}
pure
$ CHeading
Heading
{ level = lenStars,
section = newHeadingSection,
body = compressProse prose,
meta = props
}
where
starsParser :: OrgParser Text
starsParser = MP.takeWhile1P (Just "heading star") (\c -> c == '*')We fail if the heading level is not compliant with the setting of lilac_maxHeadingLevel, which is accessed via opc.maxHeadingLevel (see Lilac extensions).
We also fail if we try to nest the heading too much in f:failIfHeadingNestsWithoutDirectParent. See ti:0004-headings-cannot-skip-level for an example.
failIfHeadingNestsWithoutDirectParent :: Int -> Int -> OrgParser ()
failIfHeadingNestsWithoutDirectParent current found =
when (found > current + 1) $ do
fail
$ "headings may not skip levels (expected to see level "
<> show current
<> " or "
<> (show $ current + 1)
<> ", not "
<> show found
<> ")"9.1 Section number generation
Generating the section number relies on the previous section, and the current level of the heading for which we want to generate the section number.
genSectionNumber :: Int -> [Int] -> [Int]
genSectionNumber level prevSection
| null prevSection = 1 : (take (level - 1) $ repeat 0)
| length prevSection == level = bumpDeepestSection prevSection
| length prevSection > level =
bumpDeepestSection
$ drop (length prevSection - level) prevSection
| otherwise =
let levelDiff = (level - length prevSection - 1)
in (1 : (take levelDiff $ repeat 0)) <> prevSection -- 107
where
bumpDeepestSection :: [Int] -> [Int]
bumpDeepestSection [] = []
bumpDeepestSection (x : xs) = x + 1 : xsThe algorithm starts with the base case where we have no previous section (null prevSection), where we assign "0" section numbers for the missing levels, if any. The same thing applies for the case where we nest too deeply than a previously seen level 107.
Although f:genSectionNumber takes into account these edge cases where we have a "0" section number, the parser itself fails if we skip any levels (if we nest too deeply at any point), in f:failIfHeadingNestsWithoutDirectParent.
describe "genSectionNumber" $ do
it "gets the base case" $ do
L.genSectionNumber 2 [] `shouldBe` [1, 0]
L.genSectionNumber 1 [] `shouldBe` [1]
L.genSectionNumber 0 [] `shouldBe` [1]
L.genSectionNumber (-1) [] `shouldBe` [1]
it "bumps the last level if the nesting level matches" $ do
L.genSectionNumber 1 [1] `shouldBe` [2]
L.genSectionNumber 2 [1, 1] `shouldBe` [2, 1]
L.genSectionNumber 3 [1, 1, 1] `shouldBe` [2, 1, 1]
it "truncates, then increments the (newst) deepest level if we unnest" $ do
L.genSectionNumber 1 [3, 5] `shouldBe` [6]
L.genSectionNumber 3 [3, 5, 6, 7] `shouldBe` [6, 6, 7]
it "grows the section if we nest deeper" $ do
L.genSectionNumber 3 [2, 2] `shouldBe` [1, 2, 2]
L.genSectionNumber 5 [2, 2] `shouldBe` [1, 0, 0, 2, 2]9.2 Generating the current Org ID
f:genHeadingOidStack is quite similar to f:genSectionNumber. The point of it is to store the Org IDs up to the current point. This is ultimately used to assign the OID of the first parent heading (if any) for an implicitly-defined code block, in f:oidReferenceParser. If there are no headings, then it defaults to the top-level (required) OID of the document.
genHeadingOidStack :: Maybe Text -> Int -> [Maybe OID] -> [Maybe OID]
genHeadingOidStack maybeId level old
| length old == 0 = [newId]
| length old == level = newId : drop 1 old
| length old < level =
let nothings = take (levelDiff - 1) $ repeat Nothing
in (newId : nothings) <> old
| otherwise = newId : (drop (abs levelDiff + 1) old)
where
levelDiff = level - length old
newId = Lilac.Types.fromText =<< maybeIddescribe "genHeadingOidStack" $ do
it "replaces the head if the nesting level matches (no heading yet)" $ do
L.genHeadingOidStack
(Just "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
1
[Nothing]
`shouldBe` [Just $ fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaaa]
it "replaces the head if the nesting level matches (one heading exists)" $ do
L.genHeadingOidStack
(Just "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
1
[Just $ fromWords 0x7777777777777777 0x7777777777777777]
`shouldBe` [Just $ fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaaa]
it "adds a new head if heading nests (new heading has an ID)" $ do
L.genHeadingOidStack
(Just "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
2
[Nothing]
`shouldBe` [ Just $ fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaaa,
Nothing
]
it "adds a new head if heading nests (new heading does not have an ID)" $ do
L.genHeadingOidStack
Nothing
2
[Nothing]
`shouldBe` [ Nothing,
Nothing
]
it "truncates if we unnest (difference of 1 level)" $ do
L.genHeadingOidStack
(Just "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
1
[Just $ fromWords 0x7777777777777777 0x7777777777777777, Nothing]
`shouldBe` [Just $ fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaaa]
it "truncates if we unnest (multiple levels)" $ do
L.genHeadingOidStack
Nothing
1
[Nothing, Nothing, Just $ fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaaa]
`shouldBe` [Nothing]
it "truncates if we unnest (multiple levels, new heading has ID)" $ do
L.genHeadingOidStack
(Just "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
1
[Nothing, Nothing, Just $ fromWords 0x7777777777777777 0x7777777777777777]
`shouldBe` [Just $ fromWords 0xaaaaaaaaaaaaaaaa 0xaaaaaaaaaaaaaaaa]9.3 Tests
By default we only allow 3 levels of headings.
* 1
** 2
*** 3
**** 4
***** 5
****** 6
******* 7shouldFailWithMessage
"0004-headings"
"found 4 asterisks found for this heading, but lilac_maxHeadingLevel = 3"
19We can have many levels of headings beyond the 3 that we have by default, by setting the lilac_maxHeadingLevel metadata. We can set it to some high number, or set it to 0 (unlimited). First we demonstrate using a higher number, 20.
#+lilac_maxHeadingLevel: 20
* 1
** 2
*** 3
**** 4
***** 5
****** 6
******* 7
******** 8
********* 9
********** 10
*********** 11
************ 12shouldParseWithState
"0004-headings-20"
( nilOrgDoc
[ plainHeading 1 [1],
plainHeading 2 [1, 1],
plainHeading 3 [1, 1, 1],
plainHeading 4 [1, 1, 1, 1],
plainHeading 5 [1, 1, 1, 1, 1],
plainHeading 6 [1, 1, 1, 1, 1, 1],
plainHeading 7 [1, 1, 1, 1, 1, 1, 1],
plainHeading 8 [1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 9 [1, 1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 10 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 11 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 12 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
]
[]
)
defaultOrgParserState
{ headingOidStack = take 12 $ repeat Nothing,
headingLevel = 12,
headingSectionNumber = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
orgParserConf = L.defaultOrgParserConf {maxHeadingLevel = 20}
}Now we check using 0 for unlimited nesting.
#+lilac_maxHeadingLevel: 0
* 1
** 2
*** 3
**** 4
***** 5
****** 6
******* 7
******** 8
********* 9
********** 10
*********** 11
************ 12shouldParseWithState
"0004-headings-unlimited"
( nilOrgDoc
[ plainHeading 1 [1],
plainHeading 2 [1, 1],
plainHeading 3 [1, 1, 1],
plainHeading 4 [1, 1, 1, 1],
plainHeading 5 [1, 1, 1, 1, 1],
plainHeading 6 [1, 1, 1, 1, 1, 1],
plainHeading 7 [1, 1, 1, 1, 1, 1, 1],
plainHeading 8 [1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 9 [1, 1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 10 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 11 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
plainHeading 12 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
]
[]
)
defaultOrgParserState
{ headingOidStack = take 12 $ repeat Nothing,
headingLevel = 12,
headingSectionNumber = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
orgParserConf = L.defaultOrgParserConf {maxHeadingLevel = 0}
}Here we set it to a number less than 3. This is useful for ensuring that we don't have too much nesting.
#+lilac_maxHeadingLevel: 2
* a
** b
*** c
**** dshouldFailWithMessage
"0004-headings-limited"
"found 3 asterisks found for this heading, but lilac_maxHeadingLevel = 2"
39We can also prohibit headings entirely, by setting lilac_maxHeadingLevel to -1.
#+lilac_maxHeadingLevel: -1
* a
**** dshouldFailWithMessage
"0004-headings-prohibited"
"headings prohibited because lilac_maxHeadingLevel < 0"
29We may not skip over any intermediate levels when we nest. For example, you cannot go from level 1 to level 4 (skipping over levels 2 and 3).
#+lilac_maxHeadingLevel: 6
* a
**** dshouldFailWithMessage
"0004-headings-cannot-skip-level"
"headings may not skip levels (expected to see level 1 or 2, not 4)"
3510 Drawers
In vanilla Orgmode, drawers look like this:
:DRAWER_NAME:
...contents...
:END:And Orgmode allows the contents to be anything except headings or other (nested) drawers.
In Lilac, we only parse the :PROPERTIES: drawer, which is a special drawer that can contain key/value pairs that Org recognizes. One of these is the :ID: key, which can have an OID as the value, and which can be used as a link target from elsewhere in the document (in Orgmode this can be done with org-id-store-link).
propertiesDrawerParser :: OrgParser (Map Text Text)
propertiesDrawerParser = propertiesParser <|> (pure $ fromList [])The f:propertiesDrawerParser tries to parse a :PROPERTIES: drawer with f:propertiesParser, but if it fails it returns an empty Map.
propertiesParser :: OrgParser (Map Text Text)
propertiesParser =
fromList
<$> MP.between
(lexeme $ string ":PROPERTIES:\n")
(string ":END:\n")
(many $ lexeme propertiesEntryParser)The f:propertiesParser is just a thin wrapper around f:propertiesEntryParser which does the heavy lifting of figuring out the key value parts.
The uses of f:lexeme allow this parser to ignore comments it finds inside the drawer area. There is no need to use f:lexeme for the closing :END: bit because we already wrap all cells with f:lexeme in f:cellParser, and the last thing f:headingParser parses is f:propertiesParser (if f:headingParser parsed something else after calling f:propertiesParser, we would need to wrap :END: with f:lexeme as well).
propertiesEntryParser :: OrgParser (Text, Text)
propertiesEntryParser = do
MP.notFollowedBy $ string ":END:"
_ <- char ':'
key <- MP.label "property key" . MP.some $ MP.noneOf ['\n', ':', ' ']
_ <- char ':'
_ <- onlySpaces
value <-
MP.manyTill
(MP.label "property value" MP.anySingle)
(char '\n')
pure (T.pack key, T.pack value)The main tricky bit is MP.notFollowedBy $ string ":END:"; we need this because otherwise we will think that the string :END: is just another property key.
10.1 Tests
The :PROPERTIES: drawer may be empty.
* foo
:PROPERTIES:
:END:shouldParseWithState
"0004-properties-drawer-empty"
( nilOrgDoc
[ CHeading
Heading
{ level = 1,
section = [1],
body = Prose [ptext "foo" []],
meta = fromList []
}
]
[]
)
defaultOrgParserState
{ headingOidStack = [Nothing],
headingLevel = 1,
headingSectionNumber = [1]
}Check we can pick up keys and values from the :PROPERTIES: drawer.
* foo
:PROPERTIES:
:ID: b949ea2e-e6d1-453d-9352-5e85a82cd4e6
:FOO: BAR
:END:shouldParseWithState
"0004-properties-drawer-not-empty"
( nilOrgDoc
[ CHeading
Heading
{ level = 1,
section = [1],
body = Prose [ptext "foo" []],
meta =
fromList
[ ("ID", "b949ea2e-e6d1-453d-9352-5e85a82cd4e6"),
("FOO", "BAR")
]
}
]
[]
)
defaultOrgParserState
{ headingOidStack =
[Just $ fromWords 0xb949ea2ee6d1453d 0x93525e85a82cd4e6],
headingLevel = 1,
headingSectionNumber = [1]
}Adding comments inside drawers is allowed (we ignore them because we use f:lexeme in f:propertiesParser).
* foo
:PROPERTIES:
# Comment.
:ID: b949ea2e-e6d1-453d-9352-5e85a82cd4e6
#+begin_comment
This ":KEY:" ignored because it's inside a comment block.
:KEY: value
#+end_comment
:END:shouldParseWithState
"0004-properties-drawer-not-empty-with-comments"
( nilOrgDoc
[ CHeading
Heading
{ level = 1,
section = [1],
body = Prose [ptext "foo" []],
meta =
fromList
[ ("ID", "b949ea2e-e6d1-453d-9352-5e85a82cd4e6")
]
}
]
[]
)
defaultOrgParserState
{ headingOidStack =
[Just $ fromWords 0xb949ea2ee6d1453d 0x93525e85a82cd4e6],
headingLevel = 1,
headingSectionNumber = [1]
}11 Blocks
Blocks are regions of text that start with #+begin<...> and end with #+end<...>. They are used to delineate source code (such as those used for Noweb references), examples, quotes, and others. Some examples are src or example (as in #+begin_src and #+begin_example), although technically Org allows for any arbitrary keyword here.
The type of block (such as src or example as discussed above) is stored in the metadata map meta 109, with the __block_type key. See Expected blocks for 0007-blocks-empty for examples, as well as the Metadata section for more a more detailed discussion.
The parentOid 108 is the closest heading which has a :ID: property (aka OID), or the OID of the document if none of the headings have an OID (or if there are simply no headings preceding the block). Using Emacs functions like org-id-store-link will generate the OID automatically for the parent heading (creating a property drawer, adding the :ID: key, etc) if it doesn't already have an :ID:.
We need to store the parentOid here because ultimately, we need to build up a hashmap of symbols in f:mkSymbols, which uses the parentOid as a building block. Once we have all the symbols (using parentOid fields) defined such that we know what they point to, we are able to look at a data:LinkOid and are able to determine what that link actually points to (whether it's a block, table, figure, etc). This allows us to know how to weave these links (specifically, this information about symbols lets us know how to weave the destination, because different cells have different link anchors --- see f:weaveLinkTarget for details). All cells that can be named (not just blocks, such as tables and figures as stated above) have an equivalent parentOid field to assist with this, for this same reason.
Lastly, the body field 110 stores the actual contents of the code block. Briefly, the body can have either text or links to other code blocks --- also known as Noweb references. See Body for details.
11.1 Metadata
There are two kinds of metadata formats that blocks accept:
The #+key: value format. The most common (and probably important) one is the #+name: ... metadata, as it allows blocks to be referenced by other blacks by its name. The generic f:metaParser is used to parse these values (not just for blocks but also for Figures).
The #+header: :key value format. The most common ones here is #+header: :tangle ... which specifies the tangle path of the code block. f:blockMetaEntryParser is used to parse these ones.
The reason why blocks have two separate formats like this is because that's what Orgmode uses as well. The test case below has some examples of valid (empty) code blocks with some typical metadata.
#+begin_src haskell
#+end_src
#+name: foo
#+begin_example
#+end_example
#+begin_quote
#+end_quote
#+header: :tangle foo.hs
#+begin_src haskell
#+end_srcshouldParseWithState
"0007-blocks-empty"
( nilOrgDoc
Expected blocks for 0007-blocks-empty
[]
)
defaultOrgParserState
{ cellNames = Set.fromList ["foo"],
anonBlockNumber = 3,
tanglePaths = fromList ["foo.hs"]
}[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "1"),
("__block_type", "src"),
("__src_language", "haskell")
],
body = []
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "example"),
("name", "foo")
],
body = []
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "2"),
("__block_type", "quote")
],
body = []
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "3"),
("__block_type", "src"),
("__src_language", "haskell"),
("tangle", "foo.hs")
],
body = []
}
]The subsections below discuss each of the different types of metadata parsers.
11.1.1 Generic metadata
The generic f:metaParser is used in f:blockStartParser 122 so that we can check for certain requirements for metadata syntax across multiple cell types that use metadata. For example, the #+name: value cannot start with an open parenthesis.
The metadata is parsed with f:metaParser which relies on f:metaEntryParser. We make sure that the name given to the cell is unique inside the document being parsed, because ultimately these cell names determine how we generate HTML IDs for each cell (although we could give machine-generated ID names, we prefer to use the name if one exists because they are usually much more human-friendly).
metaParser :: OrgParser (Map Text Text)
metaParser = do
metaKVs <- many $ lexeme metaEntryParser
let metaKVsMap = fromList metaKVs
case lookup "name" metaKVsMap of
Just cellName -> assertUniqueCellName cellName
Nothing -> pure ()
pure metaKVsMapThe f:assertUniqueCellName function keeps track of all the named cells we've seen so far. This is used to detect whether we've seen a cell name more than once, which is not allowed.
assertUniqueCellName :: Text -> OrgParser ()
assertUniqueCellName cellName = do
pState <- get
when (Set.member cellName pState.cellNames)
. fail
$ "cell name "
<> show cellName
<> " is not unique"
put
pState
{ cellNames = Set.insert cellName pState.cellNames
}#+name: foo
#+begin_src sh
a
#+end_src
#+name: foo
#+begin_src sh
b
#+end_srcshouldFailWithMessage
"0007-blocks-duplicate-names"
"cell name \"foo\" is not unique"
52The f:metaEntryParser is similar to f:docMetaEntryParser. In the latter, we use avoidParsingBlockMetadata 58 to avoid touching on the toes of f:blockMetaEntryParser, and similarly we have to check that each metadata line does not start with #+begin_ 111 because otherwise this will be too greedy and rob f:blockStartParser the chance to see the #+begin_ string to denote the start of the block.
metaEntryParser :: OrgParser (Text, Text)
metaEntryParser = do
mapM_
MP.notFollowedBy
$ map
string
[ "#+begin_", -- 111
"#+header:",
"#+print_bibliography:",
"#+print_glossary:"
]
_ <- string "#+"
o <- MP.getOffset
key <- MP.takeWhile1P (Just "(meta) keyword letter") isKeywordLetter
_ <- MP.label "(meta) separator" $ string ": "
_ <- many $ char ' '
value <-
if key == "name"
then do
first <- MP.satisfy (\c -> c /= '(' && isValueLetter c) -- 112
rest <- MP.takeWhileP (Just "(meta) value letter") isValueLetter
pure $ T.cons first rest
else MP.takeWhile1P (Just "(meta) value letter") isValueLetter
when (T.isPrefixOf "__" key) -- 113
$ do
MP.setOffset o
fail
$ "cannot use leading double underscores for metadata key: "
<> show key
pure (key, value)
where
isKeywordLetter c = or $ map ($ c) [isAsciiLower, isAsciiUpper, (== '_')]
isValueLetter c = c /= '\n'For the #+name key, we prohibit the use of the opening parenthesis as the first character of the name at 112, because then that would make [[(foo)]] ambiguous (it could refer to either something named (foo), or it could refer to a line label (ref:foo).
#+name: (a)
#+begin_src sh
a
#+end_srcshouldFailWithError
"0007-blocks-no-name-with-opening-parenthesis"
(utoks "(" <> etoks " ")
811.1.2 #+header: ... metadata
Vanilla Org allows you to place these #+header: ... key/value pairs on the same line as the #+begin_... line, like this:
#+begin_src haskell :tangle foo
#+end_srcHowever for Lilac, we don't allow this. The :tangle foo must be written on its own line as #+header: :tangle foo. This enforces a cleaner (albeit verbose) style and also simplifies the parser.
blockMetaEntryParser :: OrgParser (Text, Text)
blockMetaEntryParser = do
_ <- string "#+header: :"
key <- MP.takeWhile1P (Just "(block meta) keyword letter") isKeywordLetter
_ <- char ' '
o <- MP.getOffset
value <- MP.takeWhile1P (Just "value letter") (\c -> c /= '\n')
_ <- char '\n'
assertUniqueTanglePath key value o -- 114
pure (key, value)
where
isKeywordLetter c = or $ map ($ c) [isAsciiLower, isAsciiUpper, (== '-')] -- 115The keyword (isKeywordLetter 115) cannot have underscores in them. This way, it is impossible to conflict with the keywords used internally by Lilac such as __block_type, __src_language, and so forth as seen in Expected blocks for 0007-blocks-empty.
The f:assertUniqueTanglePath function 114 checks that all tangle paths are unique; that is, no two code blocks may share the same tangle path.
assertUniqueTanglePath :: Text -> Text -> Int -> OrgParser ()
assertUniqueTanglePath "tangle" path offset = do
pState <- get
when (Set.member path pState.tanglePaths)
$ do
MP.setOffset $ offset
fail $ "tangle path " <> show path <> " is not unique"
put
pState
{ tanglePaths = Set.insert path pState.tanglePaths
}
assertUniqueTanglePath _ _ _ = pure ()The tangle path is important for literate programming because it's the way in which code blocks actually get written to disk. Forcing tangle paths to be unique guarantees that there are no clashes (one block may not overwrite another).
#+name: foo
#+header: :tangle some/path.sh
#+begin_src sh
a
#+end_src
#+name: bar
#+header: :tangle some/path.sh
#+begin_src sh
b
#+end_srcshouldFailWithMessage
"0007-blocks-duplicate-tangle-paths"
"tangle path \"some/path.sh\" is not unique"
10111.2 Body
The contents or body of a data:Block is list of data:BToken values, which are quite similar to newtype:Prose (a list of data:PToken types). A BToken can be either some raw piece of text, or a link. The link is only for the case of Noweb references. For example, if a block contains a Noweb reference, then it needs to be a link such that the Noweb reference can be dereferenced, and its contents replaced with the source of the referenced code block during tangling.
type BToken :: Type
data BToken
= BTokenString Text
| BTokenPadding
| BTokenLinkTarget LinkTarget
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)f:blockParser parses the body as a list of BToken values (bTokens) 117.
blockParser :: OrgParser Cell
blockParser = do
pState <- get
(blockName, metaKVs, blockType, srcLang) <- blockStartParser
let nowebOff =
(== Just "off") -- 116
$ lookup "noweb" metaKVs
lineLabelOff =
(== Just "off")
$ lookup "line-label" metaKVs
modify (\s -> s {insideCodeBlock = True})
bTokens <- -- 117
MP.choice -- 118
[ zeroBTokensParser blockType,
someBTokensParser blockType (not nowebOff) (not lineLabelOff)
]
let newAnonBlockNumber = case blockName of
Just _ -> pState.anonBlockNumber
Nothing -> pState.anonBlockNumber + 1
anonBlockMeta = case blockName of -- 119
Just _ -> Nothing
Nothing -> Just ("__anon_block_number", T.show newAnonBlockNumber)
block =
Block
{ parentOid =
firstParentOid pState.docOid pState.headingOidStack, -- 120
meta =
foldl'
(\acc (k, v) -> insert k v acc)
(insert "__block_type" blockType metaKVs)
$ concatMap maybeToList [blockName, srcLang, anonBlockMeta],
body =
ignoreSpecialCommas
. compressBTokens
$ padConsecutiveNowebLinks bTokens -- 121
}
modify
( \s ->
s
{ anonBlockNumber = newAnonBlockNumber,
insideCodeBlock = False
}
)
pure $ CBlock blockBefore we discuss how we parse BToken values in detail, let's discuss some of the other bits first.
f:blockStartParser identifies the beginning of a block.
blockStartParser ::
OrgParser
( Maybe (Text, Text),
Map Text Text,
Text,
Maybe (Text, Text)
)
blockStartParser = do
let blockMetaParser = many $ lexeme blockMetaEntryParser
_ <-
MP.try
. MP.lookAhead
$ do
_ <- metaParser
_ <- blockMetaParser
string "#+begin_" -- 122
metaKVs <- metaParser
let blockName = case lookup "name" metaKVs of
Just name -> Just ("name", name)
Nothing -> Nothing
blockMetaKVs <- fromList <$> blockMetaParser
_ <- string "#+begin_"
blockType <- MP.takeWhile1P (Just "block type") isAsciiLower
srcLang <- MP.optional $ do
_ <- MP.skipSome $ char ' '
src <- MP.takeWhile1P (Just "src language") isSrcLanguageChar
pure ("__src_language", src)
when (blockType == "src" && srcLang == Nothing)
$ fail "block type src requires a language specifier" -- 123
let finalKVs =
-- 124
if (blockName == Just ("name", "index"))
&& (notMember "hidden" blockMetaKVs)
then insert "hidden" "true" blockMetaKVs
else blockMetaKVs
_ <- char '\n' -- 125
pure (blockName, finalKVs, blockType, srcLang)
where
isSrcLanguageChar c = isAsciiLower c || c == '-'We first have to use the same try/lookAhead trick we use in f:figureParser at the beginning 122, because otherwise our parser thinks that as soon as we parse a #+name, that it will be a block. This is not true because other things (like figures) can have a #+name associated with them. We fail the parse if we detect a #+begin_src block without an accompanying language 123; see ti:0007-block-src-without-lang for an example.
124 automatically makes "index" blocks hidden, because their contents are not very interesting (they can be seen by looking at the left sidebar in the site contents).
Back to f:blockParser. The nowebOff bit 116 controls how we perceive text that looks like <<foo>>; if we've disabled Noweb references, then <<foo>> is not given any special treatment. See the discussion for ti:0007-blocks-noweb and ti:0007-blocks-noweb-off at Noweb reference.
Blocks that don't have a name are assigned a __anon_block_number metadata property 119, so that we can use this information to generate a link anchor for it during weaving.
The call to f:firstParentOid 120 gets the first available newtype:OID we can find, going up the headingOidStack in data:OrgParserState.
firstParentOid :: OID -> [Maybe OID] -> OID
firstParentOid docOid maybeOids = case catMaybes maybeOids of
(oid : _) -> oid
[] -> docOidOK, back to BToken values. The tricky part is finding out the end of the block. We must parse the two cases of whether there is or isn't any content inside the block separately, via f:zeroBTokensParser and f:someBTokensParser. We need this distinction because we require blocks to end with "\n#+end_..." (that is, the #+end_... must occur at the beginning of a line). This means that for empty blocks, this newline character just preceding the #+end_... will come from the newline at the end of the #+begin_... part. This then interferes with the char '\n' parser 125 we already have in f:blockStartParser just before we start parsing for bTokens.
In other words, there is no easy way to tell Megaparsec to ensure that we are at the beginning of a line. To work around this, we check whether the block is empty with f:zeroBTokensParser.
zeroBTokensParser :: Text -> OrgParser [BToken]
zeroBTokensParser blockType = do
_ <- MP.choice [noBlankLine, blankLine]
pure []
where
noBlankLine = string $ T.concat ["#+end_", blockType, "\n"]
blankLine = string $ T.concat ["\n#+end_", blockType, "\n"]If zeroBTokensParser fails, then that means there is some content inside the block, and therefore f:someBTokensParser must succeed (hence the use of MP.choice 118 in f:blockParser).
See ti:0007-blocks-ending which checks the various edge cases around making sure we can detect the "\n#+end_..." bit.
Because we treat the newline that immediately precedes the #+end_... bit as part of the ending marker, we leave it out of the parsed contents. This actually helps us when we have to tangle blocks in f:expandBlock. If we parsed in the trailing newlines as BToken values, these newlines would build up every time we nested into a block (e.g., t:0101-block-expansion-deeply-nested).
f:someBTokensParser relies on f:bTokenParser to do the actual parsing of BToken values, which form the contents of the code block. These BToken values can be either text (BTokenString) or a Noweb link (BTokenLinkTarget). Essentially all of the complexity around f:bTokenParser is due to the need to detect Noweb references in a robust way.
bTokenParser :: Bool -> Bool -> Bool -> OrgParser BToken
bTokenParser isSrcBlock nowebOn lineLabelOn = MP.choice parsers
where
baseParsers :: [OrgParser BToken]
baseParsers =
-- 127
[ BTokenString
<$> ( MP.takeWhile1P
(Just "block token")
(`notElem` ['\n', ' ', '<'])
),
BTokenString
. T.singleton
<$> (MP.label "left angle bracket" $ MP.satisfy (== '<')), -- 128
BTokenString
. T.singleton
<$> (MP.label "space inside block" $ MP.satisfy isSpace)
]
parsers = [p | (b, p) <- extraParsers, b] <> baseParsers
extraParsers =
[ ( isSrcBlock && nowebOn,
BTokenLinkTarget <$> MP.try nowebLinkTargetParser -- 129
),
(lineLabelOn, BTokenLinkTarget <$> lineLabelParser)
]The need to detect Noweb references is why baseParsers 127 has individual (separate) parsers; having these smaller separate parsers breaks up the parsing of block tokens into multiple separate steps. That way, no single parser is too greedy and we can handle the edge cases such as in ti:0007-blocks-noweb-intraline.
Along the same vein, it's important that the space parser (MP.satisfy isSpace) in baseParsers in f:bTokenParser does not use MP.takeWhile1P. Otherwise, it'll greedily consume all spaces including newlines, robbing it from the ending condition (where we expect a newline before #+end_ 126) in f:someBTokensParser. The parser for the left angle bracket 128 is there to take care of an edge case where a Noweb reference is immediately preceded by another left angle bracket <; see t:0007-blocks-noweb-intraline for details.
Going back to f:blockParser, there are three transformations 121 we do for the parsed bTokens --- f:ignoreSpecialCommas, f:compressBTokens, and f:padConsecutiveNowebLinks. Let's look at these in detail.
11.2.1 Ignore special commas
The f:ignoreSpecialCommas function ignores commas at the beginning of a line (with or without indentation), when they are used to escape asterisks (*) and hash signs followed by a plus character (#+). This is because the org-mode major mode in Emacs is unable to cope with these special characters, and requires users to put in leading commas to help the (Elisp-based) parser in that mode. Our Megaparsec-based implementation does not need such help; however we teach our parser about this behavior here so that we can remain compatible with org-mode (otherwise when users tangle their Org file using Lilac, they might get confused why we don't escape such leading commas --- not to mention the likely breakage of Emacs's syntax highlighting for these areas).
ignoreSpecialCommas :: [BToken] -> [BToken]
ignoreSpecialCommas = reverse . foldl' f []
where
f acc (BTokenString text) = (BTokenString $ removeSpecialCommas text) : acc
f acc BTokenPadding = BTokenPadding : acc
f acc link = link : acc
removeSpecialCommas text =
T.intercalate "\n"
. map removeSpecialComma
$ T.split (== '\n') text
removeSpecialComma line
| hasSpecialComma line = dropLeadingComma line
| otherwise = line
hasSpecialComma line
| (indentAndCommas, rest) <- T.breakOnEnd "," line,
(indent, commas) <- T.breakOn "," indentAndCommas,
T.stripStart indent == "",
not $ T.null commas,
T.all (== ',') commas,
T.take 1 rest == "*" || T.take 2 rest == "#+" =
True
| otherwise = False
dropLeadingComma line =
let (indent, commas) = T.breakOn "," line
in indent <> T.drop 1 commas#+begin_src sh
Special commas get removed:
,*
,#+
,,*
,,#+
Special commas may have leading whitespace:
,*
Non-special commas stay put:
,
,.
, x
#+end_srcIf you are reading this (and the above code block) from Emacs Orgmode, realize that we have to do escaping ourselves here, which can be a bit confusing. See test/Lilac/ParseSpec/0007-blocks-special-commas for the raw text to see what it really looks like. The woven HTML version agrees with the raw text, so it should be less tricky to understand.
shouldParseWithState
"0007-blocks-special-commas"
( nilOrgDoc
Expected blocks for 0007-blocks-special-commas
[]
)
defaultOrgParserState
{ anonBlockNumber = 1
}[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "1"),
("__block_type", "src"),
("__src_language", "sh")
],
body =
[ BTokenString
. T.stripEnd
$ T.unlines
[ "Special commas get removed:",
"*",
"#+",
",*",
",#+",
"Special commas may have leading whitespace:",
" *",
"Non-special commas stay put:",
",",
",.",
", x"
]
]
}
]11.2.2 Compress BToken values
The f:compressBTokens function is similar to f:compressProse, but it's simpler because it doesn't have to deal with StyledText. Instead it only deals with raw Text values directly. We need to use compressBTokens because otherwise we'll end up with many separate BToken values (because f:bTokenParser uses multiple separate parsers that all use the same BTokenString constructor for parsing text tokens).
compressBTokens :: [BToken] -> [BToken]
compressBTokens = reverse . foldl' f []
where
f :: [BToken] -> BToken -> [BToken]
f [] token = token : []
f ((BTokenString text1) : rest) (BTokenString text2) =
(BTokenString $ T.append text1 text2) : rest
f acc token = token : acc11.2.3 Noweb references
To understand f:padConsecutiveNowebLinks 121 in f:blockParser, we have to first look at how we parse Noweb references, aka links. We use the term reference and link somewhat interchangeably here, because the only way to refer to another code block is with a Noweb reference, which automatically becomes a link upon weaving.
A Noweb reference is a reference to another block of code. During tangling, the reference will be replaced with the contents of that other block of code, which we call block expansion. That other block can itself have Noweb references. Cyclic references are (obviously) forbidden, because they will result in infinite expansions. See Cycle detection for Noweb links and also f:expandBlock for how Noweb references are expanded.
To write a Noweb reference, you have to surround it with double-angle-bracket delimiters << and >>. The name itself can be anything other than a newline. Parsing Noweb references is handled by f:nowebLinkTargetParser. We wrap this with MP.try 129 in f:bTokenParser, because we need to backtrack if we fail (after parsing, and consuming, the initial << bit). Backtracking allows the other parsers in baseParsers to parse the remaining text that failed to be parsed as a valid Noweb link.
130 lets us parse <<<ok>>> as < followed by a Noweb link of ok. If we didn't have this, we would treat the initial << as starting the Noweb link and end up with <ok instead. See ti:0007-blocks-noweb-intraline.
The f:oidReferenceParser 131 parses information about the block being referenced, which we encode as a data:LinkOid 132. This is the same parser used for parsing OID references in prose text, as seen in OID references. The only difference is that we use double angle brackets instead of square brackets.
The use of f:oidReferenceParser means that we only need to specify an explicit OID if we're referring to blocks from other Org documents. If we are referring to blocks inside the current document, you can omit the id:... part for simplicity. Indeed, it is preferable to not use an explicit OID for blocks in the current document, because it becomes easier to tell at a glance (when reading the raw Org file) which blocks are from the current document and which ones are coming from elsewhere.
Below we check a typical case with some Noweb references in it. Note that we have to set #+header: :noweb off here because otherwise Orgmode will think <<foo>> and <<bar>> are Noweb references that exist in this doc, and because they don't exist, Lilac will treat it as broken links and abort tangling altogether.
#+begin_src sh
<<foo>>
<<quux
>>
echo <<bar>> x y z
#+end_srcThe <<quux is not interpreted as a Noweb reference, because it doesn't end with >> on the same line.
shouldParseWithState
"0007-blocks-noweb"
( nilOrgDoc
Expected blocks for 0007-blocks-noweb
[]
)
defaultOrgParserState
{ anonBlockNumber = 1
}[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "1"),
("__block_type", "src"),
("__src_language", "sh")
],
body =
[ BTokenLinkTarget
$ LinkTargetOid
LinkOid
{ oid = nilOid,
name = "foo"
},
BTokenString "\n\n<<quux\n>>\necho ", -- 133
BTokenLinkTarget
$ LinkTargetOid
LinkOid
{ oid = nilOid,
name = "bar"
},
BTokenString " x y z"
]
}
]We have to set #+header: :noweb off in Expected blocks for 0007-blocks-noweb as well, to avoid the <<quux\n>> bit 133 from becoming interpreted as a Noweb reference.
The test below checks the nowebOff boolean behavior from f:blockParser. That is, if Lilac sees #+header: :noweb off, it should not treat text that look like <<foo>> as a Noweb link, but as a literal <<foo>> string.
#+header: :noweb off
#+begin_src sh
<<foo>>
<<quux
>>
echo <<bar>> x y z
#+end_srcshouldParseWithState
"0007-blocks-noweb-off"
( nilOrgDoc
Expected blocks for 0007-blocks-noweb-off
[]
)
defaultOrgParserState
{ anonBlockNumber = 1
}[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "1"),
("__block_type", "src"),
("__src_language", "sh"),
("noweb", "off")
],
body =
[ BTokenString "<<foo>>\n\n<<quux\n>>\necho <<bar>> x y z"
]
}
]Because Lilac relies heavily on Noweb references, it will be turned on by default. So the only way to disable them would be to set #+header: :noweb off for each block where you'd want it disabled.
Sometimes we'll have Noweb references surrounded by punctuation. For example, see how we use <<GHC version>> in nix:ghcVersion. This is a typical use case when the referenced block only has 1 line of text in it and we want to inject those contents into an existing line.
#+begin_src sh
(<<foo>>) and "<<bar>>" and <<<ok>>> <<a->b>>
#+end_srcshouldParseWithState
"0007-blocks-noweb-intraline"
( nilOrgDoc
Expected blocks for 0007-blocks-noweb-intraline
[]
)
defaultOrgParserState
{ anonBlockNumber = 1
}[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__anon_block_number", "1"),
("__block_type", "src"),
("__src_language", "sh")
],
body =
[ BTokenString "(",
BTokenLinkTarget
$ LinkTargetOid
LinkOid
{ oid = nilOid,
name = "foo"
},
BTokenString ") and \"",
BTokenLinkTarget
$ LinkTargetOid
LinkOid
{ oid = nilOid,
name = "bar"
},
BTokenString "\" and <",
BTokenLinkTarget
$ LinkTargetOid
LinkOid
{ oid = nilOid,
name = "ok"
},
BTokenString "> ",
BTokenLinkTarget
$ LinkTargetOid
LinkOid
{ oid = nilOid,
name = "a->b"
}
]
}
]11.2.4 Pad consecutive Noweb links
When we eventually tangle blocks, we want to inject some padding (a newline) between Noweb links (assuming there are multiple consecutive Noweb links, one on each line) because each Noweb block is its own "unit" and it makes sense to add some separation between consecutive Noweb blocks. This way we get some visual separation in the final output.
We only want to add this padding for cases where there are consecutive Noweb links. For example, in
<<a>>
<<b>>
<<c>>
<<d>>
<<e>>
<<f>>
<<g>>only <<b>> <<c>> <<d>> and <<e>> <<f>> are consecutive. And in the final tangled output we would want to make it as if we had originally written
<<a>>
<<b>>
<<c>>
<<d>>
<<e>>
<<f>>
<<g>>This allows us to write in a bit more compact manner in the Org source document, while still giving each Noweb link (and associated block) some separation from the other blocks in the tangled output.
The way we do this automatic padding is by injecting newlines (padding) into the parsed block's body BToken values, as BTokenPadding. This is what we do by calling f:padConsecutiveNowebLinks in f:blockParser.
The high-level algorithm is as follows:
Concatenate the BToken values together (using the Text representation). For BTokenString values we use the inner text information as is. For BTokenLinkTarget we use the target, but also inject a special link-marker which uses an ASCII NUL character.
Split the contents by newline.
Group together the lines that have Noweb links. In other words, consecutive lines that have Noweb links are grouped together.
For a line to be considered for this group, it must end with a Noweb link; if that link is followed by any text, we leave it alone. This is because we only ever want to add newline padding to the end of a Noweb link, and if there is some other text immediately following a Noweb link, we'll end up introducing a newline in the middle of the line, which is almost never desirable.
Within each group, only mark the links at the end of the line (because there may be more than one link in a line) as needing a newline. By marking links as needing (or not needing) a newline, we can easily inject padding at the right spots in the next step.
We can't add padding for the very last line in the group though, because that would be superfluous (we want to add newlines between consecutive blocks, not after every consecutive block).
Using the information from the previous step, iterate through all BToken values and whenever we see a BTokenLinkTarget, inject the padding (if the link was marked as needing a newline).
The markers are just simple boolean values (one for each Noweb link found in the block), which denote whether or not the Noweb link needs padding.
Now let's look at the implementation in detail. Step 1 is done in f:getBlockContents.
getBlockContents :: [BToken] -> Text
getBlockContents blockBody = T.concat $ map bTokenToText blockBody
where
bTokenToText :: BToken -> Text
bTokenToText = \case
BTokenString text -> text
BTokenPadding -> ""
BTokenLinkTarget (LinkTargetOid _) ->
link-marker
BTokenLinkTarget {} -> ""The <<link-marker>> is just a special marker we use to keep track of where the links are inside the concatenated (single) Text value representing all of the block contents.
"\NUL_link"Steps 2, 3, and 4 are all done inside f:getLinkPaddingBools. This function returns a list of booleans that correspond to each of the Noweb links found. If True, this means the link needs a trailing newline (padding) added to it.
getLinkPaddingBools :: [BToken] -> [Bool]
getLinkPaddingBools blockBody =
concatMap (concatMap padLastLinkOnThisLine)
. map padLinesExceptLast -- 134
. groupBy consecutiveNowebLines
. T.lines
$ getBlockContents blockBody
where
consecutiveNowebLines :: Text -> Text -> Bool
consecutiveNowebLines a b =
(nowebLinkAtEndOfLine a)
&& (nowebLinkAtEndOfLine b) -- 135
&& (countLinks a > 0)
&& (countLinks b > 0)
padLinesExceptLast :: [Text] -> [(Bool, Text)]
padLinesExceptLast lines =
reverse
. (zip $ False : repeat True)
$ reverse lines
padLastLinkOnThisLine :: (Bool, Text) -> [Bool]
padLastLinkOnThisLine (padLinksInThisLine, line)
| padLinksInThisLine = case countLinks line of
1 -> [True]
n -> reverse $ True : replicate (n - 1) False -- 136
| otherwise = replicate (countLinks line) False
countLinks = T.count link-markerWe only pad the lines except for the last one 134, and padLastLinkOnThisLine 136 only pads the very last link found in the line. The call to f:nowebLinkAtEndOfLine 135 limits this padding to Noweb links that are at the end of the line; Noweb links that are followed by non-newline text is called an "embedded" link. See ti:0102-pad-between-consecutive-links-except-embedded for an example of the expected behavior here.
nowebLinkAtEndOfLine :: Text -> Bool
nowebLinkAtEndOfLine line = link-marker `T.isSuffixOf` lineFor the list of booleans we get from f:getLinkPaddingBools, all we have to do is go through all of the links in the block, adding a BTokenPaddding every time we see a True boolean (padLink).
padConsecutiveNowebLinks :: [BToken] -> [BToken]
padConsecutiveNowebLinks blockBody =
reverse
. fst
$ foldl' f ([], getLinkPaddingBools blockBody) blockBody
where
f (acc, bools) text@(BTokenString _) = (text : acc, bools)
f (acc, bools) BTokenPadding = (acc, bools)
f (acc, bools) link@(BTokenLinkTarget (LinkTargetLineLabel _)) =
(link : acc, bools)
f (acc, bools) linkTarget@(BTokenLinkTarget _) =
case bools of
(padLink : rest)
| padLink -> (BTokenPadding : linkTarget : acc, rest)
| otherwise -> (linkTarget : acc, rest)
[] -> (linkTarget : acc, [])See the test case inputs below for examples of this padding behavior.
11.3 Additional tests
A #+begin_src without an accompanying language should trigger an error.
#+begin_src
#+end_srcshouldFailWithMessage
"0007-block-src-without-lang"
"block type src requires a language specifier"
11Below we test the edge cases around detecting the end of a block.
#+name: empty
#+begin_src sh
#+end_src
#+name: end not in beginning of line
#+begin_src sh
#+end_src
#+end_src
#+name: end has no preceding blank line
#+begin_src sh
foo
#+end_src
#+name: end has preceding blank line
#+begin_src sh
foo
#+end_src
#+name: end has preceding newline (multiple)
#+begin_src sh
foo
#+end_src
#+name: ignore #+end_example because this is a src block
#+begin_src sh
#+end_example
#+end_srcshouldParseWithState
"0007-blocks-ending"
( nilOrgDoc
Expected blocks for 0007-blocks-ending
[]
)
defaultOrgParserState
{ cellNames =
Set.fromList
[ "empty",
"end has no preceding blank line",
"end has preceding blank line",
"end has preceding newline (multiple)",
"end not in beginning of line",
"ignore #+end_example because this is a src block"
]
}[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "sh"),
("name", "empty")
],
body = []
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "sh"),
("name", "end not in beginning of line")
],
body = [BTokenString " #+end_src"]
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "sh"),
("name", "end has no preceding blank line")
],
body = [BTokenString "foo"]
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "sh"),
("name", "end has preceding blank line")
],
body = [BTokenString "foo\n"]
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "sh"),
("name", "end has preceding newline (multiple)")
],
body = [BTokenString "foo\n\n"]
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "sh"),
("name", "ignore #+end_example because this is a src block")
],
body = [BTokenString "#+end_example"]
}
]12 List items
Lists are similar to headings, because they give you a way to structure your paragraphs.
Lists are made up of list items. A list item is made up of a list head and list body. List heads encode the structure of cells that sit in the list bodies. So list items encode both structure (with list heads) and content (with list bodies). However unlike headings, a list item can have multiple other cells (typically paragraphs) in its body, whereas a heading can only have a single line of prose. Consider the following list:
1) a
b
- c
d
2) eHere the 1) and 2) are list heads for an ordered list. The single dash - before "c" is also a list head, but for an unordered list. The list body for 1) includes the paragraphs a, b, the nested list item c, and the paragraph d. This is another difference versus headings. That is, a list item can have content for it after backing out of a nested list item. To reiterate, in the list shown above the d paragraph is part of the ordered list item 1) even though the nested item c is just above it; this sort of structure is not allowed for headings. For example, in the following situation
* a
** banything that comes after ** b will be part of the ** b heading; there is no way to make it belong to * a (any such content must be placed before ** b). List items are not restricted like this, because they always have meaningful indentation (a list head must be preceded by indentation), and this meaning of each indentation level is tracked during the parse.
List bodies may have cells of the following type:
Example paragraph.
Table 1. Example Table 1 2 and of course, more list items (nesting)
Example nested item.
Vanilla Org allows more cell types like code blocks inside list bodies, but Lilac does not. List items should be simple, and things like code blocks just makes list items more complicated. The fact that we support tables and standalone math fragments in list bodies is an accident, not a product of design. Indeed, for typical use cases, you shouldn't need anything more than paragraphs and other list items inside a list.
Below we have a look at some more detailed rules around how lists may be written in an Org file. They are somewhat numerous because of all the edge cases we have to account for.
12.1 Ordered list items
Below are some detailed rules around ordered list items, themselves expressed as ordered list items with links to test cases as examples. The discussions around space characters and indentation only make sense when viewed in the raw Org file, not the woven HTML format.
List items start out with 2-space indentation. This makes them easier to read from the raw Org file. The amount of additional indentation (in order to have nested list items) depends on where the text began in the parent item.
List items must be separated from each other with an empty blank line. This makes them easier to read when reading the Org file in plain text.
Additional hard-wrapped lines for a list item must be aligned with the text of the first line.
List items can have more than one paragraph associated with them.
This is an example of another paragraph which is still part of same list item, and has 5 spaces of indentation.
List items can be nested.
This is a nested item, underneath "5)". It has 7 spaces of initial indentation (5 needed to line up with the column where the "L" is in "5) List" in the parent item, plus another 2 to indent one more level beyond that). Subsequent lines of this paragraph have 10 spaces of indentation to line up with the text above it.
See:
Unlike Markdown, the numbers of the list item must increase by one if there are additional items at the same level. And the first number of a level must start from 1.
The format for ordered list items requires a closing parenthesis immediately after the number. So 1) is valid, but 1. is invalid.
Only numbers may be used to denote item position. You cannot use letters or Roman numerals like a) or i).
There is no limit to the number of consecutive list items at the same level (technically, we use an Int type to encode the number, so it will eventually max out, but you'd need billions of items to hit this limit).
The indentation of text changes slightly depending on the number of the item. For example, this line has 6 spaces of indentation, unlike the paragraphs in item "4)" above, which uses 5 spaces.
Similarly, this child item has text which has 11 spaces of indentation, unlike the 10 spaces of indentation in the ordered child item under item "5)" above.
See:
There can only be up to 2 additional levels of nesting (for a total of 3 levels of items). This is because excessive nesting is an anti-pattern. See the lilac_maxListLevel Lilac extension for customizing this behavior.
Ordered list items may have unordered list items as children, and vice versa. However you may not mix the two types of items at the same level.
A list item may include a table. However the table cannot be on the same line as the list item head. Below is an example of such a table:
Table 2. foo bar x y Similarly, a list item may include mathematical equations. However, such an equation must be on a separate line from the list item head, just like for tables (see Rule 13 above).
The equation above (and this paragraph) are still part of Rule 14.
Lists may be empty. For example, the following child list is valid even though it does not have any content:
See:
12.2 Unordered list items
Unordered list items use a single dash character -, and always use 2-space indentation (if you are only using unordered list items for nesting). Below is an example:
- one
- two
- foo
- bar
- threeHere's an example of mixing ordered and unordered lists:
- foo
- bar
1) one
2) two
- baz
- quux
Note how this paragraph is under item "2", because it is indented the
same way as "two", not "quux", above.
3) three
- dogYou may mix ordered and unordered lists at the same indentation level, as long as they are not consecutive (you need some intervening cells, such as the "quux" paragraph below):
- foo
1) one
2) two
quux
- a
- b12.3 Indentation and ListMarker
Parsing list items is tricky because of meaningful indentation (whitespace). Keeping track of the current level of indentation requires additional state during parsing, which we encode with Level (used by listLevels 29 in data:OrgParserState).
type Level :: Type
data Level = Level
{ indent :: Int,
number :: Maybe Int
}
deriving stock (Eq, Show)This keeps track of the indentations and the last seen number (only for ordered lists) of each (nested) level of indentation thus far during parsing. Whenever we see a list head like 1) or -, we record the number used as well as the indentations we expect for the list head and list body. The main point of listLevels is to ensure that the input is correctly formatted, as well as to detect whenever we start nesting an additional level, or if we back out (finish) one or more levels of nesting (when we see dedented text).
As we observe how these listLevels change during parsing, we create "start" and "end" markers for list items at a particular level, which are the values of the data:ListMarker type. The ListMarker objects are parsed as their own cell as a CListItem (see data:Cell).
type ListMarker :: Type
data ListMarker
= LOrderedStart
| LOrderedEnd
| LUnorderedStart
| LUnorderedEnd
| LItemStart
| LItemEnd
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)We take this model from HTML, which uses <ol> and </ol> to encode the start and end of ordered lists (<ul> and </ul> for unordered lists), and <li> and </li> to mark the start and end of a single list item.
| ListMarker | HTML equivalent |
|---|---|
| LOrderedStart | <ol> |
| LOrderedEnd | </ol> |
| LUnorderedStart | <ul> |
| LUnorderedEnd | </ul> |
| LItemStart | <li> |
| LItemEnd | </li> |
In summary when we parse a list item, we parse it as a combination of CListItem and other cell types for the list structure and content, respectively. This lets us parse things into a single global list of cells in f:cellParser, without a need to nest them inside a recursive data structure (such as in https://markkarpov.com/tutorial/megaparsec.html#nested-indented-list).
12.4 Parsing initial indentation to close list items
Before getting into parsing the list items and taking into account all of the rules, let's first consider case where we just want to close off the list items, as this is simpler than creating list items. We want to do this every time we parse initial indentation where there is no list head. For example, in the following
1) a
- b
cthe indentation for paragraph c should by itself close off the preceding unordered list item (- b). We can close off existing list items depending on the amount of indentation.
If there is no indentation, we close off all list items.
If there is maximum (already-seen) indentation (to line up with the most-nested list item), we don't close off any list items because we're just continuing the most-nested list item.
Otherwise, we have some amount of indentation and we only want to close off some list levels, depending on how much indentation we detect.
If there is more indentation than the maximum, then we are nesting another level and don't need to close off anything. That's why this situation is not listed above.
Point 1 is captured by zeroIndent in f:listItemsEndParser. Points 2 and 3 are captured by someIndent in the same function.
zeroIndent :: OrgParser (Int, Int)
zeroIndent = do
pState <- get
_ <- MP.eof <|> MP.notFollowedBy (char ' ')
let listLevelsToClose = length pState.listLevels
continuationIndent = 0 :: Int
pure (listLevelsToClose, continuationIndent)In f:zeroIndent we return the number of list levels to close off (in this case all levels that we've detected so far), and the continuation indent. The continuation indent is used by other parsers such as the f:paragraphParser to see how to join up multiple lines into the same cell, assuming those lines have the same continuation indent (the lines continue if it's the same indent, hence the name). For the case of f:zeroIndent, we set the continuation indent to 0 because there is no indentation, and any subsequent lines belonging to the same cell should not have any indentation (it is considered invalid input if we see surprise indentation; see t:0005-paragraph-indentation-mismatch-over).
For f:someIndent, we just try to line up the amount of indentation found with the existing list levels we have and how much indentation they expect.
someIndent :: OrgParser (Int, Int)
someIndent = do
pState <- get
indentFound <- T.length <$> onlySpaces
let levelsToDropPerIndentationIncrement :: [(Int, Int)]
levelsToDropPerIndentationIncrement =
zip [0 ..] -- 137
. map getContinuationIndentForLevel
$ pState.listLevels -- 138
matchingIndent =
find
(\(_, continuationIndent) -> indentFound == continuationIndent)
levelsToDropPerIndentationIncrement
case matchingIndent of
Nothing -> fail "indentation does not fit into existing levels"
Just x@(_levelsToDrop, _continuationIndent) -> pure xThe pState.listLevels 138 is grown by prepending to the list in f:listItemNester. So the most-deeply-nested level will be at the head of the list in pState.listLevels. That's why we use zip [0 ..] 137 --- we want to drop 0 levels if we see indentation matching the most deeply-nested list item.
The f:getContinuationIndentForLevel computes the total indentation for a list level. For example, in
1) foowe compute the initial 2 spaces before the 1), the 1) itself (two characters), plus a space character, or a total continuation indent of 5. The dynamic portion here is the number, which will grow as the list grows (e.g., 10) will have a total continuation indent of 6), which is why we use a function here instead of a constant.
getContinuationIndentForLevel :: Level -> Int
getContinuationIndentForLevel level =
level.indent + maybeNum + punctuation + space
where
punctuation :: Int
punctuation = 1
space :: Int
space = 1
maybeNum :: Int
maybeNum = case level.number of
Just n -> length $ (show n :: String)
Nothing -> 0Once we determine the amount of levels we need to close off (by parsing f:zeroIndent or f:someIndent), we just go through the list markers we've already seen in listMarkers and close them off via f:getEndingListMarkers. This is what we do in f:listItemsEndParser.
listItemsEndParser :: OrgParser Cell
listItemsEndParser = do
pState <- get
when (null pState.listMarkers) $ do
fail "no pending ListMarker objects to close off" -- 139
when (pState.continuationIndent /= 0) $ do
fail "not at beginning of line"
(levelsToDrop, continuationIndent) <- zeroIndent <|> someIndent
let markers = getEndingListMarkers levelsToDrop pState.listLevels
markersToDrop = levelsToDrop * 2 -- 140
put
pState
{ listLevels = drop levelsToDrop pState.listLevels,
listMarkers = drop markersToDrop pState.listMarkers,
continuationIndent = continuationIndent
}
pure $ CListItem markersIn f:listItemsEndParser, we initially fail the parser if there are no pending ListMarker objects (no list levels to close off), or if we're not at the beginning of the line (because otherwise we cannot measure indentation accurately), by checking pState.continuationIndent. This check of pState.continuationIndent is required because there is no easy way to check for the just-parsed token (or character) in Megaparsec --- all parsers only see what's in front of it. So then it's up to other parsers to set this to some value greater than 1 (we do this in f:listHeadIndentLevelParser which is part of f:listItemStartParser)
The markersToDrop 140 is always twice the number of levelsToDrop because we always add 2 (LItemStart plus either LOrderedStart or LUnorderedStart) for each level (see f:listItemNester).
If we detect the maximum allowed indentation, then there are no levels to drop, and hence, we'll end up returning CListItem [] which doesn't hold any meaning. Avoiding this is a bit difficult; if we just fail the parser outright, then the amount of parsed indentation will have to be parsed by another downstream parser (this can get messy). So instead we just return CListItem [] and then filter these out in f:orgDocParser. This way we decide once and for all how to deal with leading indentation (and how this closes off existing list items) in f:listItemsEndParser.
The f:getEndingListMarkers function understands how to close off list levels based on whether the level uses an ordered or unordered list item.
getEndingListMarkers :: Int -> [Level] -> [ListMarker]
getEndingListMarkers levelsToDrop = concatMap f . take levelsToDrop
where
f :: Level -> [ListMarker]
f level = case level.number of
Nothing -> [LItemEnd, LUnorderedEnd]
Just _ -> [LItemEnd, LOrderedEnd]12.5 Parsing list heads
In the previous section we considered how to close off list items based purely on indentation. Parsing list heads is similar (because list heads can also close off list items), but more complicated because we also have to take into account how to create list items (list levels). This means we have to deal with all of the possible values of data:ListMarker here.
Parsing the list head (and therefore the data:Level of a list item) is straightforward. For unordered lists we just want to see a dash, returning Nothing 141 because there is no list number.
For ordered lists we want to see the number and closing parenthesis.
orderedHeadParser :: OrgParser (Maybe Int)
orderedHeadParser = do
num <- MP.takeWhile1P (Just "ordered list number") isDigit
_ <- char ')'
let n = readMaybe (T.unpack num) :: Maybe Int
when (isNothing n) $ do
fail $ "could not parse " <> show (T.unpack num) <> " as Int"
pure nWe check if the number looks like an Int with readMaybe.
For both cases we also parse leading indentation and also an additional space at the end to separate the list head from the body. This is what f:listHeadIndentLevelParser does.
listHeadIndentLevelParser :: OrgParser Level
listHeadIndentLevelParser = do
minimalIndentation -- 142
spaces <- onlySpaces
listHead <- unorderedHeadParser <|> orderedHeadParser
let level =
Level
{ indent = T.length spaces,
number = listHead
}
afterListHead level
pure level
where
minimalIndentation :: OrgParser ()
minimalIndentation =
void
. MP.try
. MP.lookAhead
. MP.label "minimal indentation"
$ MP.count 2 (char ' ')
afterListHead :: Level -> OrgParser () -- 143
afterListHead level =
void
$ MP.choice
[ MP.eof,
void $ string "\n",
void $ seeingContentAfterListHead level
]
seeingContentAfterListHead :: Level -> OrgParser ()
seeingContentAfterListHead level = do
_ <- char ' '
MP.notFollowedBy (char ' ')
modify
(\s -> s {continuationIndent = getContinuationIndentForLevel level})The minimalIndentation 142 expects there to be at least 2 spaces preceding a list item. This follows Rule 1 from Ordered list items.
The afterListHead parser 143 preemptively checks what's in the input after parsing the list head. In the case where it sees some other non-empty content, it expects some other parser to continue parsing. So inside seeingContentAfterListHead we set the continuationIndent attribute to indentation of the current level, because we expect another parser to help us finish parsing the line. Then at the f:cellParser level, we set this back to 0 for any parser that is expected to consume content and end with a newline. This lets f:listItemsEndParser know whether it is looking at EOF or at the beginning of a line when it inspects continuationIndent. If Megaparsec can let us easily check the previous character that was parsed, this scheme of using a stateful continuationIndent field wouldn't be necessary.
Note, however, that f:listHeadIndentLevelParser only checks for textual correctness. It doesn't check whether what's parsed is semantically correct. For example, we wouldn't want to blindly parse the number of spaces in the indent (instead we would want to check which indents are expected, given any previous list items). Similarly we would want to check whether the list head number (for ordered lists) is the expected (incremented) number from the previously-seen list head number, if any (or if there is no previous number, that this number is 1).
We have to perform these additional checks by comparing the output of f:listHeadIndentLevelParser against the existing data:OrgParserState (listLevels). This is what we do in f:listItemStartParser below, which wraps around f:listHeadIndentLevelParser 145.
listItemStartParser :: OrgParser Cell
listItemStartParser = do
_ <- MP.try $ MP.lookAhead listHeadIndentLevelParser -- 144
pState <- get
let levels = pState.listLevels
level <- listHeadIndentLevelParser -- 145
listMarkers <-
if nestsBeyond levels level
then listItemNester level levels
else listItemFitter level levels
pure $ CListItem listMarkers
where
nestsBeyond :: [Level] -> Level -> Bool -- 146
nestsBeyond [] _ = True
nestsBeyond (deepestLevel : _) level =
getContinuationIndentForLevel deepestLevel + 2 == level.indentThe use of MP.lookAhead 144 is needed because otherwise as soon as Megaparsec sees any indentation (as part of invoking f:listHeadIndentLevelParser) it will expect to see a list head; obviously this is not always the case as not all indentation leads to a list head. This is why we have to perform this look-ahead preemptively.
The nestsBeyond function 146 just checks whether the indentation we found in the (just-parsed) level is the same as the continuation indent of the most-deeply-nested level we have so far, plus 2 extra spaces. This is how we nest one level deeper than the current level. If this is the case, we delegate to f:listItemNester. Otherwise the list item is merely continuing some existing list, for which we delegate to f:listItemFitter.
The use of 2 extra spaces is necessary because a list item must always be indented 2 spaces relative to the just-preceding paragraph.
12.6 Nesting the list item beyond the current level
f:listItemNester prepends starting markers to listMarkers. We use this information in other places when we want to know how to un-nest an item (close it off), such as in f:listItemsEndParser where we add corresponding closing markers with f:getEndingListMarkers. The other place we use listMarkers is in f:listItemFitter.
listItemNester :: Level -> [Level] -> OrgParser [ListMarker]
listItemNester level levels = do
case level.number of
Just 1 -> pure ()
Just n -> fail $ "starting list head number must be 1; got " <> show n
Nothing -> pure ()
pState <- get
let newMarker = case level.number of
Just _ -> LOrderedStart
Nothing -> LUnorderedStart
opc = pState.orgParserConf
when (opc.maxListLevel > 0 && length levels == opc.maxListLevel) $ do
fail
$ "max nesting depth exceeded: cannot have more than "
<> show opc.maxListLevel
<> " levels of list items"
put
pState -- 147
{ listLevels = level : levels,
listMarkers = LItemStart : newMarker : pState.listMarkers
}
pure [newMarker, LItemStart]We enforce some additional rules here, such as forcing the nesting list head to start off with 1 (see t:0009-ordered-lists-start-from-1) if it is an ordered list. The rest of the function just updates the pState to have the additional level and also grows the listMarkers accordingly 147.
12.7 Fitting the list item into an existing level
The goal of f:listItemFitter is twofold:
continue an existing list, and
close off (finish) pending levels which are still open.
The first task is easy --- we just append [LItemEnd, LItemStart] to signal that we are closing off the existing list item (at the same level) to start a new list item (also at the same level).
The second task is a bit more involved. We have to "close" any existing levels that are more deeply nested, just like in f:listItemsEndParser. For example, in the following list
1) foo
- bar
1) baz
2) quuxwhen we hit quux we have to close off the list environments for baz, then bar. However, unlike f:listItemsEndParser, we have to continue the existing list level that we fit into. For ordered list items, we have to drop an extra level in listLevels in the fitter function 148, and then replace it with the (incremented) level to continue it.
listItemFitter :: Level -> [Level] -> OrgParser [ListMarker]
listItemFitter level levels =
case find (\(_, l) -> l.indent == level.indent) (zip [0 ..] levels) of
Nothing -> fail "list item: invalid indentation"
Just (levelsToDrop, lvl)
| isJust level.number && isJust lvl.number ->
let expectedNum = fmap (+ 1) lvl.number
in if level.number == expectedNum
then fitter levelsToDrop
else
fail
$ "expected incremented list head "
<> (show . fromJust $ expectedNum)
<> " but got "
<> (show . fromJust $ level.number)
| isNothing level.number && isNothing lvl.number -> fitter levelsToDrop
| isJust lvl.number -> fail "expected ORDERED list item at this level"
| otherwise -> fail "expected UNORDERED list item at this level"
where
fitter :: Int -> OrgParser [ListMarker]
fitter levelsToDrop = do
pState <- get
let markers = getEndingListMarkers levelsToDrop pState.listLevels
put
pState
{ listLevels =
level : drop (levelsToDrop + 1) pState.listLevels, -- 148
listMarkers = drop (levelsToDrop * 2) pState.listMarkers
}
pure $ markers <> [LItemEnd, LItemStart]We use f:listItemStartParser and f:listItemsEndParser inside f:cellParser; the former identifies the beginnings of list items, while the latter recognizes their (complete) end when there is no more indentation.
12.8 Tests
There are so many rules associated with ordered lists; we try to check a few of them here.
1)
2)shouldParseWith
"0009-list-indentation-no-indent"
$ nilOrgDoc
[ plainParagraph "1)",
plainParagraph "2)"
]
[]The list-like text is parsed as plain paragraphs in t:0009-list-indentation-no-indent, because the lack of indentation means we don't recognize the 1) and 2) as list items.
Here's a list with the (proper) 2-space indent.
1)
2)shouldParseWith
"0009-list-indentation-proper-indent"
$ nilOrgDoc
[ CListItem
[ LOrderedStart,
LItemStart
],
CListItem
[ LItemEnd,
LItemStart
],
CListItem
[ LItemEnd,
LOrderedEnd
]
]
[]Ordered lists require a closing parenthesis ) after the number.
1.
2.shouldFailWithError
"0009-ordered-list-head-needs-parenthesis"
(utoks "." <> etoks ")" <> elabel "ordered list number")
3Ordered lists may only use numbers, not letters or Roman numerals.
a)shouldFailWithMessage
"0009-ordered-list-head-cannot-use-letters"
"invalid indentation"
2Here's a nested example.
1)
-
-
2)
1)
-
3)shouldParseWith
"0009-list-indentation-nested"
$ nilOrgDoc
[ CListItem
[ LOrderedStart, ---- Start outer ordered list.
LItemStart -------- Start outer "1)".
],
CListItem
[ LUnorderedStart, -- Start inner bullets under "1)".
LItemStart -------- Inner bullet start.
],
CListItem
[ LItemEnd, --------- Inner bullet end.
LItemStart -------- Inner bullet start.
],
CListItem
[ LItemEnd, --------- Inner bullet end.
LUnorderedEnd, ---- End of inner bullets under "1)".
LItemEnd, --------- End of outer "1)".
LItemStart -------- Start of outer "2)".
],
CListItem
[ LOrderedStart, ---- Start of nested ordered list.
LItemStart -------- Start of nested "1)".
],
CListItem
[ LUnorderedStart, -- Start deeply nested bullets.
LItemStart -------- Deeply nested bullet start.
],
CListItem
[ LItemEnd, --------- Deeply nested bullet end.
LUnorderedEnd, ---- End of deeply nested bullets.
LItemEnd, --------- End of nested "1)".
LOrderedEnd, ------ End of nested ordered list.
LItemEnd, --------- End of outer "2)".
LItemStart -------- Start of outer "3)".
],
CListItem
[ LItemEnd, --------- End of outer "3)".
LOrderedEnd ------- End outer ordered list.
]
]
[]Nesting is limited to 2 additional levels, for a total of 3 levels by default.
- 1
- 2
- 3
- 4shouldFailWithMessage
"0009-list-nesting-max"
"max nesting depth exceeded: cannot have more than 3 levels of list items"
49You can change the maximum nesting behavior with lilac_maxListLevel Lilac extension. In the test below we increase the total list level to 9.
#+lilac_maxListLevel: 8
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9shouldFailWithMessage
"0009-list-nesting-max-8"
"max nesting depth exceeded: cannot have more than 8 levels of list items"
228Notice how the parser failed only after parsing the last level which exceeded the 8 levels allowed.
We can prohibit list items entirely, much like how we can prohibit them in t:0004-headings-prohibited.
#+lilac_maxListLevel: -1
- 1shouldFailWithMessage
"0009-list-prohibited"
"lists prohibited because lilac_maxListLevel < 0"
29Now we test whether list items can also hold content (other cells) inside it, in this case paragraph cells.
1) First
paragraph.
Second paragraph.
2) Another paragraph.shouldParseWith
"0009-list-content-simple"
$ nilOrgDoc
[ CListItem
[ LOrderedStart,
LItemStart
],
plainParagraph "First paragraph.",
plainParagraph "Second paragraph.",
CListItem
[ LItemEnd,
LItemStart
],
plainParagraph "Another paragraph.",
CListItem
[ LItemEnd,
LOrderedEnd
]
]
[]Here's a more involved example using paragraph text for list items, following the pattern from ti:0009-list-indentation-nested. Note how the parsed list structure is still there, except now we have CParagraph cells interspersed in between them.
Foo.
1) one
- a
- b
2) two
1) two-point-one
hello
- X
3) three
Bar.shouldParseWith
"0009-list-indentation-nested-with-text"
$ nilOrgDoc
[ plainParagraph "Foo.",
CListItem
[ LOrderedStart, ---- Start outer ordered list.
LItemStart -------- Start outer "1)".
],
plainParagraph "one",
CListItem
[ LUnorderedStart, -- Start inner bullets under "1)".
LItemStart -------- Inner bullet start.
],
plainParagraph "a",
CListItem
[ LItemEnd, --------- Inner bullet end.
LItemStart -------- Inner bullet start.
],
plainParagraph "b",
CListItem
[ LItemEnd, --------- Inner bullet end.
LUnorderedEnd, ---- End of inner bullets under "1)".
LItemEnd, --------- End of outer "1)".
LItemStart -------- Start of outer "2)".
],
plainParagraph "two",
CListItem
[ LOrderedStart, ---- Start of nested ordered list.
LItemStart -------- Start of nested "1)".
],
plainParagraph "two-point-one",
plainParagraph "hello",
CListItem
[ LUnorderedStart, -- Start deeply nested bullets.
LItemStart -------- Deeply nested bullet start.
],
plainParagraph "X",
CListItem
[ LItemEnd, --------- Deeply nested bullet end.
LUnorderedEnd, ---- End of deeply nested bullets.
LItemEnd, --------- End of nested "1)".
LOrderedEnd, ------ End of nested ordered list.
LItemEnd, --------- End of outer "2)".
LItemStart -------- Start of outer "3)".
],
plainParagraph "three",
CListItem
[ LItemEnd, --------- End of outer "3)".
LOrderedEnd ------- End outer ordered list.
],
plainParagraph "Bar."
]
[]Check whether we can detect invalid indentations for paragraphs that are indented (indented due to them being inside a list). The first case is when the line is indented too much in the list body.
1) Foo
bar
baz.shouldFailWithError
"0005-paragraph-indentation-mismatch-over-list-body"
(utoks " ")
14The same rule about keeping the same indentation applies for plain paragraphs outside of list environments.
Foo
bar
baz.shouldFailWithError
"0005-paragraph-indentation-mismatch-over"
(utoks " ")
4Under-indentation is also a problem for continuation lines inside a single paragraph block.
1) Foo
bar
baz.shouldFailWithError
"0005-paragraph-indentation-mismatch-under-list-body"
(utoks "b" <> etoks " ")
12If we have some number of list levels already, then the paragraph must fit into one of those levels (it cannot have some arbitrary level of indentation) if there is non-zero indentation.
1) Foo
bar.
Invalid indentation (this paragraph).shouldFailWithMessage
"0005-paragraph-indentation-mismatch-initial-indent-invalid"
"indentation does not fit into existing levels"
21 1) Foo
bar.
- Baz.
Invalid indentation (this paragraph).shouldFailWithMessage
"0005-paragraph-indentation-mismatch-initial-indent-invalid-between-levels"
"indentation does not fit into existing levels"
38Paragraphs should be able to close list environments, depending on their indentation. For example, in the test below the paragraph Three should be able to close off the unordered list items.
1) One
Two
x
- Bar
- Baz
Three
a b
c
FourshouldParseWith
"0009-paragraph-can-close-deeper-levels"
$ nilOrgDoc
[ CListItem
[ LOrderedStart,
LItemStart
],
plainParagraph "One",
plainParagraph "Two x",
CListItem
[ LUnorderedStart,
LItemStart
],
plainParagraph "Bar",
CListItem
[ LUnorderedStart,
LItemStart
],
plainParagraph "Baz",
CListItem
[ LItemEnd,
LUnorderedEnd,
LItemEnd,
LUnorderedEnd
],
plainParagraph "Three a b c",
CListItem
[ LItemEnd,
LOrderedEnd
],
plainParagraph "Four"
]
[]Paragraphs can close off list items, so we can mix ordered and unordered list items at the same level, as long as the differing list type is closed off by a paragraph. In the test below, the Two closes off the unordered list, so we're allowed to start a new ordered list with 1) a.
1) One
- Bar
- Baz
Two
1) a
2) b
Three
FourshouldParseWith
"0009-paragraph-can-separate-mixed-lists"
$ nilOrgDoc
[ CListItem
[ LOrderedStart,
LItemStart
],
plainParagraph "One",
CListItem
[ LUnorderedStart,
LItemStart
],
plainParagraph "Bar",
CListItem
[ LUnorderedStart,
LItemStart
],
plainParagraph "Baz",
CListItem
[ LItemEnd,
LUnorderedEnd,
LItemEnd,
LUnorderedEnd
],
plainParagraph "Two",
CListItem
[ LOrderedStart,
LItemStart
],
plainParagraph "a",
CListItem
[ LItemEnd,
LItemStart
],
plainParagraph "b",
CListItem
[ LItemEnd,
LOrderedEnd
],
plainParagraph "Three",
CListItem
[ LItemEnd,
LOrderedEnd
],
plainParagraph "Four"
]
[]It's an error if there is no separation between list types (if we suddenly try to change list types at the same level). In the test below, the 1) c is unexpected because the unordered list has not been closed off yet.
- a
- b
1) cshouldFailWithMessage
"0009-lists-cannot-be-mixed-on-their-own"
"expected UNORDERED list item at this level"
23The same applies for going the other way (trying to fit an unordered list into an ordered one at the same level).
1) a
1) b
- cshouldFailWithMessage
"0009-lists-cannot-be-mixed-on-their-own-ordered"
"expected ORDERED list item at this level"
25List items have to be precisely indented; otherwise they'll hit an indentation error. In the test below, the 1) for the b item has only 4 spaces of indentation instead of 5 (to line up with the a above it), so that should trigger an indentation error.
1) a
1) bshouldFailWithMessage
"0009-lists-require-precise-indentation"
"list item: invalid indentation"
15List items must be separated by at least one blank line. In the test case below, there is not enough space so the 1) b gets interpreted as regular paragraph text, as part of the a above it.
1) a
1) bshouldParseWith
"0009-list-items-need-blank-line-separator"
$ nilOrgDoc
[ CListItem
[ LOrderedStart,
LItemStart
],
plainParagraph "a 1) b",
CListItem
[ LItemEnd,
LOrderedEnd
]
]
[]Ordered list items at the same level must increment by 1 each time. In the test case below, the 1) b will fail to parse because it should be 2) b instead.
1) a
1) bshouldFailWithMessage
"0009-ordered-lists-require-increment"
"expected incremented list head 2 but got 1"
13Ordered list items must start from 1.
1) a
20) bshouldFailWithMessage
"0009-ordered-lists-start-from-1"
"starting list head number must be 1; got 20"
1913 Description lists, aka glossary terms
Description lists are unordered list items that have a term followed by two colons (::), and a prose description of that term. Structurally they are quite similar to footnote definitions, which also have a leading name followed by the body (see data:Footnote).
The term "description list" comes from HTML, and it would appear that Orgmode borrowed the idea quite directly. For example, vanilla Orgmode will export the following Org syntax
- foo /hello/ :: bar
- baz :: quuxto HTML as
<dl class="org-dl">
<dt>foo <i>hello</i></dt><dd>bar</dd>
<dt>baz</dt><dd>quux</dd>
</dl>In HTML it is perfectly valid to have multiple <dt> (term) elements, or multiple <dd> (definition) elements, or both.
There are three differences worth noting for Lilac:
Only one term and one description is allowed. If multiple terms mean the same thing, there should be multiple entries and all but one should redirect the user to look up the canonical entry that has the full definition. Multiple definitions for a single term are not allowed. Instead, you should use terms like foo (1) :: ... and foo (2) :: ... to define two different definitions for the same term.
We don't call these description lists because that language is too close to the implementation detail. Instead we call them "glossary terms" (or simply "terms") because that's what they are used for most of the time.
Lilac treats each key-value pair as a single data:Cell, so you can write them anywhere a cell is expected. So whereas Orgmode only allows glossary terms them inside unordered lists, in Lilac you can write them as their own independent paragraphs, or in ordered lists, etc.
This way Lilac can mix and match glossary terms with other items (it does not need the Orgmode workarounds described here). See how glossary terms are woven to see how this plays out during HTML export.
With those things out of the way, let's look at data:GlossaryTerm, our take on definition list entries:
type GlossaryTerm :: Type
data GlossaryTerm = GlossaryTerm
{ term :: Prose,
definition :: Prose
}
deriving stock (Eq, Ord, Generic, Show)
deriving anyclass (FromJSON, ToJSON)Glossary terms are created from newtype:Prose in f:convertToGlossaryTerm. A paragraph is checked if it looks like a glossary term entry in f:paragraphParser, and delegates to f:convertToGlossaryTerm if so.
convertToGlossaryTerm :: Prose -> Int -> OrgParser Cell
convertToGlossaryTerm prose offset =
toGlossaryTerm
$ foldl' f ([], Prose [], []) prose.unProse
where
f (termPTokens, term'@(Prose finalTermPTokens), def) pToken
| finalTermPTokens == [] =
if isGlossaryTermDelimiter pToken -- 149
then
( termPTokens,
Prose . reverse $ dropWhitespacePToken termPTokens,
def
)
else (pToken : termPTokens, term', def)
| otherwise = (termPTokens, term', pToken : def)
dropWhitespacePToken :: [PToken] -> [PToken] -- 150
dropWhitespacePToken [] = []
dropWhitespacePToken tokens@(pToken : rest)
| (PTokenStyledText styledText) <- pToken,
Just (_, ' ') <- T.unsnoc styledText.body =
rest
| otherwise = tokens
toGlossaryTerm :: ([PToken], Prose, [PToken]) -> OrgParser Cell
toGlossaryTerm (_, term', def) = do
let proseDefinition = compressProse . Prose $ reverse def
assertUniqueGlossaryTerm (toText term') offset -- 151
pure
$ CGlossaryTerm
GlossaryTerm
{ term = term',
definition = proseDefinition
}150 removes any whitespace token just before the :: delimiter. This is needed because otherwise the term string will end with trailing whitespace.
f:convertToGlossaryTerm just checks each PToken in prose if it looks like a glossary term delimiter via f:isGlossaryTermDelimiter 149. We have to check each PToken because we cannot tell what the term looks like; for example it could be that the term is 15 words long before we finally see the :: delimiter.
isGlossaryTermDelimiter :: PToken -> Bool
isGlossaryTermDelimiter = \case
PTokenStyledText styledText ->
styledText.body == "::" && maskToStyles styledText.style == []
_ -> FalseThe important bit here is the check of whether no styles are on when encountering the string ::. This way, we don't interpret strings like
Foo ~a::b~ bar.as a glossary term because the :: will be inside the code (~) delimiters.
We require all glossary terms to be unique within the document in f:assertUniqueGlossaryTerm 151.
assertUniqueGlossaryTerm :: Text -> Int -> OrgParser ()
assertUniqueGlossaryTerm term offset = do
pState <- get
when (member term pState.glossary) $ do
MP.setOffset offset -- 152
fail -- 153
$ "glossary term "
<> show term
<> " is not unique"
put
pState
{ OrgParserState.glossary =
insert
term
pState.cellNumber
pState.glossary
}foo :: 1
foo :: 2shouldFailWithMessage
"0015-glossary-duplicate-term"
"glossary term \"foo\" is not unique"
14Originally we naively set the 152 to be the beginning of the paragraph (as seen by f:paragraphParser). However, this had the side effect of making Megaparsec think that we had not consumed any input when failing at 153. So then Megaparsec would also report other parser errors for those parsers that also did not consume any input and still signaled a failure in f:cellParser, namely in f:listItemsEndParser (139). In order to suppress that error, we changed f:textParser to set a glossaryTermOffset if it detected the glossary term delimiter (71)3.
13.1 Printing glossaries
Similar to Printing bibliographies, we would like to allow users to print a collection of all glossary terms found in the document (with links back to where the term was originally defined --- for added context). We use the line #+print_glossary: local for this. The local just means printing the glossary terms found in the current document. You can change this to be global to print all glossary terms found in the entire data:Archive (e.g., see Glossary (Global)).
glossaryLocationParser :: OrgParser Cell
glossaryLocationParser = do
_ <- string "#+print_glossary: "
isGlobal <-
MP.choice
[ string "local" >> pure False,
string "global" >> pure True
]
pure . CCommand $ PrintGlossary isGlobal#+print_glossary: localshouldParseWithState
"0015-glossary-location"
OrgDoc
{ oid = nilOid,
cells = [CCommand $ PrintGlossary False],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState#+print_glossary: globalshouldParseWithState
"0015-glossary-location-global"
OrgDoc
{ oid = nilOid,
cells = [CCommand $ PrintGlossary True],
pseudoEdges = [],
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta =
fromList
[ ("ID", "00000000-0000-0000-0000-000000000000")
]
}
defaultOrgParserState14 Figures
A Figure is just a link to an image file, preceded by some metadata.
The parentOid 154 is just like the one in data:Block.
The metadata (meta 155) recognizes certain reserved keys:
- type
- 🔗 (Required) Used to force there to be at least 1 key/value pair in the metadata. Otherwise when the parser comes across a link to an image on its own without metadata, it is ambiguous whether the link is part of a 1-link paragraph cell, or as part of a figure cell.
- name
- 🔗 (Optional) Used to name the cell uniquely in the surrounding Org file. Other cells can refer to this figure by this name (or as part of a global data:LinkOid). The name is optional because all figures, named or not, are always assigned a number like Figure 1 or Figure 12.
- caption
- 🔗 (Optional) Displayed as the caption for the image. If there is no caption, on weaving the only name of the figure will be just the figure number, like Figure 1 or Figure 12. You can use styled text (bold, etc), links, and inline math, just like regular paragraphs (the only caveat is that you are not allowed to have newlines).
- width
- 🔗 (Optional) Width of the image (as a percentage of page width or pixel count), for HTML weaving.
- height
- 🔗 (Optional) Height of the image (as a percentage of page width or pixel count), for HTML weaving.
The caption (taken from the metadata) is saved into the caption field 156. Finally the link 157 is the link to the image which will be displayed in the woven HTML.
Below is an example of a figure:
![[[https://en.wikipedia.org/wiki/PNG][Portable Network Graphics]] sample image.](image/PNG-Gradient.png "fig:PNG image")
Below is another example, but the figure does not have a name in the metadata, nor a caption. This is still valid.
The only practical difference is that the unnamed figure is not able to be referenced using a data:LinkOid by this or other Org files.
f:figureParser is responsible for parsing figures.
figureParser :: OrgParser Cell
figureParser = do
pState <- get
when (pState.listLevels /= [])
$ fail "figures are not allowed inside list items"
_ <-
MP.try
$ MP.lookAhead
( metaParser
>> fileLinkParser
>> endOfCell -- 158
)
metaKVs <- metaParser -- 159
when (lookup "type" metaKVs /= Just "figure")
$ fail "type must be 'figure'"
caption <- case lookup "caption" metaKVs of -- 160
Nothing -> pure $ Prose []
Just s -> case MP.runParser (runStateT captionParser pState) "" s of
Left err -> fail $ MP.errorBundlePretty err
Right (prose, _pState) -> pure prose
link <- fileLinkParser -- 161
let newFigureNumber = pState.figureNumber + 1
figureNumberMeta = ("__figure_number", T.show newFigureNumber)
case link.target of
LinkTargetOid {}
LinkTargetLineLabel {}
LinkTargetURI {}
LinkTargetFootnote {} ->
fail "'file:...' required for link to figures" -- 162
LinkTargetFilePath _ -> do
modify (\s -> s {figureNumber = newFigureNumber})
pure
. CFigure
$ Figure
{ parentOid =
firstParentOid pState.docOid pState.headingOidStack,
meta =
foldl'
(\acc (k, v) -> insert k v acc)
(delete "type" metaKVs)
[figureNumberMeta],
caption = caption,
link = link
}
where
endOfCell :: OrgParser () -- 163
endOfCell =
MP.choice
[ void . MP.try $ string "\n\n",
char '\n' >> MP.eof
]First we have to parse for the metadata 159, then the optional caption 160, and then the link to the image 161. For simplicity, we don't allow figures inside list items. For the link target itself, we check that it's pointing to a file (instead of accepting just any link) 162.
It's important that we expect the end of the cell 158 163, because otherwise we'll end up parsing the link at the start of a paragraph cell as a figure.
14.1 Tests
Below we parse a doc that has figures and blocks, both with metadata.
#+name: f:f
#+header: :xxx bar
#+begin_src haskell
f = 1 + 1
#+end_src
#+name: fig:hello
#+caption: =hello=
#+type: figure
#+width: 100%
[[file:image/hello.png]]
#+name: f:g
#+header: :yyy bar
#+begin_src haskell
g x = x + 1
#+end_src
#+type: figure
[[file:image/world.png]]
- [[file:not-figure]]
[[file:not-a-figure-2]]
[[id:00000000-0000-0000-0000-000000000000::not a figure]]
#+type: figure
[[file:foo.png]]shouldParseWithState
"0011-figures-and-blocks"
( nilOrgDoc
[ CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "haskell"),
("name", "f:f"),
("xxx", "bar")
],
body = [BTokenString "f = 1 + 1"]
},
CFigure
Figure
{ parentOid = nilOid,
meta =
fromList
[ ("__figure_number", "1"),
("width", "100%"),
("name", "fig:hello"),
("caption", "=hello=")
],
caption =
Prose
[ ptext "hello" [Verbatim]
],
link =
Link
{ target = LinkTargetFilePath "image/hello.png",
name = [stext "image/hello.png" []]
}
},
CBlock
Block
{ parentOid = nilOid,
meta =
fromList
[ ("__block_type", "src"),
("__src_language", "haskell"),
("name", "f:g"),
("yyy", "bar")
],
body = [BTokenString "g x = x + 1"]
},
CFigure
Figure
{ parentOid = nilOid,
meta =
fromList
[ ("__figure_number", "2")
],
caption = Prose [],
link =
Link
{ target = LinkTargetFilePath "image/world.png",
name = [stext "image/world.png" []]
}
},
CListItem
[ LUnorderedStart,
LItemStart
],
CParagraph
$ Prose
[ PTokenLink
Link
{ target = LinkTargetFilePath "not-figure",
name = [stext "not-figure" []]
}
],
CListItem
[ LItemEnd,
LUnorderedEnd
],
CParagraph
$ Prose
[ PTokenLink
Link
{ target = LinkTargetFilePath "not-a-figure-2",
name = [stext "not-a-figure-2" []]
}
],
CParagraph
$ Prose
[ PTokenLink
Link
{ target =
LinkTargetOid
LinkOid
{ oid = nilOid,
name = "not a figure"
},
name = [stext "not a figure" []]
}
],
CFigure
Figure
{ parentOid = nilOid,
meta =
fromList
[ ("__figure_number", "3")
],
caption = Prose [],
link =
Link
{ target = LinkTargetFilePath "foo.png",
name = [stext "foo.png" []]
}
}
]
[PseudoEdge (LCI 8, LinkOid {oid = fromWords 0 0, name = "not a figure"})]
)
defaultOrgParserState
{ cellNames =
Set.fromList
[ "f:f",
"f:g",
"fig:hello"
],
figureNumber = 3
}Metadata keys that start with a double underscore are not allowed, because Lilac expects to use them for itself internally.
#+__block_type: example
#+begin_src haskell
f = 1 + 1
#+end_srcshouldFailWithMessage
"0016-metadata-reserved-underscores"
"cannot use leading double underscores for metadata key: \"__block_type\""
215 Tables
Tables are like figures in that they have some metadata. Unlike a figure which is just a link, a table has a two-dimensional grid with some information in each row/column. The requirement is that the first non-whitespace character starts with a vertical bar |, also known as the pipe character (for piping shell commands together in UNIX).
Like figures, all tables are numbered. A table can also have a caption 165 (just like a data:Figure).
The parentOid 164 is just like the one in data:Block.
Below is an example of a table:
#+name: tbl:Programming languages
#+caption: Programming languages and typing disciplines.
| Language | Typing |
|----------+-----------------|
| LISP | dynamic, strong |
| C | static, weak |
| Haskell | static, strong |which is rendered in HTML as
| Language | Typing |
|---|---|
| LISP | dynamic, strong |
| C | static, weak |
| Haskell | static, strong |
Because we named the above with a #+name keyword, we can refer to it as [[tbl:Programming languages]] (tbl:Programming languages).
The f:tableParser parses tables.
tableParser :: OrgParser Cell
tableParser = do
pState <- get
_ <- MP.try $ MP.lookAhead (metaParser >> tableRowParser)
metaKVs <- metaParser
caption <- case lookup "caption" metaKVs of
Nothing -> pure $ Prose []
Just s -> case MP.runParser (runStateT captionParser pState) "" s of -- 167
Left err -> fail $ MP.errorBundlePretty err
Right (prose, _pState) -> pure prose
rows <- MP.some $ tableLineParser <|> tableRowParser
_ <- MP.optional tableFormulaParser -- 168
let newTableNumber = pState.tableNumber + 1
modify (\s -> s {tableNumber = newTableNumber})
pure
. CTable
$ Table
{ parentOid =
firstParentOid pState.docOid pState.headingOidStack,
meta = insert "__table_number" (T.show newTableNumber) metaKVs,
caption = caption,
grid = fromList $ rowsToTuples rows -- 169
}The f:captionParser parses the caption text from a #+caption: ... into prose. This way the caption can support other text styles, links, and even math fragments. We parse the text again by invoking MP.runParser 167, because it's easy to implement; a fancier alternative would be to enrich the f:metaParser to hold values that can be either raw text or prose.
captionParser :: OrgParser Prose
captionParser = do
caption <- proseParser $ whitespaceCompressor onlySpaces
failIfLeftoverActiveStyles
pure captionThe f:tableLineParser skips over horizontal rules (like |---+-----|) that typically separate the table heading (column names) from the rest of the rows that hold the actual data. It also skips over any leading indentation in the subsequent line, based on the continuationIndent which is set by f:listHeadIndentLevelParser. This way, we can parse tables inside list items.
tableLineParser :: OrgParser [Prose]
tableLineParser = do
_ <- string "|-"
_ <- MP.takeWhile1P (Just "table line characters") isTableLineChar
_ <- char '\n'
pState <- get
_ <-
MP.label "indentation for next table line"
$ MP.count pState.continuationIndent (char ' ')
pure [tableHorizontalLine]
where
isTableLineChar c = c == '-' || c == '+' || c == '|'The f:tableHorizontalLine is a special value to signify a horizontal line with the trailing \NUL byte in horizontal-line\NUL. This use of the \NUL is the only way we can guarantee that the user won't accidentally put in a value that looks like this in the table.
tableHorizontalLine :: Prose
tableHorizontalLine =
Prose
[ PTokenStyledText
StyledText {body = "horizontal-line\NUL", style = styleMaskFrom []}
]The f:tableRowParser parses rows with data in them. It also skips over indentation found in the subsequent line (if any). Unlike f:tableLineParser, we wrap this indentation parsing with MP.optional 170 because we cannot tell if there are more table rows after this or not. This is unlike the tableLineParser, which is never expected to be the last line of a table.
tableRowParser :: OrgParser [Prose]
tableRowParser = do
_ <- char '|'
_ <- onlySpaces
tableCells <- MP.some $ emptyTableCellParser <|> tableCellParser
_ <- char '\n'
pState <- get
_ <-
MP.optional -- 170
. MP.label "indentation for next table line"
$ MP.count pState.continuationIndent (char ' ')
pure tableCells
where
emptyTableCellParser :: OrgParser Prose -- 171
emptyTableCellParser = do
_ <- char '|'
_ <- MP.optional onlySpaces
pure $ Prose []The emptyTableCellParser 171 parses an empty table cell (because f:proseParser requires non-empty text). We need this in order to parse (essentially skip over) empty table cells, while still recording their existence. See how empty table cells are parsed in t:0012-tables-and-prose.
The f:tableCellParser parses prose text inside a single row and column combination --- which we call a table cell. The main idea is that we parse each table cell, which may not contain pipe characters in it (otherwise it confuses Orgmode in Emacs when we press TAB, which results in the table becoming formatted with extra columns that we don't want).
tableCellParser :: OrgParser Prose
tableCellParser = do
prose <-
onlyMarkingCharacterParser
<|> (proseParser $ whitespaceCompressor spacesInsideProse)
_ <- MP.optional onlySpaces
_ <- char '|'
_ <- MP.optional onlySpaces
pure $ compressProse prose
where
spacesInsideProse :: OrgParser Text -- 172
spacesInsideProse = do
_ <- MP.try $ MP.lookAhead (onlySpaces >> tableCellContent)
onlySpaces
tableCellContent :: OrgParser ()
tableCellContent = void $ MP.noneOf (" \n|" :: String)
onlyMarkingCharacterParser :: OrgParser Prose -- 173
onlyMarkingCharacterParser = do
_ <- MP.try . MP.lookAhead $ do
_ <- MP.satisfy (`elem` markingChars)
_ <- onlySpaces
_ <- char '|'
pure ()
c <- MP.satisfy (`elem` markingChars)
pure
$ Prose
[ PTokenStyledText
StyledText {body = T.singleton c, style = styleMaskFrom []}
]The spacesInsideProse 172 is a special space consumer inside table cells, where newlines and pipe characters are prohibited. This way, when f:proseParser uses it to find some contiguous amount of text, it will know to stop when it sees the pipe character.
The onlyMarkingCharacterParser 173 is there to parse f:markingChars which are special (optional) characters used only inside tables, but which share the same syntax as the characters used for styling prose, especially _, *, and /. These characters will break things if we let f:proseParser naively look at it, because that parser has no concept of table cell boundaries. The onlyMarkingCharacterParser parses one of these marking characters, and creates the corresponding newtype:Prose value. See this page for more information about marking characters.
After the rows are parsed, we need to convert them to a grid 166. We do this with f:rowsToTuples 169.
rowsToTuples :: [[Prose]] -> [((Int, Int), Prose)]
rowsToTuples rows = concatMap rowToTuples $ zip [1 ..] rowsWe assign each row its row index with zip [1 ..], then delegate most of the work to f:rowToTuples where each row's table cells are given its column index to form a completed table cell. These are simply accumulated and returned as a list. The order of accumulation (consing) does not matter because this is ultimately fed into a Map type (where the keys are not guaranteed to be ordered).
rowToTuples :: (Int, [Prose]) -> [((Int, Int), Prose)]
rowToTuples (rowIndex, row)
| null row = [((rowIndex, 1), Prose [])]
| otherwise = fst $ foldl' f ([], 1) row
where
f :: ([((Int, Int), Prose)], Int) -> Prose -> ([((Int, Int), Prose)], Int)
f (acc, colIndex) tableCell =
( ((rowIndex, colIndex), tableCell) : acc,
colIndex + 1
)The f:tableFormulaParser 168 checks for a table formula line (#+TBLFM: ...), which is neither useful for tangling nor weaving. These table formulas always come after a table, not before it. We need to be able to parse it in order to ignore it (otherwise we will think that the #+TBLFM: ... is part of another Org cell's metadata).
tableFormulaParser :: OrgParser ()
tableFormulaParser = do
_ <- string "#+"
_ <- string "tblfm: " <|> string "TBLFM: "
_ <- MP.takeWhile1P (Just "table formula") (/= '\n')
_ <- char '\n'
pure ()15.1 Numeric table cells
It's useful to know if a table cell contains mostly numeric content, at the time of weaving (see isNumericText in f:weaveTable). This way, we can right-align the contents of that table cell, just like popular spreadsheet programs.
f:isNumericText parses the input through f:isNumericTextParser.
isNumericText :: Text -> Bool
isNumericText input = case MP.runParser
(runStateT isNumericTextParser defaultOrgParserState)
""
input of
Left _ -> False
Right (b, _) -> bf:isNumericTextParser allows for a currency symbol, which can be either after or before a - or + sign. Then any combination of digits, periods, commas, and spaces are allowed. We check that there is at least 1 numeric digit in the string.
isNumericTextParser :: OrgParser Bool
isNumericTextParser = do
_ <- MP.optional sign
_ <- MP.optional $ MP.satisfy isCurrencySymbol
_ <- MP.optional sign
s <- MP.some $ MP.satisfy isDigit <|> MP.satisfy (`elem` ['.', ',', ' '])
MP.eof
let numberFound = (> 0) . length $ filter isDigit s
pure numberFound
where
isCurrencySymbol c = generalCategory c == CurrencySymbol
sign :: OrgParser Char
sign = MP.satisfy (`elem` ['-', '+'])15.2 Tests
The test below checks if we can parse a table with all 3 types of prose text (styled text, links, and math fragments).
#+name: tbl:famous-eqs
#+caption: *Bold*. [[tbl:famous-eqs][A self-link]]. \(E = mc^2\)
| Name | Equation | Link |
|-----------------+---------------------+------|
| Pythagoras \vert{} Foo | \(a^2 + b^2 = c^2\) | [[https://wiki/x][Wiki]] |
| Albert *Einstein* | \(E = mc^2\) | [[https://wiki/y][Wiki]] |
| Unknown | | |
| _ | * | / |
#+tblfm: $xshouldParseWithState
"0012-tables-and-prose"
( nilOrgDoc
Expected cells for 0012-tables-and-prose
[]
)
defaultOrgParserState
{ cellNames =
Set.fromList
[ "tbl:famous-eqs"
],
tableNumber = 1
}[ CTable
Table
{ parentOid = nilOid,
meta =
fromList
[ ("name", "tbl:famous-eqs"),
( "caption",
"*Bold*. [[tbl:famous-eqs][A self-link]]. \\(E = mc^2\\)"
),
("__table_number", "1")
],
caption =
Prose
[ ptext "Bold" [Bold],
ptext "." [],
ptext " " [],
PTokenLink
Link
{ name =
[ stext "A" [],
stext " " [],
stext "self-link" []
],
target =
LinkTargetOid
LinkOid {oid = nilOid, name = "tbl:famous-eqs"}
},
ptext "." [],
ptext " " [],
PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "E = mc^2"
}
],
grid =
fromList
[ ( (1, 1),
Prose [ptext "Name" []]
),
( (1, 2),
Prose [ptext "Equation" []]
),
( (1, 3),
Prose [ptext "Link" []]
),
( (2, 1),
L.tableHorizontalLine
),
( (3, 1),
Prose [ptext "Pythagoras | Foo" []]
),
( (3, 2),
Prose
[ PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "a^2 + b^2 = c^2"
}
]
),
( (3, 3),
Prose
[ PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "wiki",
N.uriPort = ""
},
N.uriPath = "/x",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "Wiki" []]
}
]
),
( (4, 1),
Prose
[ ptext "Albert " [],
ptext "Einstein" [Bold]
]
),
( (4, 2),
Prose
[ PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "E = mc^2"
}
]
),
( (4, 3),
Prose
[ PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "wiki",
N.uriPort = ""
},
N.uriPath = "/y",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "Wiki" []]
}
]
),
((5, 1), Prose [ptext "Unknown" []]),
((5, 2), Prose []),
((5, 3), Prose []),
((6, 1), Prose [ptext "_" []]),
((6, 2), Prose [ptext "*" []]),
((6, 3), Prose [ptext "/" []])
]
}
]Below is the same as ti:0012-tables-and-prose, but for the case where we are indented as a list item.
1) foo
#+name: tbl:famous-equations
# This is a comment. The lexeme skips it.
#+bar: baz
| Name | Equation | Link |
|-----------------+---------------------+------|
| Pythagoras \vert{} Foo | \(a^2 + b^2 = c^2\) | [[https://wiki/x][Wiki]] |
| Albert *Einstein* | \(E = mc^2\) | [[https://wiki/y][Wiki]] |shouldParseWithState
"0012-tables-inside-lists"
( nilOrgDoc
Expected cells for 0012-tables-inside-lists
[]
)
defaultOrgParserState
{ cellNames =
Set.fromList
[ "tbl:famous-equations"
],
tableNumber = 1
}[ CListItem
[ LOrderedStart,
LItemStart
],
CParagraph $ Prose [ptext "foo" []],
CTable
Table
{ parentOid = nilOid,
meta =
fromList
[ ("name", "tbl:famous-equations"),
("bar", "baz"),
("__table_number", "1")
],
caption = Prose [],
grid =
fromList
[ ( (1, 1),
Prose [ptext "Name" []]
),
( (1, 2),
Prose [ptext "Equation" []]
),
( (1, 3),
Prose [ptext "Link" []]
),
( (2, 1),
L.tableHorizontalLine
),
( (3, 1),
Prose [ptext "Pythagoras | Foo" []]
),
( (3, 2),
Prose
[ PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "a^2 + b^2 = c^2"
}
]
),
( (3, 3),
Prose
[ PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "wiki",
N.uriPort = ""
},
N.uriPath = "/x",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "Wiki" []]
}
]
),
( (4, 1),
Prose
[ ptext "Albert " [],
ptext "Einstein" [Bold]
]
),
( (4, 2),
Prose
[ PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "E = mc^2"
}
]
),
( (4, 3),
Prose
[ PTokenLink
Link
{ target =
LinkTargetURI
N.URI
{ N.uriScheme = "https:",
N.uriAuthority =
Just
N.URIAuth
{ N.uriUserInfo = "",
N.uriRegName = "wiki",
N.uriPort = ""
},
N.uriPath = "/y",
N.uriQuery = "",
N.uriFragment = ""
},
name = [stext "Wiki" []]
}
]
)
]
},
CListItem
[ LItemEnd,
LOrderedEnd
]
]We shouldn't try to parse #not as a marking character.
#not a comment
| foo | bar |
|----------------+----------------|
| #not a comment | #not-a-comment |shouldParseWithState
"0012-non-marking-char"
( nilOrgDoc
Expected cells for 0012-non-marking-char
[]
)
defaultOrgParserState
{ tableNumber = 1
}[ CParagraph
$ Prose
[ ptext "#not a comment" []
],
CTable
Table
{ parentOid = nilOid,
meta =
fromList
[ ("__table_number", "1")
],
caption = Prose [],
grid =
fromList
[ ( (1, 1),
Prose [ptext "foo" []]
),
( (1, 2),
Prose [ptext "bar" []]
),
( (2, 1),
L.tableHorizontalLine
),
( (3, 1),
Prose [ptext "#not a comment" []]
),
( (3, 2),
Prose [ptext "#not-a-comment" []]
)
]
}
]16 Mathematical expressions
There are two ways to write math expressions, which we call math fragments: inline fragments (inside prose text), or on their own as standalone fragments. Let's look at standalone math fragments first.
16.1 Standalone math fragments
In vanilla Org, Latex fragments begin with \begin{<ENVIRONMENT>} and \end{<ENVIRONMENT>}, where <ENVIRONMENT> can be anything, not just for math. However those other non-mathematical facilities are not supported by Lilac. Because we only support math, we call them math fragments.
type MathFragment :: Type
data MathFragment = MathFragment
{ parentOid :: OID,
meta :: Map Text Text,
body :: Text
}
deriving stock (Eq, Generic, Ord, Show)
deriving anyclass (FromJSON, ToJSON)There are three ways to write math fragments: using \begin{equation} and \end{equation}, \[ ... \], or \( ... \). The first two ways requires them to start at the beginning of the line, while the third way allows them to be embedded inside prose text (and displayed inside the paragraph). The third way has is discussed in Inline math fragments.
We limit our discussion here to parsing the first two ways, which are standalone math fragments. They are called standalone because they are their own cells (as CMathFragment in f:mathFragmentStandaloneParser) and don't need to be part of another cell.
Use the first form for a standalone equation like this
\begin{equation}
2 + 2 = 4
\end{equation}to get
\begin{equation}2 + 2 = 4\end{equation}
where the equation will get numbered and centered on its own. Use
\[2 + 2 = 4\]to get
to achieve the same effect, but without numbering.
When using the standalone form, you can assign a #+name keyword to it to refer to the equation using a human-friendly name, like this:
#+name: eq:Pythagorean Theorem
\begin{equation}
a^2 + b^2 = c^2
\end{equation}which becomes
\begin{equation}a^2 + b^2 = c^2\end{equation}
and you can link back to it using an internal link like eq:Pythagorean Theorem.
Let's first see how to parse \begin{equation} in f:mathFragmentNumberedParser.
mathFragmentNumberedParser :: OrgParser MathFragment
mathFragmentNumberedParser = do
pState <- get
let number = pState.equationNumber + 1
metaKVs <- metaParser
_ <- lexeme $ string "\\begin{equation}"
latexBody <-
MP.someTill
(MP.label "LaTeX standalone numbered math input" MP.anySingle)
(string "\\end{equation}")
modify (\s -> s {equationNumber = number})
pure
MathFragment
{ parentOid =
firstParentOid pState.docOid pState.headingOidStack,
meta =
insert "__equation_number" (T.pack $ show number) metaKVs,
body = trimTrailingWhitespace $ T.pack latexBody -- 174
}The call to f:trimTrailingWhitespace 174 cleans up any trailing whitespace we parsed into latexBody.
trimTrailingWhitespace :: Text -> Text
trimTrailingWhitespace = T.dropWhileEnd (\c -> T.elem c " \n\r\t")Parsing unnumbered equations with f:mathFragmentUnnumberedParser is almost the same as f:mathFragmentNumberedParser.
mathFragmentUnnumberedParser :: OrgParser MathFragment
mathFragmentUnnumberedParser = do
pState <- get
metaKVs <- metaParser
_ <- lexeme $ string "\\["
latex <-
MP.someTill
(MP.label "LaTeX standalone unnumbered math input" MP.anySingle)
(string "\\]")
anonNumber <- case lookup "name" metaKVs of
Just _ -> pure Nothing
Nothing -> do
let n = pState.anonEquationNumber + 1
modify (\s -> s {anonEquationNumber = n})
pure $ Just n
pure
MathFragment
{ parentOid =
firstParentOid pState.docOid pState.headingOidStack,
meta = case anonNumber of
Just n ->
insert "__equation_anon_number" (T.pack $ show n) metaKVs
Nothing -> metaKVs,
body = trimTrailingWhitespace $ T.pack latex
}These numbered and unnumbered parsers are combined together in the top-level f:mathFragmentStandaloneParser.
mathFragmentStandaloneParser :: OrgParser Cell
mathFragmentStandaloneParser =
CMathFragment
<$> (MP.try mathFragmentNumberedParser <|> mathFragmentUnnumberedParser)Unlike f:tableLineParser and f:tableRowParser, we don't use pState.continuationIndent to check which lines are part of the math fragment; we simply parse everything up until the end marker (\end{equation} or \]). This simplifies the parsing implementation. The downside is that we may accept input where the indentation across multiple math lines looks messy, but this is something the user should take care of anyway.
Let's check that we can parse multiple math fragments. We discard any trailing whitespace after the initial opening \begin{equation} or \[, as well as trailing whitespace for the contents inside (just before the \end{equation} marker).
\begin{equation}
1 + 1 = 2
\end{equation}
\begin{equation}
1 + 1 = 2
\end{equation}
\[
2 + 2 = 4
\]
#+name: eq:x
\[
2 + 2 = 4
\]
#+name: eq:y
\begin{equation}
1 + 1 = 2
\end{equation}shouldParseWithState
"0010-standalone-math-fragments"
( nilOrgDoc
[ CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("__equation_number", "1")
],
body = "1 + 1 = 2"
},
CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("__equation_number", "2")
],
body = "1 + 1 = 2"
},
CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("__equation_anon_number", "1")
],
body = "2 + 2 = 4"
},
CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("name", "eq:x")
],
body = "2 + 2 = 4"
},
CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("__equation_number", "3"),
("name", "eq:y")
],
body = "1 + 1 = 2"
}
]
[]
)
defaultOrgParserState
{ equationNumber = 3,
anonEquationNumber = 1,
cellNames =
Set.fromList
[ "eq:x",
"eq:y"
]
}Standalone math fragments may be included as part of a list item.
1) foo
#+name: eq:x
\[
2 + 2 = 4
\]
#+name: eq:y
\begin{equation}
1 + 1 = 2
\end{equation}shouldParseWithState
"0010-standalone-math-inside-lists"
( nilOrgDoc
[ CListItem
[ LOrderedStart,
LItemStart
],
CParagraph $ Prose [ptext "foo" []],
CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("name", "eq:x")
],
body = "2 + 2 = 4"
},
CMathFragment
MathFragment
{ parentOid = nilOid,
meta =
fromList
[ ("__equation_number", "1"),
("name", "eq:y")
],
body = "1 + 1 = 2"
},
CListItem
[ LItemEnd,
LOrderedEnd
]
]
[]
)
defaultOrgParserState
{ equationNumber = 1,
cellNames =
Set.fromList
[ "eq:x",
"eq:y"
]
}16.2 Inline math fragments
Inline math fragments are delimited by \( and \), such as in \(1 - 1 = 0\) which can be placed inside prose text (and displayed inside the paragraph) as in \(1 - 1 = 0\) .
mathFragmentInlineParser :: OrgParser MathFragment
mathFragmentInlineParser = do
failIfVerbatimOrCode
pState <- get
_ <- string "\\("
latex <- MP.someTill (MP.label "LaTeX math input" MP.anySingle) (string "\\)")
pure
MathFragment
{ parentOid =
firstParentOid pState.docOid pState.headingOidStack,
meta = fromList [], -- 175
body = T.pack latex
}Just like f:linkParser, we fail the parser if we detect that verbatim or code styles are active.
We delegate the typesetting of the math to MathJax, which expects math input based on LaTeX syntax. So there's no additional work required here during the parsing stage. The reason why it still needs a meta value 175 is because we use the same data:MathFragment type.
\(1 + 1\) is the same as \(2\), better known as \(1 + 1 = 2\).shouldParseWithState
"0010-inline-math"
( nilOrgDoc
[ CParagraph
$ Prose
[ PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "1 + 1"
},
ptext " is the same as " [],
PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "2"
},
ptext ", better known as " [],
PTokenMathFragment
MathFragment
{ parentOid = nilOid,
meta = fromList [],
body = "1 + 1 = 2"
},
ptext "." []
]
]
[]
)
defaultOrgParserState17 Testing framework
The tests here are all designed to read from mostly fully-fledged Org source documents. (We say "mostly fully-fledged" because many of the test inputs don't have the required document OID in it (because it is redundant and largely meaningless for test cases that don't are about the document OID); these tests rely on f:nilOidOrgDocPrefix to be prefixed to the input.)
Anyway, there are some benefits of writing test inputs as self-contained Org documents this way:
The raw input file (on disk) for test cases can be looked at if necessary. This is especially useful if the input has any special Org characters in it we have to escape. See ti:0007-blocks-ending and ti:0004-properties-drawer-empty where we need to escape some parts of the input with a leading comma (so as to distinguish between the Org syntax of this document versus the nested Org syntax of the input file we are defining).
We can organize the input files in a systematic way. We use a NNNN-<description> naming scheme for the inputs, which all live under test/Lilac/ParseSpec. Each unique NNNN number groups together files meant to test similar functionality. This makes it easy to see at a glance what sorts of things we test.
The downside is that we have to perform lots of small read operations (IO) to run the unit tests. But this cost is small enough that we can ignore it (we spend far more time compiling the tests themselves).
17.1 Test module
module Lilac.ParseSpec (spec) where
import Citeproc qualified as CP
import Data.Map.Strict qualified as M
import Data.Set qualified as Set
import Data.Text qualified as T
import Data.Void (Void)
import Lilac.Parse qualified as L
import Lilac.Types
( BToken (BTokenLinkTarget, BTokenString),
Block (Block, body, meta, parentOid),
Cell
( CBlock,
CCommand,
CFigure,
CFootnote,
CHeading,
CListItem,
CMathFragment,
CParagraph,
CTable
),
Citation (Citation, cites, noteNumber, prefix, styleVariation, suffix),
Cite (Cite, id, locator, prefix, suffix),
Command (PrintBibliography, PrintGlossary),
Figure (Figure, caption, link, meta, parentOid),
Footnote (Footnote, body, name),
Heading (Heading, body, level, meta, section),
LCI (LCI),
Link (Link, name, target),
LinkOid (LinkOid, name, oid),
LinkTarget
( LinkTargetFilePath,
LinkTargetFootnote,
LinkTargetLineLabel,
LinkTargetOid,
LinkTargetURI
),
ListMarker
( LItemEnd,
LItemStart,
LOrderedEnd,
LOrderedStart,
LUnorderedEnd,
LUnorderedStart
),
MathFragment (MathFragment, body, meta, parentOid),
OrgDoc
( OrgDoc,
bibPaths,
cells,
citations,
cslPath,
footnoteRefs,
glossary,
lineLabels,
meta,
oid,
pseudoEdges
),
OrgParserConf (maxHeadingLevel),
OrgParserState
( anonBlockNumber,
anonEquationNumber,
cellNames,
docOid,
equationNumber,
figureNumber,
footnoteDefinitions,
footnoteNumber,
footnoteRefs,
glossary,
headingLevel,
headingOidStack,
headingSectionNumber,
lineLabelNumber,
lineLabels,
orgParserConf,
prevCellIsFootnote,
tableNumber,
tanglePaths
),
PToken (PTokenCitation, PTokenLink, PTokenMathFragment, PTokenStyledText),
Prose (Prose),
PseudoEdge (PseudoEdge),
Style (Bold, Code, Italic, Underline, Verbatim),
StyledText (StyledText, body, style),
Table (Table, caption, grid, meta, parentOid),
defaultOrgParserState,
fromWords,
nilOid,
nilOidOrgDocPrefix,
ptext,
stext,
styleMaskFrom,
)
import Lilac.Util qualified as LU
import Network.URI qualified as N
import Relude
( Bool (False, True),
FilePath,
IO,
Identity,
Int,
Maybe (Just, Nothing),
String,
Text,
Type,
forM_,
fromList,
repeat,
runStateT,
show,
take,
($),
(+),
(.),
(<>),
(==),
)
import Test.Hspec (Spec, before, describe, it, shouldBe)
import Test.Hspec.Megaparsec
( ET,
elabel,
err,
errFancy,
etoks,
fancy,
shouldFailOn,
shouldFailWith,
shouldParse,
shouldSucceedOn,
utoks,
)
import Text.Megaparsec qualified as MP
import Text.Megaparsec.Char (char)
spec :: Spec -- 176
spec = do
All test cases
ParseSpec helpersWe have to export a function named spec 176 for discovery of test cases to happen automatically.
17.2 Test helpers
Most of the test helpers are concerned about reading a raw input file from disk, and then running the f:orgDocParser against it.
type Parser :: Type -> Type
type Parser = Concrete type for Megaparsec -- 177
f:shouldParseWith
f:shouldParseWithState
f:parseAs
f:shouldFailWithMessage
f:shouldFailWithError
f:shouldFail
f:plainParagraph
f:plainHeading
f:nilOrgDoc
f:checkTextToProseRoundTripThe Parser type synonym 177 uses the same concrete type as we do for type:OrgParser. It is used in tests such as t:0005-notFollowedBy-behavior-for-empty-input.
We need a helper function around parseOrgDoc, specifically to read in a file from a given path. We name this helper f:shouldParseWith.
This helper in turn relies on f:parseAs 178. Notice that we prefix the input with f:nilOidOrgDocPrefix if the test case isn't concerned about the document OID. The test cases that do care about this have to check the behavior of the top-level properties drawer manually, such as in t:0002-doc-metadata-properties-drawer-clash.
parseAs ::
Text ->
FilePath ->
(OrgDoc, OrgParserState) ->
IO ()
parseAs input path expected@(doc, _) =
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
path
input'
`shouldParse` expected
where
input' =
if doc.oid == nilOid
then nilOidOrgDocPrefix <> input
else inputThe f:nilOidOrgDocPrefix is technically valid enough to be its own complete Org document, as seen in t:empty-input.
nilOidOrgDocPrefix :: Text
nilOidOrgDocPrefix =
"""
:PROPERTIES:
:ID: 00000000-0000-0000-0000-000000000000
:END:
"""The f:nilOrgDoc helper is a basic "empty" Org document, used in many test cases as a nice starting point for the expected (parsed) output.
nilOrgDoc :: [Cell] -> [PseudoEdge] -> OrgDoc
nilOrgDoc cells pseudoEdges =
OrgDoc
{ oid = nilOid,
cells = cells,
pseudoEdges = pseudoEdges,
footnoteRefs = M.empty,
glossary = M.empty,
lineLabels = M.empty,
cslPath = Nothing,
bibPaths = [],
citations = [],
meta = fromList [("ID", "00000000-0000-0000-0000-000000000000")]
}If we want to check the state (the parsing operation is stateful), we need to check the expected end state as well. We do this with f:shouldParseWithState, which is basically the same as f:shouldParseWith but with the addition of the OrgParserState parameter.
shouldParseWithState :: FilePath -> OrgDoc -> OrgParserState -> Spec
shouldParseWithState path expectedOrgDoc expectedState = do
before (LU.readFrom "test/Lilac/ParseSpec/" path) $ do
it path $ \input -> parseAs input path expected
where
expected = (expectedOrgDoc, expectedState)Sometimes we want to test that a file will fail to parse. We do this with f:shouldFailWithMessage.
shouldFailWithMessage :: FilePath -> String -> Int -> Spec
shouldFailWithMessage path errMsg pos = do
before (LU.readFrom "test/Lilac/ParseSpec/" path) $ do
it path $ \input -> do
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
path
(inputPrefix <> input)
`shouldFailWith` (errFancy newPos . fancy $ MP.ErrorFail errMsg) -- 179
where
inputPrefix = nilOidOrgDocPrefix
newPos = pos + T.length inputPrefixWe use the MP.ErrorFail constructor 179 because that's the only way we can check for failures in parsers where we manually return an error with Control.Monad.fail (as opposed to Megaparsec itself throwing the error because of a failure of the given parser combinators to account for the shape of the input text).
The f:shouldFailWithError helper checks for failures thrown by Megaparsec (the ones that f:shouldFailWithMessage do not check for). The ET likely stands for "trivial errors" or (error "trivial"), because it is documented in upstream as "Auxiliary type for construction of trivial parse errors.".
shouldFailWithError :: FilePath -> ET Text -> Int -> Spec
shouldFailWithError path errorTrivial pos = do
before (LU.readFrom "test/Lilac/ParseSpec/" path) $ do
it path $ \input -> do
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
path
(inputPrefix <> input)
`shouldFailWith` (err newPos errorTrivial)
where
inputPrefix = nilOidOrgDocPrefix
newPos = pos + T.length inputPrefixThe f:shouldFail helper simply asserts that the parse should fail, without regard to the exact failure message.
shouldFail :: FilePath -> Spec
shouldFail path = do
before (LU.readFrom "test/Lilac/ParseSpec/" path) $ do
it path $ \input -> do
MP.runParser
(runStateT L.orgDocParser defaultOrgParserState)
path
`shouldFailOn` (inputPrefix <> input)
where
inputPrefix = nilOidOrgDocPrefixThe f:plainParagraph function helps to cut down on some of the repetition.
plainParagraph :: Text -> Cell
plainParagraph body =
CParagraph
$ Prose
[ PTokenStyledText
StyledText
{ body = body,
style = styleMaskFrom []
}
]The same goes for function f:plainHeading.
plainHeading :: Int -> [Int] -> Cell
plainHeading level section =
CHeading
Heading
{ level = level,
section = section,
body =
Prose
[ PTokenStyledText
StyledText
{ body = T.pack $ show level,
style = styleMaskFrom []
}
],
meta = fromList []
}17.3 All test cases
Below are all of the individual test cases.
t:empty-input
t:self-parse
t:0001-blankline
t:0001-comments-only
t:0002-doc-metadata-properties-drawer-empty
t:0002-doc-metadata-missing-value
t:0002-doc-metadata-whitespace-before-value
t:0002-doc-metadata:basic
t:0003-doc-metadata-with-comments
t:0002-doc-metadata-properties-drawer
t:0002-doc-metadata-properties-drawer-combine
t:0002-doc-metadata-properties-drawer-clash
t:0004-headings
t:0004-headings-20
t:0004-headings-unlimited
t:0004-headings-limited
t:0004-headings-prohibited
t:0004-headings-cannot-skip-level
t:Generate headings
t:Generate Org ID stack
t:0004-properties-drawer-empty
t:0004-properties-drawer-not-empty
t:0004-properties-drawer-not-empty-with-comments
t:0005-paragraph-indentation-mismatch-initial-indent-invalid-between-levels
t:0005-paragraph-indentation-mismatch-initial-indent-invalid
t:0005-paragraph-indentation-mismatch-over-list-body
t:0005-paragraph-indentation-mismatch-over
t:0005-paragraph-indentation-mismatch-under-list-body
t:0005-prose-paragraphs
t:0005-notFollowedBy-behavior-for-empty-input
t:0005-styled-strings-bold
t:0005-styled-strings-complex-punctuation
t:0005-styled-strings-slash-separated-words
t:0005-styled-strings-complex
t:0005-styled-strings-compression
t:0005-styled-strings-escape-marker
t:0005-styled-strings-leftover-active-styles
t:0005-styled-strings-messy-boundaries
t:0005-styled-strings-no-style
t:0005-styled-strings-verbatim-code-ignore-immediate-bold
t:0005-styled-strings-verbatim-precedence
t:0005-styled-strings-verbatim
t:0005-styled-strings-escaping
t:0006-links-inside-parentheses
t:0006-links-nonlink
t:0006-links-raw
t:0006-links
t:0006-links-web-protocols
t:0006-links-oids
t:0006-links-oids-lenient-whitespace
t:0006-links-oids-line-labels
t:0006-unique-line-labels
t:0007-block-src-without-lang
t:0007-blocks-empty
t:0007-blocks-ending
t:0007-blocks-noweb-off
t:0007-blocks-noweb
t:0007-blocks-noweb-intraline
t:0007-blocks-duplicate-names
t:0007-blocks-duplicate-tangle-paths
t:0007-blocks-special-commas
t:0007-blocks-no-name-with-opening-parenthesis
t:0008-indentation-comment
t:0008-indentation-headings
t:0008-indentation-paragraph
t:0009-list-content-simple
t:0009-list-indentation-nested-with-text
t:0009-list-indentation-nested
t:0009-list-nesting-max
t:0009-list-nesting-max-8
t:0009-list-indentation-no-indent
t:0009-list-indentation-proper-indent
t:0009-list-items-need-blank-line-separator
t:0009-lists-cannot-be-mixed-on-their-own-ordered
t:0009-lists-cannot-be-mixed-on-their-own
t:0009-lists-require-precise-indentation
t:0009-ordered-lists-require-increment
t:0009-ordered-lists-start-from-1
t:0009-ordered-list-head-needs-parenthesis
t:0009-ordered-list-head-cannot-use-letters
t:0009-paragraph-can-close-deeper-levels
t:0009-paragraph-can-separate-mixed-lists
t:0010-standalone-math-fragments
t:0010-standalone-math-inside-lists
t:0010-inline-math
t:0011-figures-and-blocks
t:0012-tables-and-prose
t:0012-tables-inside-lists
t:0012-non-marking-char
t:0013-footnotes-broken-link
t:0013-footnotes-unused-def
t:0013-footnotes-simple
t:0013-footnotes-complex
t:0013-footnotes-multiple-groups
t:0014-citations-csl-path
t:0014-citations-csl-json-paths
t:0014-citations-bibliography-location
t:0014-citations-single-cite
t:0014-citations-locators
t:0014-citations-locators-affixes
t:0014-citations-worg
t:0014-citations-unbalanced-brackets-prefix-opening
t:0014-citations-unbalanced-brackets-prefix-closing
t:0014-citations-unbalanced-brackets-suffix-opening
t:0014-citations-unbalanced-brackets-suffix-closing
t:0014-citations-extra-semicolons-basic
t:0014-citations-extra-semicolons-consecutive
t:0014-citations-nested
t:0014-citations-forbidden-in-verbatim-code
t:0014-citations-forbidden-in-headings
Text to Prose round trip
Drop text chars from Prose
t:0015-glossary-duplicate-term
t:0015-glossary-location
t:0015-glossary-location-global
t:0016-metadata-reserved-underscores18 Glossary
- a name
- ↑ what's displayed to the reader, and
- a target
- ↑ where we want to take the reader when they click on the link.
- caption
- ↑ (Optional) Displayed as the caption for the image. If there is no caption, on weaving the only name of the figure will be just the figure number, like Figure 1 or Figure 12. You can use styled text (bold, etc), links, and inline math, just like regular paragraphs (the only caveat is that you are not allowed to have newlines).
- height
- ↑ (Optional) Height of the image (as a percentage of page width or pixel count), for HTML weaving.
- lilac_maxHeadingLevel
- ↑ f:maxHeadingLevelParser This takes an integer as a value. Using -1 means no headings are allowed at all. Using 0 means headings can have as many asterisks as desired. Using a positive integer means headings may only have that many (or less) asterisks; otherwise Lilac will fail the parse.
- lilac_maxListLevel
- ↑ f:maxListLevelParser This has the same semantics as lilac_maxHeadingLevel but is for list items, not headings.
- name
- ↑ (Optional) Used to name the cell uniquely in the surrounding Org file. Other cells can refer to this figure by this name (or as part of a global data:LinkOid). The name is optional because all figures, named or not, are always assigned a number like Figure 1 or Figure 12.
- type
- ↑ (Required) Used to force there to be at least 1 key/value pair in the metadata. Otherwise when the parser comes across a link to an image on its own without metadata, it is ambiguous whether the link is part of a 1-link paragraph cell, or as part of a figure cell.
- width
- ↑ (Optional) Width of the image (as a percentage of page width or pixel count), for HTML weaving.
19 Footnotes
1. We deviate from the terminology used in the Org syntax description, where the terms element and object are used for what we call a cell. This is to avoid any unintentional overlap of meaning. ↑ ¶ (Cell 2)
2. For example, see the implementation for GHC 9.12 (found via base-4.21.0.0). ↑ ¶ (Cell 740)
3. We're supposed to use things like MP.region or MP.parseError according to the Megaparsec tutorial, but it doesn't seem to work (at least with the way we've constructed things with relation to our use of f:listItemsEndParser in f:cellParser). ↑ ¶ (Cell 1348)
Backlinks (99)
- compile.org (37)
- data:LinkOid (10)
- data:OrgDoc (5)
- f:oidReferenceParser (3)
- newtype:LCI (2)
- § 6 Citations (2)
- § 6.3 Printing bibliographies (2)
- data:BToken ← ¶ (Cell 47)
- data:Block ← ¶ (Cell 154)
- data:LinkTarget ← ¶ (Cell 135)
- data:PToken ← ¶ (Cell 47)
- f:bTokenParser ← ¶ (Cell 136)
- f:nowebLinkTargetParser ← ¶ (Cell 162)
- f:oidLinkParser ← ✿ backlinks
- instance:Show LinkOid ← ¶ (Cell 18)
- module:Lilac.ParseSpec ← ¶ (Cell 315)
- newtype:PseudoEdge ← ✿ backlinks
- t:0007-blocks-noweb ← ¶ (Cell 162)
- § 6.1 CSL style file (XML) ← ✿ csls
- § 6.2 CSL JSON (bibliographical entries) ← ✿ bibs
- weave.org (37)
- newtype:Prose (8)
- data:Citation (3)
- data:Cell (2)
- data:LinkOid (2)
- data:PToken (2)
- ParseSpec helpers ← ¶ (Cell 687)
- data:Cite ← ¶ (Cell 478)
- data:Footnote ← ¶ (Cell 420)
- data:MathFragment ← ¶ (Cell 114)
- data:OrgParserState ← ¶ (Cell 25)
- data:Style ← ¶ (Cell 211)
- f:blockStartParser ← ¶ (Cell 96)
- f:getBlockContents ← ¶ (Cell 321)
- f:headingParser ← ¶ (Cell 548)
- f:mathFragmentNumberedParser ← ¶ (Cell 122)
- f:padConsecutiveNowebLinks ← ¶ (Cell 127)
- type:OrgParser ← ¶ (Cell 25)
- § 11.2.3 Noweb references ← ¶ (Cell 295)
- § 12.1 Ordered list items ← ¶ (Cell 258)
- § 13 Description lists, aka glossary terms ← ¶ (Cell 428)
- § 13.1 Printing glossaries ← ¶ (Cell 442)
- § 14 Figures ← ¶ (Cell 355)
- § 15 Tables ← ¶ (Cell 367)
- § 5.8 Line labels ← ¶ (Cell 221)
- § 9 Headings ← ¶ (Cell 557)
- index.org (8)
- types.org (8)
- cli.org (4)
- tangle.org (2)
- user-manual.org (2)
- build.org (1)
Tangled files (111)
- src/Lilac/Parse.hs
- test/Lilac/ParseSpec.hs
- test/Lilac/ParseSpec/0001-blankline
- test/Lilac/ParseSpec/0001-comments-only
- test/Lilac/ParseSpec/0002-doc-metadata
- test/Lilac/ParseSpec/0002-doc-metadata-missing-value
- test/Lilac/ParseSpec/0002-doc-metadata-properties-drawer
- test/Lilac/ParseSpec/0002-doc-metadata-properties-drawer-clash
- test/Lilac/ParseSpec/0002-doc-metadata-properties-drawer-combine
- test/Lilac/ParseSpec/0002-doc-metadata-whitespace-before-value
- test/Lilac/ParseSpec/0003-doc-metadata-with-comments
- test/Lilac/ParseSpec/0004-headings
- test/Lilac/ParseSpec/0004-headings-20
- test/Lilac/ParseSpec/0004-headings-cannot-skip-level
- test/Lilac/ParseSpec/0004-headings-limited
- test/Lilac/ParseSpec/0004-headings-prohibited
- test/Lilac/ParseSpec/0004-headings-unlimited
- test/Lilac/ParseSpec/0004-properties-drawer-empty
- test/Lilac/ParseSpec/0004-properties-drawer-not-empty
- test/Lilac/ParseSpec/0004-properties-drawer-not-empty-with-comments
- test/Lilac/ParseSpec/0005-paragraph-indentation-mismatch-initial-indent-invalid
- test/Lilac/ParseSpec/0005-paragraph-indentation-mismatch-initial-indent-invalid-between-levels
- test/Lilac/ParseSpec/0005-paragraph-indentation-mismatch-over
- test/Lilac/ParseSpec/0005-paragraph-indentation-mismatch-over-list-body
- test/Lilac/ParseSpec/0005-paragraph-indentation-mismatch-under-list-body
- test/Lilac/ParseSpec/0005-prose-paragraphs
- test/Lilac/ParseSpec/0005-styled-strings-bold
- test/Lilac/ParseSpec/0005-styled-strings-complex
- test/Lilac/ParseSpec/0005-styled-strings-complex-punctuation
- test/Lilac/ParseSpec/0005-styled-strings-compression
- test/Lilac/ParseSpec/0005-styled-strings-escape-marker
- test/Lilac/ParseSpec/0005-styled-strings-escaping
- test/Lilac/ParseSpec/0005-styled-strings-leftover-active-styles
- test/Lilac/ParseSpec/0005-styled-strings-messy-boundaries
- test/Lilac/ParseSpec/0005-styled-strings-no-style
- test/Lilac/ParseSpec/0005-styled-strings-slash-separated-words
- test/Lilac/ParseSpec/0005-styled-strings-verbatim
- test/Lilac/ParseSpec/0005-styled-strings-verbatim-code-ignore-immediate-bold
- test/Lilac/ParseSpec/0005-styled-strings-verbatim-precedence
- test/Lilac/ParseSpec/0006-links
- test/Lilac/ParseSpec/0006-links-inside-parentheses
- test/Lilac/ParseSpec/0006-links-nonlink
- test/Lilac/ParseSpec/0006-links-oids
- test/Lilac/ParseSpec/0006-links-oids-lenient-whitespace
- test/Lilac/ParseSpec/0006-links-oids-line-labels
- test/Lilac/ParseSpec/0006-links-raw
- test/Lilac/ParseSpec/0006-links-web-protocols
- test/Lilac/ParseSpec/0006-unique-line-labels
- test/Lilac/ParseSpec/0007-block-src-without-lang
- test/Lilac/ParseSpec/0007-blocks-duplicate-names
- test/Lilac/ParseSpec/0007-blocks-duplicate-tangle-paths
- test/Lilac/ParseSpec/0007-blocks-empty
- test/Lilac/ParseSpec/0007-blocks-ending
- test/Lilac/ParseSpec/0007-blocks-no-name-with-opening-parenthesis
- test/Lilac/ParseSpec/0007-blocks-noweb
- test/Lilac/ParseSpec/0007-blocks-noweb-intraline
- test/Lilac/ParseSpec/0007-blocks-noweb-off
- test/Lilac/ParseSpec/0007-blocks-special-commas
- test/Lilac/ParseSpec/0008-indentation-comment
- test/Lilac/ParseSpec/0008-indentation-headings
- test/Lilac/ParseSpec/0008-indentation-paragraph
- test/Lilac/ParseSpec/0009-list-content-simple
- test/Lilac/ParseSpec/0009-list-indentation-nested
- test/Lilac/ParseSpec/0009-list-indentation-nested-with-text
- test/Lilac/ParseSpec/0009-list-indentation-no-indent
- test/Lilac/ParseSpec/0009-list-indentation-proper-indent
- test/Lilac/ParseSpec/0009-list-items-need-blank-line-separator
- test/Lilac/ParseSpec/0009-list-nesting-max
- test/Lilac/ParseSpec/0009-list-nesting-max-8
- test/Lilac/ParseSpec/0009-lists-cannot-be-mixed-on-their-own
- test/Lilac/ParseSpec/0009-lists-cannot-be-mixed-on-their-own-ordered
- test/Lilac/ParseSpec/0009-lists-prohibited
- test/Lilac/ParseSpec/0009-lists-require-precise-indentation
- test/Lilac/ParseSpec/0009-ordered-list-head-cannot-use-letters
- test/Lilac/ParseSpec/0009-ordered-list-head-needs-parenthesis
- test/Lilac/ParseSpec/0009-ordered-lists-require-increment
- test/Lilac/ParseSpec/0009-ordered-lists-start-from-1
- test/Lilac/ParseSpec/0009-paragraph-can-close-deeper-levels
- test/Lilac/ParseSpec/0009-paragraph-can-separate-mixed-lists
- test/Lilac/ParseSpec/0010-inline-math
- test/Lilac/ParseSpec/0010-standalone-math-fragments
- test/Lilac/ParseSpec/0010-standalone-math-inside-lists
- test/Lilac/ParseSpec/0011-figures-and-blocks
- test/Lilac/ParseSpec/0012-non-marking-char
- test/Lilac/ParseSpec/0012-tables-and-prose
- test/Lilac/ParseSpec/0012-tables-inside-lists
- test/Lilac/ParseSpec/0013-footnotes-broken-link
- test/Lilac/ParseSpec/0013-footnotes-complex
- test/Lilac/ParseSpec/0013-footnotes-multiple-groups
- test/Lilac/ParseSpec/0013-footnotes-simple
- test/Lilac/ParseSpec/0013-footnotes-unused-def
- test/Lilac/ParseSpec/0014-citations-bibliography-location
- test/Lilac/ParseSpec/0014-citations-csl-json-paths
- test/Lilac/ParseSpec/0014-citations-csl-path
- test/Lilac/ParseSpec/0014-citations-extra-semicolons-basic
- test/Lilac/ParseSpec/0014-citations-extra-semicolons-consecutive
- test/Lilac/ParseSpec/0014-citations-forbidden-in-headings
- test/Lilac/ParseSpec/0014-citations-forbidden-in-verbatim-code
- test/Lilac/ParseSpec/0014-citations-locators
- test/Lilac/ParseSpec/0014-citations-locators-affixes
- test/Lilac/ParseSpec/0014-citations-nested
- test/Lilac/ParseSpec/0014-citations-single-cite
- test/Lilac/ParseSpec/0014-citations-unbalanced-brackets-prefix-closing
- test/Lilac/ParseSpec/0014-citations-unbalanced-brackets-prefix-opening
- test/Lilac/ParseSpec/0014-citations-unbalanced-brackets-suffix-closing
- test/Lilac/ParseSpec/0014-citations-unbalanced-brackets-suffix-opening
- test/Lilac/ParseSpec/0014-citations-worg
- test/Lilac/ParseSpec/0015-glossary-duplicate-term
- test/Lilac/ParseSpec/0015-glossary-location
- test/Lilac/ParseSpec/0015-glossary-location-global
- test/Lilac/ParseSpec/0016-metadata-reserved-underscores
Named cells (471)
- All test cases
- CiteprocOutput typeclass
- Concrete type for Megaparsec
- Drop text chars from Prose
- Expected blocks for 0007-blocks-empty
- Expected blocks for 0007-blocks-ending
- Expected blocks for 0007-blocks-noweb
- Expected blocks for 0007-blocks-noweb-intraline
- Expected blocks for 0007-blocks-noweb-off
- Expected blocks for 0007-blocks-special-commas
- Expected cells for 0012-non-marking-char
- Expected cells for 0012-tables-and-prose
- Expected cells for 0012-tables-inside-lists
- Expected cells for 0013-footnotes-complex
- Expected cells for 0013-footnotes-simple
- Expected prose for 0005-prose-paragraphs
- Expected prose for 0005-styled-strings-complex
- Expected prose for 0005-styled-strings-complex-punctuation
- Expected prose for 0005-styled-strings-compression
- Expected prose for 0005-styled-strings-escape-marker
- Expected prose for 0005-styled-strings-escaping
- Expected prose for 0005-styled-strings-messy-boundaries
- Expected prose for 0005-styled-strings-slash-separated-words
- Expected prose for 0005-styled-strings-verbatim
- Expected prose for 0005-styled-strings-verbatim-code-ignore-immediate-bold
- Expected prose for 0005-styled-strings-verbatim-precedence
- Expected prose for 0014-citations-forbidden-in-verbatim-code
- Expected prose for 0014-citations-locators
- Expected prose for 0014-citations-locators-affixes
- Expected prose for 0014-citations-single-cite
- Expected prose for 0014-citations-worg
- Footnote (fn:cell)
- Footnote (fn:megaparsec-errors)
- Footnote (fn:semigroup-monoid-for-list)
- ParseSpec helpers
- Text to Prose round trip
- data:BToken
- data:Block
- data:Cell
- data:Citation
- data:Cite
- data:Command
- data:Figure
- data:Footnote
- data:GlossaryTerm
- data:Heading
- data:Level
- data:Link
- data:LinkOid
- data:LinkTarget
- data:ListMarker
- data:MathFragment
- data:OrgDoc
- data:OrgParserConf
- data:OrgParserState
- data:PToken
- data:Style
- data:StyledText
- data:Table
- eq:Pythagorean Theorem
- f:assertUniqueCellName
- f:assertUniqueGlossaryTerm
- f:assertUniqueLineLabel
- f:assertUniqueTanglePath
- f:bTokenParser
- f:bibliographyLocationParser
- f:blockMetaEntryParser
- f:blockParser
- f:blockStartParser
- f:captionParser
- f:cellBoundary
- f:cellParser
- f:checkTextToProseRoundTrip
- f:citationParser
- f:citeAffixParser
- f:citeIdParser
- f:citeLocatorCoordParser
- f:citeLocatorCoordRangeParser
- f:citeLocatorCoordsParser
- f:citeLocatorParser
- f:citeLocatorTermParser
- f:citeParser
- f:collectCitations
- f:collectValsFor
- f:compressBTokens
- f:compressProse
- f:compressStyledText
- f:compressedTextParser
- f:convertToGlossaryTerm
- f:defaultOrgDoc
- f:defaultOrgParserConf
- f:defaultOrgParserState
- f:deleteRedundantKeys
- f:detectInvalidIndentation
- f:docMetaEntryParser
- f:docMetaParser
- f:dropTextFromStyledText
- f:escapedSymbolParser
- f:failIfHeadingNestsWithoutDirectParent
- f:failIfLeftoverActiveStyles
- f:failIfUnbalancedBrackets
- f:failIfVerbatimOrCode
- f:figureParser
- f:fileLinkParser
- f:firstParentOid
- f:footnoteDefinitionParser
- f:footnoteLinkParser
- f:footnoteReferenceParser
- f:genHeadingOidStack
- f:genSectionNumber
- f:getBlockContents
- f:getContinuationIndentForLevel
- f:getEndingListMarkers
- f:getLinkPaddingBools
- f:glossaryLocationParser
- f:headingParser
- f:ignoreSpecialCommas
- f:intraCellWhitespaceParser
- f:isAlphaChar
- f:isAlphaNumChar
- f:isCode
- f:isDigitChar
- f:isEmptyMask
- f:isGlossaryTermDelimiter
- f:isLineLabel
- f:isNumericText
- f:isNumericTextParser
- f:isPunctuation
- f:isStyleMarker
- f:isVerbatim
- f:lexeme
- f:lilacExtensionsParser
- f:lineLabelParser
- f:linkDisplayNameParser
- f:linkParser
- f:listHeadIndentLevelParser
- f:listItemFitter
- f:listItemNester
- f:listItemStartParser
- f:listItemsEndParser
- f:maskToStyles
- f:mathFragmentInlineParser
- f:mathFragmentNumberedParser
- f:mathFragmentStandaloneParser
- f:mathFragmentUnnumberedParser
- f:maxHeadingLevelParser
- f:maxListLevelParser
- f:metaEntryParser
- f:metaParser
- f:newlineAndIndentParser
- f:nilOidOrgDocPrefix
- f:nilOrgDoc
- f:nowebLinkAtEndOfLine
- f:nowebLinkTargetParser
- f:oidLinkParser
- f:oidReferenceParser
- f:onlySpaces
- f:orderedHeadParser
- f:orgDocParser
- f:padConsecutiveNowebLinks
- f:paragraphParser
- f:parseAs
- f:parseOrgDoc
- f:plainHeading
- f:plainParagraph
- f:propertiesDrawerParser
- f:propertiesEntryParser
- f:propertiesParser
- f:proseParser
- f:proseSpaceConsumer
- f:protocolWebLinkParser
- f:ptext
- f:rawLinkParser
- f:reconcileFootnotes
- f:reconcileLineLabels
- f:resetState
- f:rowToTuples
- f:rowsToTuples
- f:shouldFail
- f:shouldFailWithError
- f:shouldFailWithMessage
- f:shouldParseWith
- f:shouldParseWithState
- f:signedIntParser
- f:skipNewlinesAndComments
- f:slashSeparatedWords
- f:someBTokensParser
- f:someIndent
- f:stext
- f:styleMarkerParser
- f:styleMarkers
- f:styleMaskFlip
- f:styleMaskFrom
- f:styledTextParser
- f:tableCellParser
- f:tableFormulaParser
- f:tableHorizontalLine
- f:tableLineParser
- f:tableParser
- f:tableRowParser
- f:textParser
- f:toStyleMask
- f:trimTrailingWhitespace
- f:unorderedHeadParser
- f:unparseLink
- f:unparseMathFragment
- f:unparseStyledText
- f:validateCslPath
- f:validateOidText
- f:verbatimCodeParser
- f:whitespaceCompressor
- f:zeroBTokensParser
- f:zeroIndent
- fig:PNG image
- instance:CiteprocOutput Prose
- instance:CiteprocOutput Prose:addDisplay
- instance:CiteprocOutput Prose:addFontStyle
- instance:CiteprocOutput Prose:addFontVariant
- instance:CiteprocOutput Prose:addFontWeight
- instance:CiteprocOutput Prose:addHyperlink
- instance:CiteprocOutput Prose:addQuotes
- instance:CiteprocOutput Prose:addTextCase
- instance:CiteprocOutput Prose:addTextDecoration
- instance:CiteprocOutput Prose:addVerticalAlign
- instance:CiteprocOutput Prose:fromText
- instance:CiteprocOutput Prose:inNote
- instance:CiteprocOutput Prose:localizeQuotes
- instance:CiteprocOutput Prose:mapText
- instance:CiteprocOutput Prose:movePunctuationInsideQuotes
- instance:CiteprocOutput Prose:toText
- instance:Monoid Prose
- instance:Semigroup Prose
- instance:Show LinkOid
- instance:Show Stylemask
- link-marker
- module:Lilac.Parse
- module:Lilac.ParseSpec
- newtype:LCI
- newtype:OID
- newtype:Prose
- newtype:PseudoEdge
- newtype:StyleMask
- orgmode-citation-breakage-workaround
- some message
- t:0001-blankline
- t:0001-comments-only
- t:0002-doc-metadata-missing-value
- t:0002-doc-metadata-properties-drawer
- t:0002-doc-metadata-properties-drawer-clash
- t:0002-doc-metadata-properties-drawer-combine
- t:0002-doc-metadata-properties-drawer-empty
- t:0002-doc-metadata-whitespace-before-value
- t:0002-doc-metadata:basic
- t:0003-doc-metadata-with-comments
- t:0004-headings
- t:0004-headings-20
- t:0004-headings-cannot-skip-level
- t:0004-headings-limited
- t:0004-headings-prohibited
- t:0004-headings-unlimited
- t:0004-properties-drawer-empty
- t:0004-properties-drawer-not-empty
- t:0004-properties-drawer-not-empty-with-comments
- t:0005-notFollowedBy-behavior-for-empty-input
- t:0005-paragraph-indentation-mismatch-initial-indent-invalid
- t:0005-paragraph-indentation-mismatch-initial-indent-invalid-between-levels
- t:0005-paragraph-indentation-mismatch-over
- t:0005-paragraph-indentation-mismatch-over-list-body
- t:0005-paragraph-indentation-mismatch-under-list-body
- t:0005-prose-paragraphs
- t:0005-styled-strings-bold
- t:0005-styled-strings-complex
- t:0005-styled-strings-complex-punctuation
- t:0005-styled-strings-compression
- t:0005-styled-strings-escape-marker
- t:0005-styled-strings-escaping
- t:0005-styled-strings-leftover-active-styles
- t:0005-styled-strings-messy-boundaries
- t:0005-styled-strings-no-style
- t:0005-styled-strings-slash-separated-words
- t:0005-styled-strings-verbatim
- t:0005-styled-strings-verbatim-code-ignore-immediate-bold
- t:0005-styled-strings-verbatim-precedence
- t:0006-links
- t:0006-links-inside-parentheses
- t:0006-links-nonlink
- t:0006-links-oids
- t:0006-links-oids-lenient-whitespace
- t:0006-links-oids-line-labels
- t:0006-links-raw
- t:0006-links-web-protocols
- t:0006-unique-line-labels
- t:0007-block-src-without-lang
- t:0007-blocks-duplicate-names
- t:0007-blocks-duplicate-tangle-paths
- t:0007-blocks-empty
- t:0007-blocks-ending
- t:0007-blocks-no-name-with-opening-parenthesis
- t:0007-blocks-noweb
- t:0007-blocks-noweb-intraline
- t:0007-blocks-noweb-off
- t:0007-blocks-special-commas
- t:0008-indentation-comment
- t:0008-indentation-headings
- t:0008-indentation-paragraph
- t:0009-list-content-simple
- t:0009-list-indentation-nested
- t:0009-list-indentation-nested-with-text
- t:0009-list-indentation-no-indent
- t:0009-list-indentation-proper-indent
- t:0009-list-items-need-blank-line-separator
- t:0009-list-nesting-max
- t:0009-list-nesting-max-8
- t:0009-lists-cannot-be-mixed-on-their-own
- t:0009-lists-cannot-be-mixed-on-their-own-ordered
- t:0009-lists-prohibited
- t:0009-lists-require-precise-indentation
- t:0009-ordered-list-head-cannot-use-letters
- t:0009-ordered-list-head-needs-parenthesis
- t:0009-ordered-lists-require-increment
- t:0009-ordered-lists-start-from-1
- t:0009-paragraph-can-close-deeper-levels
- t:0009-paragraph-can-separate-mixed-lists
- t:0010-inline-math
- t:0010-standalone-math-fragments
- t:0010-standalone-math-inside-lists
- t:0011-figures-and-blocks
- t:0012-non-marking-char
- t:0012-tables-and-prose
- t:0012-tables-inside-lists
- t:0013-footnotes-broken-link
- t:0013-footnotes-complex
- t:0013-footnotes-multiple-groups
- t:0013-footnotes-simple
- t:0013-footnotes-unused-def
- t:0014-citations-bibliography-location
- t:0014-citations-csl-json-paths
- t:0014-citations-csl-path
- t:0014-citations-extra-semicolons-basic
- t:0014-citations-extra-semicolons-consecutive
- t:0014-citations-forbidden-in-headings
- t:0014-citations-forbidden-in-verbatim-code
- t:0014-citations-locators
- t:0014-citations-locators-affixes
- t:0014-citations-nested
- t:0014-citations-single-cite
- t:0014-citations-unbalanced-brackets-prefix-closing
- t:0014-citations-unbalanced-brackets-prefix-opening
- t:0014-citations-unbalanced-brackets-suffix-closing
- t:0014-citations-unbalanced-brackets-suffix-opening
- t:0014-citations-worg
- t:0015-glossary-duplicate-term
- t:0015-glossary-location
- t:0015-glossary-location-global
- t:0016-metadata-reserved-underscores
- t:Generate Org ID stack
- t:Generate headings
- t:empty-input
- t:self-parse
- tbl:List markers
- tbl:Programming languages
- ti:0001-blankline
- ti:0001-comments-only
- ti:0002-doc-metadata-missing-value
- ti:0002-doc-metadata-properties-drawer
- ti:0002-doc-metadata-properties-drawer-clash
- ti:0002-doc-metadata-properties-drawer-combine
- ti:0002-doc-metadata-whitespace-before-value
- ti:0002-doc-metadata:basic
- ti:0003-doc-metadata-with-comments
- ti:0004-headings
- ti:0004-headings-20
- ti:0004-headings-cannot-skip-level
- ti:0004-headings-limited
- ti:0004-headings-prohibited
- ti:0004-headings-unlimited
- ti:0004-properties-drawer-empty
- ti:0004-properties-drawer-not-empty
- ti:0004-properties-drawer-not-empty-with-comments
- ti:0005-paragraph-indentation-mismatch-initial-indent-invalid
- ti:0005-paragraph-indentation-mismatch-initial-indent-invalid-between-levels
- ti:0005-paragraph-indentation-mismatch-over
- ti:0005-paragraph-indentation-mismatch-over-list-body
- ti:0005-paragraph-indentation-mismatch-under-list-body
- ti:0005-prose-paragraphs
- ti:0005-styled-strings-bold
- ti:0005-styled-strings-complex
- ti:0005-styled-strings-complex-punctuation
- ti:0005-styled-strings-compression
- ti:0005-styled-strings-escape-marker
- ti:0005-styled-strings-escaping
- ti:0005-styled-strings-leftover-active-styles
- ti:0005-styled-strings-messy-boundaries
- ti:0005-styled-strings-no-style
- ti:0005-styled-strings-slash-separated-words
- ti:0005-styled-strings-verbatim
- ti:0005-styled-strings-verbatim-code-ignore-immediate-bold
- ti:0005-styled-strings-verbatim-precedence
- ti:0006-links
- ti:0006-links-inside-parentheses
- ti:0006-links-nonlink
- ti:0006-links-oids
- ti:0006-links-oids-lenient-whitespace
- ti:0006-links-oids-line-labels
- ti:0006-links-raw
- ti:0006-links-web-protocols
- ti:0006-unique-line-labels
- ti:0007-block-src-without-lang
- ti:0007-blocks-duplicate-names
- ti:0007-blocks-duplicate-tangle-paths
- ti:0007-blocks-empty
- ti:0007-blocks-ending
- ti:0007-blocks-no-name-with-opening-parenthesis
- ti:0007-blocks-noweb
- ti:0007-blocks-noweb-intraline
- ti:0007-blocks-noweb-off
- ti:0007-blocks-special-commas
- ti:0008-indentation-comment
- ti:0008-indentation-headings
- ti:0008-indentation-paragraph
- ti:0009-list-content-simple
- ti:0009-list-indentation-nested
- ti:0009-list-indentation-nested-with-text
- ti:0009-list-indentation-no-indent
- ti:0009-list-indentation-proper-indent
- ti:0009-list-items-need-blank-line-separator
- ti:0009-list-nesting-max
- ti:0009-list-nesting-max-8
- ti:0009-lists-cannot-be-mixed-on-their-own
- ti:0009-lists-cannot-be-mixed-on-their-own-ordered
- ti:0009-lists-prohibited
- ti:0009-lists-require-precise-indentation
- ti:0009-ordered-list-head-cannot-use-letters
- ti:0009-ordered-list-head-needs-parenthesis
- ti:0009-ordered-lists-require-increment
- ti:0009-ordered-lists-start-from-1
- ti:0009-paragraph-can-close-deeper-levels
- ti:0009-paragraph-can-separate-mixed-lists
- ti:0010-inline-math
- ti:0010-standalone-math-fragments
- ti:0010-standalone-math-inside-lists
- ti:0011-figures-and-blocks
- ti:0012-non-marking-char
- ti:0012-tables-and-prose
- ti:0012-tables-inside-lists
- ti:0013-footnotes-broken-link
- ti:0013-footnotes-complex
- ti:0013-footnotes-multiple-groups
- ti:0013-footnotes-simple
- ti:0013-footnotes-unused-def
- ti:0014-citations-bibliography-location
- ti:0014-citations-csl-json-paths
- ti:0014-citations-csl-path
- ti:0014-citations-extra-semicolons-basic
- ti:0014-citations-extra-semicolons-consecutive
- ti:0014-citations-forbidden-in-headings
- ti:0014-citations-forbidden-in-verbatim-code
- ti:0014-citations-locators
- ti:0014-citations-locators-affixes
- ti:0014-citations-nested
- ti:0014-citations-single-cite
- ti:0014-citations-unbalanced-brackets-prefix-closing
- ti:0014-citations-unbalanced-brackets-prefix-opening
- ti:0014-citations-unbalanced-brackets-suffix-closing
- ti:0014-citations-unbalanced-brackets-suffix-opening
- ti:0014-citations-worg
- ti:0015-glossary-duplicate-term
- ti:0015-glossary-location
- ti:0015-glossary-location-global
- ti:0016-metadata-reserved-underscores
- type:OrgParser