you should hand-write a parser
I found myself adding attribute support to Comrak on the weekend. Attributes in Markdown (as they’re often supported; there’s nothing for them in CommonMark) look like this:
Henlo [world](https://agave.syrup){.delicious}
This would, in most dialects that support it, attach a “class” attribute called delicious, which, in most dialects that then render these to HTML, would produce something like:
Henlo <a href="https://agave.syrup" class="delicious">world</a>
You can also attach IDs with #bweh and arbitrary key-pairs with wan=nyan, not to mention double-quote and escaping support, leading to {#monstrosities .somewhat like=this}. But y’know, it’s fine, and it’s even helpful for attaching things like height and width to images without having to resort to HTML. GitLab happens to use them for this purpose, though it implements it with a Very Special Post-Processing Step.
So how do we parse these fun strings? Ignoring the actual situating-them-in-the-document bit — assume we get to write a function that gets called whenever a { is found in a place where it might be an attribute (like right after a link or image, say), and it’s given what follows, and we need to (a) decide if it’s actually an attribute, and if so, (b) return the structured data from it, and how long that was (so the parent parser knows how much to skip).
I don’t know what your first impulse is, but it should probably be to just hand-write a state machine driven parser. Here’s mine! Consider reading the parse function closely enough that it makes sense to you, if your impulse is to reach for a regular expression, lexer/parser generator, or otherwise.
Some notes:
- Locally defined state enums are fun!
- Other than the result (which is random-write, as attributes may arrive in any order), all state is stored in the state enum member; this is why
ValueembedsKindandQuote, and whyKindembedsPair. I have to shuffle these around in some states, but it’s cleaner than having more function locals which are effectively “globals” that only apply sometimes, and which you can forget to reinit. - We use
str::char_indicesbecause the input is Unicode, we really do prefer to enumerate characters and not bytes, but also we return the (byte) index to the parent parser so it knows how many to skip (since a character count would necessitate O(n) through the string, thank’s UTF-8). Note we fearlessly calculate things likei + 1when we know the character underiis single-byte (i.e.'}'). Noneis returned if the input didn’t have a valid attribute set as a prefix, i.e. any parse error whatsoever; we take advantage of this to callci.next()?, since it’ll bubbleNoneup if we hit the end of the string without the parse finishing.- The double
loopis a side-effect of the particular way I’ve chosen to deal with a certain property of many state machines, including this one: most state/character combinations result in abreak(which means we repeat the outer loop and go to the next character), but some, like seeing a whitespace or}while reading a value, instead change the state back to some parent one (because we know the value has ended), and then let the new state handle the specifics of the character.-
There’s more than one way to handle this kind of thing, where you want to transition to a state and let it handle that same character too, this was just least-effort. Some setups mean you’ll be able to only ‘consume’ the input token in branches that you want to (i.e. everywhere I
break), others let you “unput” it back onto the input so it gets read again next loop, you can be creative. :) -
Concretely, to make our one cleaner, you could DIY unput with something like:
// ...let mut unput: Option<(usize, char)> = None;loop {let (i, c) = unput.take().or_else(|| ci.next())?;// ...Then you can drop the inner
loop, replace each point that doesn’t break withunput = Some((i, c));, and remove all thebreaks! PRs welcome :D
-
That’s basically all I have to mention. Hand-write a parser! You can do really silly things like the function that follows and even do a partial parse backwards, if your input syntax is sympathetic to it :)
