More fine-grained events

[not-my-profile]

The Event enum of Jotdown is currently quite high-level. For example for this span:

what{a=test}

Jotdown generates the events:

(Start(Span, {a="test"}), 0..0)
(Str("what"), 0..4)
(End(Span), 4..12)

While djot.js generates the events:

,{ startpos: 0, endpos: 3, annot: "str" }
,{ startpos: 4, endpos: 4, annot: "+attributes" }
,{ startpos: 5, endpos: 5, annot: "key" }
,{ startpos: 6, endpos: 6, annot: "attr_equal_marker" }
,{ startpos: 7, endpos: 10, annot: "value" }
,{ startpos: 11, endpos: 11, annot: "-attributes" }

I think having more fine-grained events is quite valuable for all use cases that care about individual syntax tokens such as syntax highlighting or auto formatting. While it has been suggested to add more low-level details to Jotdown's existing Event types I think this would make for a rather awkward API. Also finer-grained events would enable API users to avoid unnecessary allocations, e.g. if you don't care about attributes at all you can simply skip over the events (whereas the current Event type always allocates a Vec for attributes). If you don't care about low-level events you probably want to use an AST anyway (#17), so I think it would be worthwhile to replace the current Event type with a more fine-grained one.

[hellux]

Low-level events like the ones djot.js produces would make the parsing a lot simpler, but I don't think you could still easily use these events for rendering. The djot.js crate seems to rely on its AST for rendering. Relying on an AST to render would prevent us from parsing and rendering in a streaming fashion and would require major changes to the crate API. I think a crate that uses low-level events and produces a higher level AST for filtering and rendering can be useful, both approaches each have their benefits: Fine-grained events+AST: - Parsing events is much simpler, - fine-grained events are useful for interpreting or editing source text, - can easily skip unneeded events/allocations, - AST API always available. High-level events (like jotdown): - rendering can be done directly on events, allowing streamed parse+render (potential benefits to memory usage and performance), - does not rely on an AST, - higher-level APIs (e.g. an AST) can be implemented independently of both parsing and rendering. I think it would warrant a new crate to implement this API. The main drawback is that it creates a split, dividing spent efforts on two crates instead of one and requiring users to choose between multiple. On the other hand, converting jotdown would force current users to use an AST for filtering/rendering or stay on an outdated version.

[not-my-profile]

Just because you can build an AST from events doesn't mean that you have to create an AST structure in memory. The tree builder can defer the storing of the nodes to a trait, which could then if streaming parsing and rendering is desired, always output a node as soon as it's closed. This would even be more efficient than jotdown memory and performance wise because e.g. a single buffer could be reused for the attributes across all events.

I want to note that the lower-level events of djot.js still take care of the parsing, it's not just lexing it's doing the necessary lookaheads (e.g. for what{a=test without a closing } it only emits a str event). So with such lower-level events higher-level APIs can also be implemented independently of both parsing and rendering ... only that you then actually can have the control over what you allocate for in case you need it.

I might at some point have the time to implement such a crate. Currently I don't really need the lower-level events, I'd just like some Rust binary that I can depend on from a Python package to parse Djot into an AST. Since Jotdown cannot create an AST yet, I've briefly looked into contributing that but I'm not really interested in writing code, relying on such higher-level events.

[hellux]

Just because you can build an AST from events doesn't mean that you have to create an AST structure in memory. The tree builder can defer the storing of the nodes to a trait, which could then if streaming parsing and rendering is desired, always output a node as soon as it's closed.

Would these AST nodes be streamed linearly and be quite similar to the current events, then? Or are they fundamentally different somehow?

This would even be more efficient than jotdown memory and performance wise because e.g. a single buffer could be reused for the attributes across all events.

How come a single buffer be reused in that case but not with current jotdown?

I want to note that the lower-level events of djot.js still take care of the parsing, it's not just lexing it's doing the necessary lookaheads (e.g. for `what{a=test` without a closing `}` it only emits a `str` event). So with such lower-level events higher-level APIs can also be implemented independently of both parsing and rendering ... only that you then actually can have the control over what you allocate for in case you need it.

It might be possible to render directly from the events then, but it would move some responsibilities from the parser to the renderer, e.g. buffering of events. I.e. to emit the starting span element of `[*content*]{a=b}`, one must buffer the content to get to the attributes. I might do some experiments with this if I get some time.

[not-my-profile]

Would these AST nodes be streamed linearly and be quite similar to the current events, then?

There'd be a trait with methods for all of the current Event variants: e.g. open_node for Start, close_node for End (though the quotation marks that are balanced should also be nodes and for the smart punctuation variants there could be a single method that just takes an enum). See for example the TreeSink trait from html5ever, though since djot is much simpler than HTML the trait would be simpler as well. Yes that trait could then be easily used to create an Event iterator (or build an AST, or directly render or whatever).

How come a single buffer be reused in that case but not with current jotdown?

jotdown::Event is currently owning the Vec inside of the Attributes (and also makes another opinionated choice by wrapping the borrowed strings in a Cow for mutation convenience). If you delegate the type construction via a trait then the user can freely choose what to store and how. (It would also be possible to make jotdown::Event more flexible by introducing generic type params, though I think that avoiding generic type parameters is more elegant).

it would move some responsibilities from the parser to the renderer, e.g. buffering of events. I.e. to emit the starting span element of [*content*]{a=b}, one must buffer the content to get to the attributes.

Jotdown can only parse from a &str though, so the whole input is already in memory, so the content doesn't need to be buffered by the parser. But yes some looking around will be necessary but that's currently the case as well, no?

I've just realized that the trait I have in mind could really quite easily be introduced in current jotdown, still generating the current Events from it for backwards compatibility while allowing an AST to be build via the trait and then in the future possibly also driving the trait via more fine-grained events. I think I can make a PR to introduce such a trait.

While looking at jotdown::Event I also just noticed that it's odd that many variants don't have attributes. djot.js does support attributes for symbols, smart punctuation and even non-breaking spaces and line breaks in its AST ... it's just it's default HTML renderer that ignores them. So I think attributes can really be attached to all inline elements.