Updates on version 4

[jsinger67]

Version 4 is in preparation

While version 3 is in its stabilization phase and will surely be declared production-ready soon, further developments have taken place. The main focus was on improving scanning performance. This had suffered a setback with the switch to scnr, leading to problems in some areas.
sncr was a big step towards supporting mode switching within the scanner itself and establishing a clean interface between scanner and parser. But as with so many things in life, sometimes you need a second attempt.
In this case, this second attempt is called scnr2. The successor to scnr places greater emphasis on simplicity and speed. It relies on compile-time code generation using Rust macros. The macro syntax used offers the possibility of defining transitions between scanner modes in various ways. Specifically, there are three types of transitions: set, push, and pop. Furthermore, as with scnr, you can define a positive or negative lookahead for each terminal.
Additionally, scnr2 offers more support for regex features such as case insensitivity.
However, there are also some new limitations. Manually setting a new position in the input string and manually setting the current scanner mode are no longer supported. This decision was made to keep the scanner's runtime overhead as low as possible.
For parol V4, this means that parser-controlled mode changes using %sc, %push, and %pop within productions are no longer possible.
As a replacement, parol 4 allows scanner mode transitions to be defined using %push and %pop in the %scanner directives, besides the already available %enter instruction. This moves scanner mode control completely from the grammar to the scanner.

I will announce further development on upcoming version 4 here.

As always I'm grateful for feedback and suggestions.

[ryo33]

@jsinger67 I'm looking forward to the next generation scanner system! Btw, I have a question: Will it support unicode properties like Identifier : /\p{XID_Start}[\p{XID_Continue}-]*/ ; or more complex case like QuotedString : /"(\\[nrt\\"0]|\p{Letter}|\p{Mark}|\p{Number}|[\p{Punctuation}--\\"]|\p{Symbol}|\p{Space_Separator})*"/ ;?

[jsinger67]

@ryo33, good question.

scnr2 uses the regex-syntax crate to parse regexes. So all Unicode properties this crate supports are also supported by scnr2.
Two restrictions regarding supported regex features should be mentioned, but these are not new.

• Non-greedy repetitions are not supported
• Anchors like word boundaries, start of text, end of text, start of line and end of line are not supported

[ryo33]

Thank you for clarifying! It's good to hear

[ryo33]

@jsinger67 Though I've not validated the following way is actually feasible yet, and don't understand the detailed timelines of stabilization and scnr2 for version 4, I'd like to share my idea of the release process.

1. yank 3.0.0 and 3.0.1 on crates.io as it's not stable yet.
2. release 3.1.0-alpha.0, *-alpha.1, *-beta.0, *-rc.0 for rapid release and feedback cycle.
3. if version 3 is eventually considered as stabilized, release it as 3.1.0
4. (Optionally) scnr2 lands on 3.1.0

[ryo33]

Or, a more realistic idea is to remain 3.0.0 and 3.0.1 as not yanked and usable by users, and to start a release cycle for 4.0.0 with alpha, beta, and rc suffixes.

[jsinger67]

I would rather refrain from yanking version 3. You might not be aware of it but the regex_automata feature is used by veryl for performance reasons. As far as I know veryl also relies heavily on the grammar based scanner state switching. As long as I can't offer a working substitute with acceptable performance this would surely break this project.
If I could provide an approach for veryl I think scnr2's performance would likely be sufficient or even better. To prove it I had unfortunately no time yet.

[jsinger67]

I should add that the feature regex_automata will be discontinued with introducing scnr2.

Concerning the release cycle of version 3 I think the next step should be to tag the latest version on main and use this tag as branching point of a release3 branch where fixes and updates for version 3 can be made.

For version 4 this means that as soon as this is done scnr2 branch can be merged to main and first releases could be published when functionality is decent enough.

This week I'm away from keyboard due to a little holiday 🌞, so this is going to happen not before next week.

[ryo33]

Thanks. I have now better understanding of the current situation and milestones. Happy holidays!

[jsinger67]

Start of version 4 development

2025-07-11

I released version 3.1 of parol, parol-runtime and parol-ls today.
I also created and published branch release3.1 for fixes and updates of these versions.
Then I merged branch scnr2 to main and bumped crate versions to 4.0.0.
So we can start with version 4. 🥳

Feedback appreciated 😎

[jsinger67]

Release of version 4 on crates.io

2025-07-27

I published version 4.0.0 of parol, parol-runtime and parol-ls today on crates.io to make this new version more accessible for users.
The brave ones should use it and report any issues. 🤓

Have fun and good luck!

PS: The documentation of this new version is not up to date yet. Please read the introduction of this discussion for changes that were planned. I realized these plans nearly exactly as described above. Also, feel free to ask questions here.

[jsinger67]

Further updates are shared via CHANGELOG.md

2025-08-14

To lower the amount of information to maintain I would as of now announce new features and changes only in the CHANGELOG.md.

Hint: Check out the new scanner directive: %allow_unmatched

Nevertheless this place can be used to give feedback and bring in own ideas.