Feature request: Using subparsers

[jonahbeckford]

I'm embedding sublanguages (ex. POSIX shell commands) inside a parent language (ex. JSON). I didn't see a natural way to do this so implemented the following:

https://gitlab.com/dkml/build-tools/MlFront/-/blob/d6735dca8f0580584d5ba36feaf02dafb8cb68c4/src/MlFront_Thunk/ThunkParsers.ml#L351-L524

and made the lexer configuration driven:

https://gitlab.com/dkml/build-tools/MlFront/-/blob/d6735dca8f0580584d5ba36feaf02dafb8cb68c4/src/MlFront_Thunk/ThunkLexers.ml#L326-L379

and finally did the embedding here:

https://gitlab.com/dkml/build-tools/MlFront/-/blob/d6735dca8f0580584d5ba36feaf02dafb8cb68c4/src/MlFront_Thunk/ThunkCommand.ml#L1028-L1047

It works, but I suspect that there may be a better pattern for this.

Random factoid: The JSON parser inside ThunkParsers and ThunkLexers was based initially on your JSON parser example, but should now fully implement the JSON spec.

[hbr]

I don't understand fully what you are trying to do.

I assume the following:

The JSON language is fully specified. Strings (like numbers and booleans) are atomic values usually recognized by the tokenizer. If I understand you correctly you want to treat some of the strings as posix shell commands and see if they satisfy the shell syntax. I.e. inside the parser you get a string and want to parse the string according to the specification of your sublanguage (e.g. posix shell).

How is this sublanguage parsing triggered? By some property names used as keywords?

Can you give me some examples of JSON strings?

[jonahbeckford]

It is easiest to see what I mean with a working example.

Here is a test case:

https://gitlab.com/dkml/build-tools/MlFront/-/blob/93a6803ea0384db08aefa2d3a78351a7f7025537/tests/MlFront_Thunk/basic.t/run.t#L17-L27

which runs on the JSON file:

https://gitlab.com/dkml/build-tools/MlFront/-/blob/93a6803ea0384db08aefa2d3a78351a7f7025537/tests/MlFront_Thunk/basic.t/SubparseNormalizeSemanticError.thunk.json#L17-L27

Callout 1. Notice that the JSON string at https://gitlab.com/dkml/build-tools/MlFront/-/blob/93a6803ea0384db08aefa2d3a78351a7f7025537/tests/MlFront_Thunk/basic.t/SubparseNormalizeSemanticError.thunk.json#L17:

"get-\u006Fbject OurStd_Std.Build.GCC@1.0.0 File.2Agnostic -f gcc"

is parsed correctly into an OCaml string at https://gitlab.com/dkml/build-tools/MlFront/-/blob/93a6803ea0384db08aefa2d3a78351a7f7025537/tests/MlFront_Thunk/basic.t/run.t#L19:

get-object OurStd_Std.Build.GCC@1.0.0 File.2Agnostic -f gcc

Callout 2. Notice also that with my subparser hack the error is located correctly within the raw source code at https://gitlab.com/dkml/build-tools/MlFront/-/blob/93a6803ea0384db08aefa2d3a78351a7f7025537/tests/MlFront_Thunk/basic.t/run.t#L25:

   17 │            "get-\u006Fbject OurStd_Std.Build.GCC@1.0.0 File.2Agnostic -f gcc"
      •                                                             ┬──────── 
      •                                                             ╰╸ Expected a namespace term for an object slot like `SomeModule`, not `2Agnostic`.
  ────╯

---

What Ifs

1. What if I parsed the JSON like any other OCaml JSON parser? And then parsed the POSIX command (the get-object ... string) after? The error location would be inaccurate and shifted to the left by 5 characters because the OCaml string has lost the width of the \u0006F JSON escape character.
2. What if I split the JSON token String into the lexer tokens for a POSIX shell commands (essentially a lexer of Words that respect POSIX quoting rules)? Well, that would be a single language but quite complicated (a "Frankenstein" language), especially since I have more sublanguages than just the POSIX shell commands. Also, I do and will re-use the POSIX shell command language separately. Ideally I would to compose parsers rather than create a Frankenstein parser language.

---

For completeness, here are the other uses for the POSIX shell language:

• I have a CLI that re-uses the POSIX shell language, and it works today on the Sys.argv string array directly using my POSIX shell parser (the JSON parser is not involved):

  $ mlfront-shell -f ext/MlFront/tests/MlFront_Thunk/basic.t/SubparseNormalize.thunk.json \
      --errors-pretty -- get-object StdStd_Std.Fail@1.0.0 File.3Agnostic -f bin/gcc
  
  FATAL: The build failed.
  [warning] Could not parse the shell command `get-object StdStd_Std.Fail@1.0.0 File.3Agnostic -f bin/gcc`.
  In <mlfront-shell>:0.38-0.47:
  1 | get-object StdStd_Std.Fail@1.0.0 File.3Agnostic -f bin/gcc
                                            ^^^^^^^^^
  
  Could not parse the object slot `File.3Agnostic`.
  Expected a namespace term for an object slot like `SomeModule`, not `3Agnostic`.

• Eventually I will also have a REPL for that POSIX shell language ... it does not exist today, but it would be something like:

  $ mlfront-shell -f ext/MlFront/tests/MlFront_Thunk/basic.t/SubparseNormalize.thunk.json
  
  REPL> get-object StdStd_Std.Fail@1.0.0 File.4Agnostic -f bin/gcc
  
  In <REPL>:0.38-0.47:
  1 | get-object StdStd_Std.Fail@1.0.0 File.4Agnostic -f bin/gcc
                                            ^^^^^^^^^

Each of the three uses of the POSIX shell parser (embedded in a JSON string, parsed from a argv array, parsed from a REPL string) has slightly different lexing rules because each has different escaping and quoting rules. The state monad (State functor parameter) is sufficient today so I can make one lexer that handles all three situations.

[jonahbeckford]

I also realize I may be the only person who will ever use sub-parsers, and my hack to get compositionality is okay but not great. So please consider this a low priority.