shell comment lost

[tst2005]

Hello,
I'm playing with LPeg and lot of things around.

I started to test your sh-parser that seems really cool.

I see shell's comment is not managed like the rest of the body

My shell sample is

# comment 1
FOO=foo
# comment 2
BAR=bar

The parsed result is

t = {
   ["body"] = {
      [1] = {
         ["type"] = "SimpleCommand",
         ["prefix"] = {
            [1] = {
               ["type"] = "Assignment",
               ["value"] = {
                  ["type"] = "Word",
                  ["content"] = {
                     [1] = "foo",
                  },
               },
               ["name"] = {
                  ["type"] = "Name",
                  ["text"] = "FOO",
               },
            },
         },
      },
      [2] = {
         ["type"] = "SimpleCommand",
         ["prefix"] = {
            [1] = {
               ["type"] = "Assignment",
               ["value"] = {
                  ["type"] = "Word",
                  ["content"] = {
                     [1] = "bar",
                  },
               },
               ["name"] = {
                  ["type"] = "Name",
                  ["text"] = "BAR",
               },
            },
         },
      },
      ["__comment"] = {
         ["type"] = "Comment",
         ["text"] = " comment 2",
      },
   },
   ["type"] = "Program",
}

• Is it wanted ?
• why not processing comment like any other shell command ?

I would like to get something like :

t = {
   ["body"] = {
      [1] = {type= "SimpleCommand", ...},
      [2] = {type= "Comment", ...},
      [3] = {type= "SimpleCommand", ...},
      [4] = {type= "Comment", ...},
   },
   ["type"] = "Program",
}

[jirutka]

Hi,

do you use the latest revision from the master branch? Do you use default AST builder sh-parser.ast.simple (e.g. via sh-parser.parser) or your custom? If the former, then pass { comments = true } as the second argument to the parse function.

Comments are handled specially. They are parsed, the parser calls create_node handler for them, but does not include the created nodes in captures. Comments may appear almost anywhere, so they would broke low-level API (based on positional captures) or complicate the grammar significantly.

If consumer of the low-level API wants to include comments, then it must store them aside and inject into appropriate AST node itself. That’s what sh-parser.ast.simple currently do; comments are eventually injected into the root node (Program).

[tst2005]

Hello,

I started with a simple sample, without custom stuff.
Then my sample was done with the sh-parser default grammar (that should be sh-parser.ast.simple).

For my sample Im using something like :

local sh = [[
FOO=bar
]]

local parse = require 'sh-parser.parser'.parse

local t = parse(sh)
local prettyprint = require "tprint"
print( "t = "..prettyprint(t) )

I used my own simple prettyprinter : tprint

I think there is 2 approach :

• parse to eval : then comment lost is not a problem)
• parse to rewrite like beautifier : then lost something is a problem

did you done sh-parser mainly for the 1st one ?

Regards,

[jirutka]

Try this:

  local sh = [[
  FOO=bar
  ]]

  local parse = require 'sh-parser.parser'.parse

- local t = parse(sh)
+ local t = parse(sh, { comments = true })
  local prettyprint = require "tprint"
  print( "t = "..prettyprint(t) )

You can read about available options in sh-parser.ast.imple.

did you done sh-parser mainly for the 1st one ?

No, the main motivation was programmatic modification of shell scripts, so comments are important for me too.

Please make sure that you use the latest version from the master. I had some problem with my deployment to LuaRocks, there should be up-to-date rock file now.

[tst2005]

Hello,

I'm using directly the git repository in the master branch, then I don't have issue with deployment.

I tried with { comment = true } and it is better, but far to be perfect.
I got :

t = {
    ["body"] = {
        [1] = {
            ["type"] = "SimpleCommand",
            ["prefix"] = {
                [1] = {
                    ["value"] = {
                        ["type"] = "Word",
                        ["content"] = {
                            [1] = "value1",
                        },
                    },
                    ["name"] = {
                        ["type"] = "Name",
                        ["text"] = "var1",
                    },
                    ["type"] = "Assignment",
                },
            },
        },
        [2] = {
            ["type"] = "SimpleCommand",
            ["prefix"] = {
                [1] = {
                    ["value"] = {
                        ["type"] = "Word",
                        ["content"] = {
                            [1] = "value2",
                        },
                    },
                    ["name"] = {
                        ["type"] = "Name",
                        ["text"] = "var2",
                    },
                    ["type"] = "Assignment",
                },
            },
        },
    },
    ["comments"] = {
        [1] = {
            ["type"] = "Comment",
            ["text"] = " comment line 1",
        },
        [2] = {
            ["type"] = "Comment",
            ["text"] = " comment line 3",
        },
    },
    ["type"] = "Program",
}

Now I have all comment lines in t.comments but still outside of the t.body.

• Why is it so hard to consider comment line like any other shell instruction ?

Do you know metalua and his ast format ?
IMHO the metalua's AST format seems the one that is closed to be a standard.

Are you interested to modify the ash result to follow the metalua format ?

• rename type to tag
• use numeric index instead of key index

I'm also starting to work with lpeg and lpeg.re.
Do you already know lpeg.re ?
Do you think it will hard try to define the shell posix grammar in lpeg.re syntax ?