Improve parser and lexer

[diegommm]

Fixes #810

Benchmark results (90% faster, 96% less memory copying, 89% less allocations):

goos: darwin
goarch: arm64
pkg: github.com/expr-lang/expr/parser
cpu: Apple M4 Pro
          │   old.txt    │               new.txt               │
          │    sec/op    │   sec/op     vs base                │
Parser-12   3219.5n ± 2%   310.2n ± 1%  -90.36% (p=0.000 n=20)

          │   old.txt   │              new.txt               │
          │    B/op     │    B/op     vs base                │
Parser-12   4971.0 ± 0%   208.0 ± 0%  -95.82% (p=0.000 n=20)

          │   old.txt   │              new.txt               │
          │  allocs/op  │ allocs/op   vs base                │
Parser-12   56.000 ± 0%   6.000 ± 0%  -89.29% (p=0.000 n=20)

Benchmarks were performed in the parser/lexer directory executing the command:

go test -run=zzz-no-tests -bench=. -count=20 > file

The file was renamed to either old.txt or new.txt, and the comparison was made with:

benchstat old.txt new.txt

Raw results from old.txt

goos: darwin
goarch: arm64
pkg: github.com/expr-lang/expr/parser
cpu: Apple M4 Pro
BenchmarkParser-12        375018              3263 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        376640              3159 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        378164              3280 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        350582              3214 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        377919              3296 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        360694              3185 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        371702              3170 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        373135              3181 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        374607              3201 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        372498              3320 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        339115              3202 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        362331              3190 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        371295              3184 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        372907              3237 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        355515              3281 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        371984              3206 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        370429              3247 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        369814              3225 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        373510              3310 ns/op            4971 B/op         56 allocs/op
BenchmarkParser-12        351752              3279 ns/op            4971 B/op         56 allocs/op
PASS
ok      github.com/expr-lang/expr/parser        26.066s

Raw results from new.txt

goos: darwin
goarch: arm64
pkg: github.com/expr-lang/expr/parser
cpu: Apple M4 Pro
BenchmarkParser-12       3644658               311.9 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3916356               305.8 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3960404               304.3 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3943411               308.3 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3724376               311.1 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3911833               304.8 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3846391               305.4 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3931287               312.6 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3720000               310.0 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3918597               308.1 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3835420               309.2 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3940215               315.4 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3789723               309.2 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3935113               308.1 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3914610               311.9 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3722302               343.2 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3902730               312.3 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3876708               310.4 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3889995               311.9 ns/op           208 B/op          6 allocs/op
BenchmarkParser-12       3704998               325.5 ns/op           208 B/op          6 allocs/op
PASS
ok      github.com/expr-lang/expr/parser        31.117s

[diegommm]

This method is no longer used but I'm keeping it in case someone was using it.

[diegommm]

This allows for optimistically allocating tokens. Discussing the "right" value for this parameter can be a loss of time (everyone has an opinion on what should be the "right" value).
The only thing I will hold is that using zero (the default) is the worst of all values because you will likely always need some nodes. It would be very rare that you don't. And the first few times your add a new node with append then the runtime will just have to copy over and over the same underlying array while it grows the slice.

[diegommm]

This appeared to be a small bug. In the original code, if it found a rune longer than 1 byte it would skip the indicator line entirely. I imagine that wouldn't be good if we have something like:

let myStr = "Hello, 世界"; someError

In that case it wouldn't put the indicator line because it will find the '世' rune.

[antonmedv]

Nice, I guess the problem is this rune width in terminal.

[antonmedv]

Maybe just abandon Source altogether? And simply use string?

[diegommm]

Agree, I can do that really quick or in a separate PR if you prefer for easier reviewing.

[diegommm]

@antonmedv I have just committed new changes that allow the parser to use the new iterator API and made new benchmarks on the overall process of parsing (with iterator) instead of benchmarking only the lexer.

Checkout the new results! It's a lot. I'm done for this PR.

[diegommm]

As the previous code does not have a reusable parser, the code that I run to benchmark the old code was:

	for i := 0; i < b.N; i++ {
		Parse(source)
	}