Commit dd4573d
committed
Fix issue caused by pretokenising of UTF8 strings (which get converted after the pretokenise step and cause the length to be wrong)
Even if length were correct, pretokenising the string stops the scan for UTF8 char patterns so the string wouldn't be treated correctly1 parent 12f7edd commit dd4573d
1 file changed
+1
-1
lines changed
0 commit comments