Incorrect string syntax highlighting in attributes in VS Code

[brendanzab]

When using logos, I get the following broken syntax highlighting:

Original source for replication purposes:

#[derive(Debug, Clone, Logos)]
enum Quoted<'source> {
    #[regex(r#"[^\\"']+"#)]
    Text(&'source str),
    #[token("\\")]
    StartEscape,
    #[token("\'", |_| Quote::Single)]
    #[token("\"", |_| Quote::Double)]
    End(Quote),

    #[error]
    Error,
}

#[derive(Debug, Clone, Logos)]
enum Escape {
    #[token("\\", |_| '\\')]
    #[token("n", |_| '\n')]
    #[token("r", |_| '\r')]
    #[token("t", |_| '\t')]
    #[token("0", |_| '\0')]
    #[token("\'", |_| '\'')]
    #[token("\"", |_| '\"')]
    Single(char),
    #[token("u")]
    StartUnicodeEscape,
    #[token("x")]
    StartAsciiEscape,

    #[error]
    Error,
}

#[derive(Debug, Clone, Logos)]
enum UnicodeEscape<'source> {
    #[regex(r"\{[0-9a-fA-F]*\}", |lexer| &lexer.slice()[1..(lexer.slice().len() - 1)])]
    CharCode(&'source str),
    #[token("\'", |_| Quote::Single)]
    #[token("\"", |_| Quote::Double)]
    End(Quote),

    #[error]
    Error,
}

A workaround (suggested by @Kixiron and seen in the wild in Crunch) is to add // workaround highlighting: " to the offending lines:

 #[derive(Debug, Copy, Clone, PartialEq)]
 enum Quote {
     Single,
@@ -33,7 +61,7 @@ enum QuotedLiteral {

 #[derive(Debug, Clone, Logos)]
 enum Quoted<'source> {
-    #[regex(r#"[^\\"']+"#)]
+    #[regex(r#"[^\\"']+"#)] // workaround highlighting: "
     Text(&'source str),
     #[token("\\")]
     StartEscape,
@@ -53,7 +81,7 @@ enum Escape {
     #[token("t", |_| '\t')]
     #[token("0", |_| '\0')]
     #[token("\'", |_| '\'')]
-    #[token("\"", |_| '\"')]
+    #[token("\"", |_| '\"')] // workaround highlighting: "
     Single(char),
     #[token("u")]
     StartUnicodeEscape,

[bjorn3]

Does this also happen when the rust-analyzee extension is disabled? If so then this issue is likely caused by an incorrect regex in the language definition of vscode.

[brendanzab]

Ahh yeah, it seems like it is still doing something similar with rust-analyzer disabled:

[lnicola]

I think there's another issue about the same thing. But is the built-in grammar still used if semantic highlighting is enabled?

[bjorn3]

The rust-analyzer extension ships with a grammar that is I believe adapted from the built-in grammar of vscode. I think it only has some tweaks to better make it match what semantic highlighting would output. (#4397) I am not sure if semantic highlighting enhances or replaces the output of the vscode grammar.

[sloane-shark]

Hi all! I'm not sure if this problem is necessarily caused by rust-analyzer, but it seems like maybe rust-analyzer could fix it? VS Code ships with a rust grammar that is pulled from atom which incorrectly excludes raw string literals from attributes. I have opened a PR here, but as far as I can tell it hasn't been noticed or picked up. I'm using a local version of this change to make my own syntax highlighting work properly. Perhaps this could be implemented in rust-analyzer with semantic highlighting?

[brendanzab]

Yeah, seems like the project is looking for maintainers? zargony/atom-language-rust#144

[matklad]

@dustypomerleau is this fixed by the new grammar?

[dustypomerleau]

@matklad Nearly. Cases 2 and 3 are ok:

Case 1 still needs a little work:

The way that strings and chars are included in attributes is allowing them to gobble up the closing ] of the attribute itself. I will have a look.