reduce the amount of panics in {TokenStream, Literal}::from_str calls
#147859
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
|
rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead. cc @rust-lang/rust-analyzer |
rustbot
added
S-waiting-on-review
Collaborator
|
r? @davidtwco rustbot has assigned @davidtwco. Use |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Contributor
Author
|
@rustbot author |
rustbot
added
S-waiting-on-author
Collaborator
|
Reminder, once the PR becomes ready for a review, use |
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
e2c0009 to
80b67bb
Compare
Contributor
Author
|
@rustbot ready |
rustbot
added
S-waiting-on-review
bjorn3
reviewed
Oct 20, 2025
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
ba098bc to
0274d78
Compare
Contributor
Author
|
This does not fix #58736 entirely yet: TokenStream::from_str("0b2");is still a compile error because it internally calls |
Member
|
I'm not familiar enough with this part of the compiler r? compiler |
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
0274d78 to
13d3d06
Compare
This comment has been minimized.
This comment has been minimized.
cyrgani
changed the title
TokenStream::from_str panics with LexErrors{TokenStream, Literal}::from_str calls
jhpratt
added a commit
to jhpratt/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
jhpratt
added a commit
to jhpratt/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
Zalathar
added a commit
to Zalathar/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
JaclynCodes
added a commit
to JaclynCodes/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
13d3d06 to
faf91ab
Compare
This comment has been minimized.
This comment has been minimized.
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
faf91ab to
bfa95c7
Compare
This comment has been minimized.
This comment has been minimized.
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
bfa95c7 to
599d183
Compare
Contributor
Author
|
r? compiler |
Member
|
I've only skimmed the PR description but this might actually help us lex frontmatter again in Edit: Well, it'll still run into the "manifold negation regression" ig, hmph. |
Collaborator
|
☔ The latest upstream changes (presumably #147498) made this pull request unmergeable. Please resolve the merge conflicts. |
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
599d183 to
91a861c
Compare
This comment has been minimized.
This comment has been minimized.
Member
|
@rustbot reroll |
JonathanBrouwer
requested changes
Nov 29, 2025
rustbot
added
S-waiting-on-author
Collaborator
|
☔ The latest upstream changes (presumably #149499) made this pull request unmergeable. Please resolve the merge conflicts. |
rust-cloud-vms
bot
force-pushed
the
nonfatal-tokenstream-parse
branch
from
91a861c to
24b96f1
Compare
Collaborator
|
This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
cyrgani
added
S-waiting-on-review
rust-cloud-vms
bot
force-pushed
the
nonfatal-tokenstream-parse
branch
from
24b96f1 to
d847df3
Compare
Contributor
|
I think there is still work to do in improving the error messages, but this can happen in future PRs, this is already a big improvement. Once that is clear, r=me |
Contributor
|
@rustbot author |
rustbot
added
S-waiting-on-author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this PR, calling
TokenStream::from_strorLiteral::from_strwith an invalid argument would always cause a compile error, even if theTokenStreamis not used afterwards at all.This PR changes this so it returns a
LexErrorinstead in some cases.This is very theoretically a breaking change, but the doc comment on the impl already says
Fixes some cases of #58736.