Fix generating partially valid tokens#3

Open

mattiasarro wants to merge 2 commits intor2d4:mainfrom

mattiasarro:main

Commits on May 30, 2023

ReTokenFilter.is_valid_token: partial=False
mattiasarro
committed

Commits on May 31, 2023

fix handling of partial tokens
mattiasarro
committed