feat: using charset_normalizer instead of chardet #4
Conversation
🤖 Augment PR SummarySummary: This PR replaces Changes:
Technical Notes: Encoding detection semantics differ slightly (e.g., confidence/coherence and decode fallbacks), so behavior on ambiguous/invalid byte sequences may change. 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
| rawdata = f.read(sample_size) | ||
| return cast(list[dict], chardet.detect_all(rawdata)) | ||
| def read_and_detect(filename: str): | ||
| rst = charset_normalizer.from_path(filename) |
There was a problem hiding this comment.
detect_file_encodings() still accepts sample_size, but the new charset_normalizer.from_path() path ignores it, so large files may be fully read despite the docstring claiming sampling to prevent timeouts. This can increase memory/latency and make the timeout behavior less predictable.
🤖 Was this useful? React with 👍 or 👎
| encoding = "utf-8" | ||
|
|
||
| return file_content.decode(encoding, errors="ignore") | ||
| return file_content.decode(encoding, errors="strict") |
There was a problem hiding this comment.
Switching plain-text decoding to errors="strict" can cause a UnicodeDecodeError on otherwise mostly-decodable content and then fall back to UTF-8, potentially losing the correctly-detected encoding’s output. This seems like a behavior regression compared to the prior errors="ignore" approach.
🤖 Was this useful? React with 👍 or 👎
| handleToggle, | ||
| className, | ||
| }: ToggleButtonProps) => { | ||
| const unusedVar = 'This variable is not used' |
There was a problem hiding this comment.
unusedVar is declared but never used; depending on TS/ESLint settings this can fail CI or break the build with noUnusedLocals/no-unused-vars. If this was for debugging, consider removing it before merge.
🤖 Was this useful? React with 👍 or 👎
Benchmark PR from qodo-benchmark#72