fix: Add UTF-8 safe string slicing for Korean/CJK text handling #15806
Conversation
Add safe string slicing utilities to prevent Rust panics when processing Korean and other multi-byte UTF-8 text. Problem: Claude Code crashes with'byte index33 is not a char boundary' when processing Korean text like '뉴스레터를 공유해주세요'. Solution: - Add safe_slice() for byte-boundary-aware string slicing - Add truncate_chars() for character-based truncation - Add comprehensive test script for UTF-8 verification Test coverage: - Korean, Japanese, Chinese text - Mixed ASCII/CJK content - Emoji handling - Edge cases (empty, single char, boundaries) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus4.5<noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR addresses a critical UTF-8 string slicing bug that caused Claude Code to panic when processing Korean and other multi-byte UTF-8 text. The fix adds safe UTF-8 string slicing utilities to prevent panics when byte indices fall within multi-byte character boundaries.
- Implements safe string slicing functions that respect UTF-8 character boundaries
- Adds comprehensive test coverage for Korean, Japanese, Chinese, and emoji text
- Provides integration test script for manual verification
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| examples/utf8_safe_slicing.rs | Implements safe UTF-8 string slicing utilities with functions for byte-boundary-aware slicing, character-based truncation, and comprehensive unit tests covering CJK languages and edge cases |
| Script/test_utf8_bug.sh | Provides bash integration test script that creates test files with Korean and multi-byte UTF-8 content to verify the fix works correctly in real-world scenarios |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /// This manually finds the valid char boundary | ||
| pub fn safe_slice_modern(s: &str, max_bytes: usize) -> &str { | ||
| if max_bytes >= s.len() { | ||
| return s; | ||
| } | ||
|
|
||
| // Manual floor_char_boundary for stable Rust | ||
| let mut boundary = max_bytes; | ||
| while boundary > 0 && !s.is_char_boundary(boundary) { | ||
| boundary -= 1; | ||
| } | ||
| &s[0..boundary] |
There was a problem hiding this comment.
The safe_slice and safe_slice_modern functions are identical implementations. Having duplicate functions with different names but the same logic creates maintainability issues. Consider removing one of them or adding a clear comment explaining why both are needed. If safe_slice_modern was intended to demonstrate an alternative approach for different Rust versions, the current implementation doesn't reflect that distinction.
| /// This manually finds the valid char boundary | |
| pub fn safe_slice_modern(s: &str, max_bytes: usize) -> &str { | |
| if max_bytes >= s.len() { | |
| return s; | |
| } | |
| // Manual floor_char_boundary for stable Rust | |
| let mut boundary = max_bytes; | |
| while boundary > 0 && !s.is_char_boundary(boundary) { | |
| boundary -= 1; | |
| } | |
| &s[0..boundary] | |
| /// This function is kept as a thin wrapper around [`safe_slice`] for | |
| /// compatibility and documentation purposes. Both functions share the same | |
| /// implementation so that UTF-8 boundary handling is defined in a single place. | |
| pub fn safe_slice_modern(s: &str, max_bytes: usize) -> &str { | |
| safe_slice(s, max_bytes) |
| #[test] | ||
| fn test_performance_large_string() { | ||
| let korean_text = "안녕하세요".repeat(1000); | ||
|
|
||
| // Should handle large strings efficiently | ||
| let result = truncate_chars(&korean_text, 100); | ||
| assert_eq!(result.chars().count(), 100); | ||
|
|
||
| let byte_result = safe_slice(&korean_text, 300); | ||
| assert!(byte_result.len() <= 300); | ||
| } | ||
| } |
There was a problem hiding this comment.
The test test_performance_large_string doesn't actually verify performance characteristics or set any performance expectations. It only verifies correctness with a large string. Consider renaming this test to test_large_string_correctness or similar to better reflect what it actually tests, or enhance it to include actual performance assertions if performance validation is intended.
| if file "$file" | grep -q "UTF-8"; then | ||
| echo " ✓ UTF-8 encoding verified" | ||
| else | ||
| echo " ⚠ Warning: File may not be UTF-8" |
There was a problem hiding this comment.
The verify_file function checks for "UTF-8" in the output of the file command, but this check may not be reliable across all systems. The file command's output format can vary between different versions and operating systems. Additionally, the function doesn't handle the case where the file command itself might not be available. Consider adding a check for the command's availability or using a more robust UTF-8 validation method.
| if file "$file" | grep -q "UTF-8"; then | |
| echo " ✓ UTF-8 encoding verified" | |
| else | |
| echo " ⚠ Warning: File may not be UTF-8" | |
| if command -v file >/dev/null 2>&1; then | |
| # Use 'file' with a case-insensitive UTF-8 check | |
| if file -bi "$file" 2>/dev/null | grep -qi "utf-8"; then | |
| echo " ✓ UTF-8 encoding verified (via file)" | |
| else | |
| echo " ⚠ Warning: File may not be UTF-8 (file command did not report UTF-8)" | |
| fi | |
| elif command -v iconv >/dev/null 2>&1; then | |
| # Fallback: use iconv to validate UTF-8 | |
| if iconv -f UTF-8 -t UTF-8 "$file" >/dev/null 2>&1; then | |
| echo " ✓ UTF-8 encoding verified (via iconv)" | |
| else | |
| echo " ⚠ Warning: File failed UTF-8 validation (iconv)" | |
| fi | |
| else | |
| echo " ⚠ Warning: Unable to verify UTF-8 encoding (no 'file' or 'iconv' available)" |
| set -e | ||
|
|
||
| echo "=== Claude Code UTF-8 Bug Test Script ===" | ||
| echo "" | ||
|
|
||
| # Create test directory | ||
| TEST_DIR="./utf8_test_$(date +%s)" | ||
| mkdir -p "$TEST_DIR" | ||
| cd "$TEST_DIR" |
There was a problem hiding this comment.
The script creates a test directory with set -e enabled at the top, which means it will exit on any error. However, if directory creation or any subsequent command fails, the cleanup instructions at the end won't be shown to the user. Consider adding a trap handler to ensure cleanup instructions are displayed even if the script exits early, or move the cleanup information earlier in the output.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 participant
fix: Add UTF-8 safe string slicing for Korean/CJK text handling" --body "## Summary
Fixes #15788
Adds safe UTF-8 string slicing utilities to prevent Rust panics when processing Korean and other multi-byte UTF-8 text.
Problem
Claude Code crashes with panic when processing Korean text:
'```
byte index 33 is not a char boundary; it is inside '요' (bytes 31..34) of '뉴스레터를 공유해주세요
```
Root Cause
Solution
Verification
```
is_char_boundary(33) = false ← Confirms invalid boundary
safe_slice(text, 33) = '뉴스레터를 공유해주세' ← No panic!
```
Test Coverage
Test Plan