⚡️ Speed up method BasicTokenizer._tokenize_chinese_chars by 12%

[codeflash-ai]

📄 12% (0.12x) speedup for BasicTokenizer._tokenize_chinese_chars in src/transformers/models/prophetnet/tokenization_prophetnet.py

⏱️ Runtime : 1.60 milliseconds → 1.43 milliseconds (best of 209 runs)

📝 Explanation and details

The optimized code achieves an 11% speedup through two key optimizations in the _tokenize_chinese_chars method:

1. Method Localization: The optimization moves frequently accessed methods into local variables (is_chinese_char = self._is_chinese_char and append = output.append). This eliminates the overhead of attribute lookups during the tight loop over each character. In Python, local variable access is faster than attribute access because it avoids the attribute resolution mechanism.

2. Precomputed Range Storage: The Chinese character ranges are now stored as tuples in self._chinese_char_ranges during initialization rather than being hardcoded in the _is_chinese_char method. While the range checking logic remains the same, this preparation centralizes the Unicode block definitions and slightly improves cache locality.

Performance Impact: The test results show the optimization is most effective for:

• Large texts with many Chinese characters (19.3% faster for 500 Chinese chars)
• Mixed Chinese/ASCII content (13.6-14.4% faster for large mixed texts)
• Moderate improvements (2-7%) for shorter texts with Chinese characters

The optimization provides minimal overhead for edge cases like empty strings or ASCII-only text, with some showing negligible slowdown due to the additional local variable assignments.

Why It Works: The speedup comes primarily from reducing Python's attribute lookup overhead during character iteration. Since _tokenize_chinese_chars processes text character-by-character and calls _is_chinese_char and output.append for each character, eliminating repeated attribute lookups provides measurable performance gains, especially for longer texts where the loop executes hundreds or thousands of times.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 141 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

🌀 Generated Regression Tests and Runtime

from transformers.models.prophetnet.tokenization_prophetnet import BasicTokenizer


# unit tests

# --- Basic Test Cases ---


def test_ascii_only():
    # No CJK characters, should be unchanged
    tokenizer = BasicTokenizer()
    text = "Hello, world!"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 3.69μs -> 3.69μs (0.108% slower)


def test_single_chinese_char():
    # Single Chinese character should be surrounded by spaces
    tokenizer = BasicTokenizer()
    text = "你"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 1.87μs -> 1.80μs (3.83% faster)


def test_chinese_in_english():
    # Chinese character in the middle of English text
    tokenizer = BasicTokenizer()
    text = "Hello你world"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 3.90μs -> 3.75μs (4.08% faster)


def test_multiple_chinese_chars():
    # Multiple adjacent Chinese characters
    tokenizer = BasicTokenizer()
    text = "你好世界"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.77μs -> 2.70μs (2.63% faster)


def test_chinese_and_punctuation():
    # Chinese character followed by punctuation
    tokenizer = BasicTokenizer()
    text = "你,好!"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.79μs -> 2.69μs (4.02% faster)


def test_chinese_and_whitespace():
    # Chinese characters with whitespace
    tokenizer = BasicTokenizer()
    text = "你 好"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.43μs -> 2.44μs (0.328% slower)


def test_mixed_chinese_and_nonchinese():
    # Mixed Chinese and non-Chinese characters
    tokenizer = BasicTokenizer()
    text = "abc你def"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 3.18μs -> 2.98μs (6.68% faster)


def test_chinese_char_in_fullwidth_block():
    # CJK Compatibility Ideographs (e.g. U+FA0E)
    tokenizer = BasicTokenizer()
    text = "\ufa0e"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 1.91μs -> 1.90μs (0.263% faster)


def test_chinese_char_in_extension_b():
    # CJK Unified Ideographs Extension B (e.g. U+20000)
    tokenizer = BasicTokenizer()
    text = "\U00020000"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 1.99μs -> 1.98μs (0.861% faster)


# --- Edge Test Cases ---


def test_empty_string():
    # Empty string should return empty string
    tokenizer = BasicTokenizer()
    text = ""
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 697ns -> 868ns (19.7% slower)


def test_only_spaces():
    # String with only spaces should be unchanged
    tokenizer = BasicTokenizer()
    text = "     "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.20μs -> 2.27μs (2.91% slower)


def test_non_cjk_unicode():
    # Non-CJK unicode characters (e.g. emoji, Greek, Cyrillic)
    tokenizer = BasicTokenizer()
    text = "αβγ😊ЖЗИ"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 3.57μs -> 3.53μs (0.934% faster)


def test_japanese_hiragana_and_katakana():
    # Japanese Hiragana and Katakana should NOT be surrounded by spaces
    tokenizer = BasicTokenizer()
    text = "あいうアイウ"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.78μs -> 2.79μs (0.573% slower)


def test_korean_hangul():
    # Korean Hangul should NOT be surrounded by spaces
    tokenizer = BasicTokenizer()
    text = "한글"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.09μs -> 2.18μs (4.04% slower)


def test_surrogate_pairs():
    # Surrogate pairs for CJK Extension B (e.g. U+20000)
    tokenizer = BasicTokenizer()
    text = "a\U00020000b"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.68μs -> 2.68μs (0.149% faster)


def test_adjacent_cjk_and_non_cjk():
    # CJK character adjacent to non-CJK unicode
    tokenizer = BasicTokenizer()
    text = "你😊"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.34μs -> 2.35μs (0.468% slower)


def test_cjk_at_string_boundaries():
    # CJK character at start and end
    tokenizer = BasicTokenizer()
    text = "你abc界"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 3.19μs -> 3.08μs (3.77% faster)


def test_cjk_compatibility_ideographs():
    # CJK Compatibility Ideographs (U+F900-U+FAFF)
    tokenizer = BasicTokenizer()
    text = "\uf900\ufaff"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.57μs -> 2.47μs (3.80% faster)


def test_cjk_extension_blocks():
    # Test all extension blocks with one char from each
    tokenizer = BasicTokenizer()
    chars = [
        "\u4e00",  # Unified Ideographs
        "\u3400",  # Extension A
        "\U00020000",  # Extension B
        "\U0002a700",  # Extension C
        "\U0002b740",  # Extension D
        "\U0002b820",  # Extension E
        "\U0002ceaf",  # Extension F
        "\uf900",  # Compatibility Ideograph
        "\U0002f800",  # Compatibility Supplement
        "\U0002fa1f",  # Compatibility Supplement end
    ]
    text = "".join(chars)
    expected = "".join(f" {c} " for c in chars)
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 5.26μs -> 5.11μs (3.11% faster)


def test_multiple_spaces_between_cjk():
    # Multiple spaces between CJK characters
    tokenizer = BasicTokenizer()
    text = "你  好"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.78μs -> 2.75μs (0.835% faster)


def test_combining_characters():
    # Combining characters should not be treated as CJK
    tokenizer = BasicTokenizer()
    text = "a\u0301b"  # a + combining acute accent + b
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.13μs -> 2.15μs (1.07% slower)


# --- Large Scale Test Cases ---


def test_long_ascii_string():
    # Long ASCII string, should be unchanged
    tokenizer = BasicTokenizer()
    text = "a" * 1000
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 142μs -> 128μs (10.7% faster)


def test_long_cjk_string():
    # Long string of Chinese characters
    tokenizer = BasicTokenizer()
    text = "你" * 1000
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 147μs -> 124μs (18.2% faster)
    # Each char surrounded by spaces
    expected = " ".join([""] + ["你"] * 1000 + [""])
    expected = expected.replace(" ", " ")  # keep as is


def test_mixed_large_string():
    # Large string with alternating Chinese and ASCII characters
    tokenizer = BasicTokenizer()
    text = "你a" * 500  # 1000 chars, 500 Chinese, 500 ASCII
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 144μs -> 127μs (13.9% faster)
    # Should be " 你 a 你 a ..." (spaces around each 你)
    expected = " 你 a" * 500


def test_large_string_with_various_blocks():
    # Large string with CJK chars from different blocks and other unicode
    tokenizer = BasicTokenizer()
    cjk_blocks = [
        "\u4e00",  # Unified Ideographs
        "\u3400",  # Extension A
        "\U00020000",  # Extension B
        "\U0002a700",  # Extension C
        "\U0002b740",  # Extension D
        "\U0002b820",  # Extension E
        "\U0002ceaf",  # Extension F
        "\uf900",  # Compatibility Ideograph
        "\U0002f800",  # Compatibility Supplement
        "\U0002fa1f",  # Compatibility Supplement end
    ]
    text = "".join(cjk_blocks * 50)  # 500 chars
    text += "abc" * 100  # add 300 ASCII
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 153μs -> 138μs (10.8% faster)
    # All CJK chars should be surrounded by spaces, ascii unchanged
    expected = "".join(f" {c} " for c in cjk_blocks * 50) + "abc" * 100


def test_performance_on_large_input():
    # Test that function runs efficiently on 1000 chars
    import time

    tokenizer = BasicTokenizer()
    text = "你a" * 500
    start = time.time()
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 144μs -> 127μs (12.9% faster)
    end = time.time()


# --- Negative/Mutation-Detection Test Cases ---


def test_mutation_detection_removal_of_spaces():
    # If the function fails to add spaces, the test will fail
    tokenizer = BasicTokenizer()
    text = "你abc好"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.96μs -> 2.89μs (2.52% faster)


def test_mutation_detection_wrong_block():
    # If the function incorrectly marks non-CJK as CJK, the test will fail
    tokenizer = BasicTokenizer()
    text = "α你β"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.42μs -> 2.44μs (1.02% slower)


def test_mutation_detection_double_spaces():
    # If the function adds double spaces, the test will fail
    tokenizer = BasicTokenizer()
    text = "你你"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 2.01μs -> 2.14μs (6.09% slower)


def test_mutation_detection_missing_last_char():
    # If the function omits the last char, test will fail
    tokenizer = BasicTokenizer()
    text = "你"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 1.60μs -> 1.62μs (1.42% slower)


def test_mutation_detection_strip():
    # If the function strips the result, test will fail
    tokenizer = BasicTokenizer()
    text = "你"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 1.63μs -> 1.61μs (1.18% faster)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

import pytest  # used for our unit tests

from transformers.models.prophetnet.tokenization_prophetnet import BasicTokenizer


# unit tests

# ---------------------- BASIC TEST CASES ----------------------


def test_empty_string():
    # Empty string should return empty string
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("")  # 621ns -> 821ns (24.4% slower)


def test_ascii_only():
    # ASCII text should remain unchanged
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("hello world!")  # 3.48μs -> 3.41μs (1.96% faster)


def test_single_chinese_char():
    # Single Chinese character should be surrounded by spaces
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("你")  # 1.59μs -> 1.69μs (5.92% slower)


def test_chinese_char_in_sentence():
    # Chinese character embedded in English sentence
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("hello你world")  # 3.88μs -> 3.69μs (5.04% faster)


def test_multiple_chinese_chars():
    # Multiple Chinese characters together
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("你好")  # 2.17μs -> 2.24μs (2.86% slower)


def test_chinese_and_ascii_mixed():
    # Mixed Chinese and ASCII chars
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("A你B好C")  # 2.96μs -> 2.89μs (2.32% faster)


def test_chinese_char_with_space():
    # Chinese character with spaces around
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars(" 你 ")  # 2.28μs -> 2.31μs (1.13% slower)


def test_chinese_char_with_punctuation():
    # Chinese character with punctuation
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("你,好!")  # 2.65μs -> 2.65μs (0.151% faster)


def test_chinese_char_with_multiple_punctuations():
    # Multiple Chinese chars and punctuation
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("你，好！")  # 2.92μs -> 2.85μs (2.74% faster)


def test_chinese_char_with_digits():
    # Chinese char with digits
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("你123好")  # 3.00μs -> 3.01μs (0.366% slower)


def test_non_cjk_unicode():
    # Non-CJK Unicode chars (e.g. Japanese Hiragana, Korean Hangul) should not be surrounded
    tokenizer = BasicTokenizer()
    hiragana = "\u3042"  # あ
    hangul = "\uac00"  # 가
    codeflash_output = tokenizer._tokenize_chinese_chars(hiragana)  # 1.33μs -> 1.39μs (4.38% slower)
    codeflash_output = tokenizer._tokenize_chinese_chars(hangul)  # 657ns -> 757ns (13.2% slower)


# ---------------------- EDGE TEST CASES ----------------------


def test_surrogate_pairs():
    # Chinese character outside BMP (surrogate pairs)
    tokenizer = BasicTokenizer()
    # U+20000 is a CJK Unified Ideograph Extension B character
    char = "\U00020000"
    codeflash_output = tokenizer._tokenize_chinese_chars(char)  # 1.95μs -> 1.91μs (1.88% faster)


def test_mixed_cjk_blocks():
    # Chinese chars from different CJK blocks
    tokenizer = BasicTokenizer()
    chars = [
        "\u4e00",  # CJK Unified Ideographs
        "\u3400",  # CJK Unified Ideographs Extension A
        "\uf900",  # CJK Compatibility Ideographs
        "\U00020000",  # CJK Unified Ideographs Extension B
        "\U0002a700",  # CJK Unified Ideographs Extension C
        "\U0002b740",  # CJK Unified Ideographs Extension D
        "\U0002b820",  # CJK Unified Ideographs Extension E
        "\U0002f800",  # CJK Compatibility Ideographs Supplement
    ]
    text = "".join(chars)
    expected = "".join(f" {c} " for c in chars)
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 4.54μs -> 4.43μs (2.57% faster)


def test_adjacent_chinese_chars():
    # Adjacent Chinese chars should each get their own spaces
    tokenizer = BasicTokenizer()
    text = "你们好"
    expected = " 你 们 好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.50μs -> 2.41μs (3.70% faster)


def test_chinese_char_at_start_and_end():
    # Chinese char at beginning and end
    tokenizer = BasicTokenizer()
    text = "你hello好"
    expected = " 你 hello 好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 3.28μs -> 3.31μs (0.907% slower)


def test_only_spaces():
    # Only spaces should remain unchanged
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("     ")  # 2.08μs -> 2.19μs (4.62% slower)


def test_only_punctuation():
    # Only punctuation should remain unchanged
    tokenizer = BasicTokenizer()
    codeflash_output = tokenizer._tokenize_chinese_chars("!@#$%^&*()")  # 3.11μs -> 3.14μs (1.11% slower)


def test_non_printable_chars():
    # Non-printable chars should remain unchanged
    tokenizer = BasicTokenizer()
    text = "\n\t\r"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 1.77μs -> 1.84μs (3.65% slower)


def test_unicode_combining_chars():
    # Unicode combining marks should not be surrounded
    tokenizer = BasicTokenizer()
    text = "a\u0301"  # á (a + combining acute accent)
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.01μs -> 1.98μs (1.52% faster)


def test_mixed_non_cjk_blocks():
    # Mixed with Greek, Cyrillic, Arabic, etc.
    tokenizer = BasicTokenizer()
    text = "αбج你好"
    expected = "αбج 你 好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 3.20μs -> 3.12μs (2.83% faster)


def test_long_repeating_chinese_chars():
    # Long string of repeating Chinese chars
    tokenizer = BasicTokenizer()
    text = "你" * 10
    expected = " " + " 你 " * 10
    expected = expected.strip()
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 3.82μs -> 3.61μs (5.76% faster)


def test_chinese_char_with_emoji():
    # Chinese char with emoji
    tokenizer = BasicTokenizer()
    text = "你😊好"
    expected = " 你 😊好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.88μs -> 2.88μs (0.104% faster)


def test_chinese_char_with_control_chars():
    # Chinese char with control characters
    tokenizer = BasicTokenizer()
    text = "你\n好\t"
    expected = " 你 \n好\t"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.65μs -> 2.72μs (2.65% slower)


def test_chinese_char_with_surrounding_spaces():
    # Chinese char with spaces before and after
    tokenizer = BasicTokenizer()
    text = " 你 好 "
    expected = " 你  好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.77μs -> 2.85μs (2.85% slower)


def test_chinese_char_with_nonbreaking_space():
    # Chinese char with non-breaking space
    tokenizer = BasicTokenizer()
    text = "\u00a0你\u00a0"
    expected = "\u00a0 你 \u00a0"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.54μs -> 2.47μs (3.04% faster)


# ---------------------- LARGE SCALE TEST CASES ----------------------


def test_large_ascii_text():
    # Large ASCII text should remain unchanged
    tokenizer = BasicTokenizer()
    text = "a" * 1000
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 141μs -> 128μs (10.1% faster)


def test_large_chinese_text():
    # Large Chinese text should have spaces around each char
    tokenizer = BasicTokenizer()
    text = "你" * 500
    expected = " " + " 你 " * 500
    expected = expected.strip()
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 75.8μs -> 63.6μs (19.3% faster)


def test_large_mixed_text():
    # Large text with alternating Chinese and ASCII
    tokenizer = BasicTokenizer()
    text = "".join("你a" for _ in range(500))
    expected = "".join(" 你 a" for _ in range(500))
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 144μs -> 127μs (13.6% faster)


def test_large_text_with_all_cjk_blocks():
    # Large text with all CJK blocks and some ASCII
    tokenizer = BasicTokenizer()
    chars = ["\u4e00", "\u3400", "\uf900", "\U00020000", "\U0002a700", "\U0002b740", "\U0002b820", "\U0002f800"]
    text = "".join(chars * 100) + "abc" * 100
    expected = "".join(f" {c} " for c in chars * 100) + "abc" * 100
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 210μs -> 187μs (11.9% faster)


def test_performance_large_text():
    # Performance: Should not take excessive time for 1000 chars
    import time

    tokenizer = BasicTokenizer()
    text = "你" * 500 + "a" * 500
    start = time.time()
    codeflash_output = tokenizer._tokenize_chinese_chars(text)
    result = codeflash_output  # 145μs -> 127μs (14.4% faster)
    end = time.time()
    # Check correctness
    expected = " " + " 你 " * 500
    expected = expected.strip() + "a" * 500


# ---------------------- ADDITIONAL EDGE CASES ----------------------


def test_none_input_raises():
    # Passing None should raise TypeError
    tokenizer = BasicTokenizer()
    with pytest.raises(TypeError):
        tokenizer._tokenize_chinese_chars(None)  # 1.03μs -> 1.27μs (18.7% slower)


def test_non_str_input_raises():
    # Passing non-str types should raise TypeError
    tokenizer = BasicTokenizer()
    with pytest.raises(TypeError):
        tokenizer._tokenize_chinese_chars(123)
    with pytest.raises(TypeError):
        tokenizer._tokenize_chinese_chars(["你", "好"])


def test_chinese_char_with_surrogate_escape():
    # Surrogate-escaped string should not crash
    tokenizer = BasicTokenizer()
    text = b"\xed\xa0\x81".decode("utf-8", errors="ignore")  # invalid surrogate, ignored
    # Should just return empty string
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 769ns -> 986ns (22.0% slower)


def test_chinese_char_with_combining_and_non_cjk():
    # Chinese char with combining marks and non-CJK unicode
    tokenizer = BasicTokenizer()
    text = "你\u0301好"  # 你 + combining acute accent + 好
    expected = " 你́ 好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 3.02μs -> 2.81μs (7.47% faster)


def test_chinese_char_with_multiple_spaces_between():
    # Chinese chars with multiple spaces between
    tokenizer = BasicTokenizer()
    text = "你  好"
    expected = " 你   好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.89μs -> 2.86μs (1.30% faster)


def test_chinese_char_with_tab_and_newline():
    # Chinese chars with tab and newline between
    tokenizer = BasicTokenizer()
    text = "你\t好\n"
    expected = " 你 \t好\n"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.73μs -> 2.78μs (1.80% slower)


def test_chinese_char_with_zero_width_space():
    # Chinese char with zero-width space
    tokenizer = BasicTokenizer()
    text = "你\u200b好"
    expected = " 你 \u200b好 "
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.56μs -> 2.45μs (4.33% faster)


def test_chinese_char_with_private_use_area():
    # Private use area chars should not be surrounded
    tokenizer = BasicTokenizer()
    text = "\ue000你\ue001"
    expected = "\ue000 你 \ue001"
    codeflash_output = tokenizer._tokenize_chinese_chars(text)  # 2.63μs -> 2.64μs (0.417% slower)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

To edit these changes git checkout codeflash/optimize-BasicTokenizer._tokenize_chinese_chars-misilokt and push.