feature(search): support searchNormalize for Greek and Cyrillic characters

[gnbm]

---

Issue number: resolves #279

---

What is the current behavior?

• The normalizeString utility used the regex /[^\w]/g to strip non-word characters after NFD decomposition.
• The \w character class in JavaScript only matches [a-zA-Z0-9_], so every non-Latin script (Greek, Cyrillic, Vietnamese, Chinese, Japanese, Korean, Arabic, Thai, …) was treated as non-word and stripped entirely during normalization. This made searchNormalize: true completely broken for non-Latin scripts — labels became empty strings, so nothing could be matched.

What is the new behavior?

• The normalization regex is changed from /[^\w]/g to /\p{M}/gu, which uses the Unicode property escape for combining marks (category M).
• This correctly strips diacritic/accent marks from NFD-decomposed characters across all Unicode scripts, while preserving the underlying letters/ideographs.

Language coverage

searchNormalize: true now works correctly for a single dropdown containing options across many writing systems:

Script	Example	Search input	Status
Latin (French/Spanish)	Crème brûlée, Niño	creme, nino	✅ matches
German (ä, ö, ü)	München, Mädchen, Köln	Munchen, Madchen, Koln	✅ matches
German ß	Größe	Grosse	⚠️ does not match — ß is an atomic letter (no NFD decomposition)
Norwegian å	Ålesund	Alesund	✅ matches
Norwegian ø, æ	Bjørn, Tromsø	Bjorn, Tromso	⚠️ does not match — ø and æ are atomic letters
Swedish (å, ä, ö)	Göteborg, Malmö	Goteborg, Malmo	✅ matches
Finnish (ä, ö)	Jyväskylä, Hämeenlinna	Jyvaskyla, Hameenlinna	✅ matches
Greek	Ένα	Ενα	✅ matches
Cyrillic	Ёжик, Йогурт	Ежик, Иогурт	✅ matches
Vietnamese	Việt Nam, Hà Nội	Viet Nam, Ha Noi	✅ matches
Arabic (tashkeel)	مُرَحَّباً	مرحبا	✅ matches
Korean (Hangul)	서울, 한국어	exact text	✅ matches (NFD-symmetric on both sides)
Chinese	北京, 你好	exact text	✅ matches (no combining marks; previously broken)
Japanese kanji & katakana	東京, カタカナ	exact text	✅ matches (previously broken)

Note on Thai and Japanese hiragana: stripping combining marks affects Thai vowel signs (e.g. สวัสดี → สวสด) and Japanese hiragana voicing marks (e.g. が → か). This enables fuzzy matching but loses some semantic precision. Use searchNormalize: false if exact-match behavior is required for those scripts.

Atomic letters (ø, æ, ß, etc.) are not decomposable under NFD and are therefore preserved literally — typing the ASCII fallback (Bjorn, Grosse) will not match the original (Bjørn, Größe). This is a Unicode-level limitation, not a regex limitation.

Performance optimization

• The COMBINING_MARKS_REGEX constant is now defined at module scope instead of being re-created inside normalizeString() on every call.
• Build toolchains (e.g. Babel targeting ES5/ES2015) transpile /\p{M}/gu into a ~2 KB character-class regex. Re-compiling that pattern on each keystroke during search was unnecessarily expensive.
• Hoisting to module scope means the regex is compiled once. Benchmarked against 10,000 calls with the actual transpiled pattern: ~29% faster (177 ms → 126 ms).

Documentation and examples

• Added a unified Multi-language search normalize section that demonstrates a single dropdown spanning Latin (French/Spanish/German/Norwegian/Swedish/Finnish), Greek, Cyrillic, Vietnamese, Chinese, Japanese, Korean, Arabic, and Thai — both with searchNormalize: true and searchNormalize: false variants for direct comparison.
• Updated docs/examples.md, docs/assets/script.js, and the table of contents.

Tests

Added Cypress describe blocks with two comprehensive describes against the unified multi-language dropdown:

• Multi-language search with searchNormalize: true (~25 specs) covers all listed scripts including positive cases (e.g. Munchen → München, Goteborg → Göteborg, Jyvaskyla → Jyväskylä, Ежик → Ёжик, Viet Nam → Việt Nam, مرحبا → مُرَحَّباً) and the documented atomic-letter limitations (Grosse does NOT match Größe; Bjorn does NOT match Bjørn).
• Multi-language search with searchNormalize: false verifies exact matches succeed (Greek/Cyrillic/Chinese/Japanese/Korean exact text) and that accent-stripped queries correctly find no options across all scripts.
• Latin diacritics regression suite (brulee → brûlée, cafe → café, nino → niño) preserved.

Does this introduce a breaking change?

• Yes
• No

Validations

Ran regression scenarios in the documentation using the branch - ✅
Run automated tests - ✅

[Copilot]

Pull request overview

This PR updates the search normalization logic to support Greek and Cyrillic text when searchNormalize: true, and adds docs/examples + Cypress coverage to validate the behavior.

Changes:

• Update normalizeString() to strip Unicode combining marks after NFD normalization (instead of stripping non-ASCII “non-word” chars).
• Add documentation examples for Greek/Cyrillic normalization and wire them into the docs demo script.
• Add Cypress E2E coverage for Greek/Cyrillic normalization and Latin-diacritics regression checks.

Reviewed changes

Copilot reviewed 8 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/utils/utils.js	Updates the normalization regex used by searchNormalize.
package.json	Updates devDependencies (build/test tooling) and adds ts-api-utils.
docs/examples.md	Adds a new Greek/Cyrillic “searchNormalize” example section.
docs/assets/virtual-select.js	Updates built docs asset to reflect new normalization logic.
docs/assets/script.js	Initializes new Greek/Cyrillic example selects in the docs demo page.
dist/virtual-select.js	Updates distributed (unminified) build with new normalization logic.
dist/virtual-select.min.js	Updates distributed minified build with new normalization logic.
cypress/e2e/examples.cy.ts	Adds E2E tests for Greek/Cyrillic searchNormalize and Latin regression.
.github/PULL_REQUEST_TEMPLATE.md	Adds a PR template for future contributions.
.claude/settings.local.json	Adds Claude tooling permissions config.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 12 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This changes normalizeString() semantics from “strip all non-\\w chars” (including punctuation/whitespace) to “strip only combining marks”. That can regress search matching for labels/descriptions containing punctuation or separators (e.g., co-op no longer normalizes to coop, so searching coop may stop matching when searchNormalize: true). If the intent is to keep prior behavior while fixing non-Latin scripts, consider removing combining marks and filtering out non-letter/number characters in a Unicode-aware way (e.g., keep \\p{L}\\p{N}_ across scripts), or explicitly document that punctuation is now preserved and update search expectations accordingly.

Suggested change

	return text.normalize('NFD').replace(COMBINING_MARKS_REGEX, '');
	const NON_WORD_CHARS_REGEX = /[^\p{L}\p{N}_]/gu;
	return text
	.normalize('NFD')
	.replace(COMBINING_MARKS_REGEX, '')
	.replace(NON_WORD_CHARS_REGEX, '');

[Copilot]

Several tests appear to rely on inter-test state (e.g., the first test opens the dropdown but doesn’t close it, and subsequent tests call cy.getVs(id).search(...) without clearly re-opening). This can make the suite order-dependent and flaky. Prefer isolating each it by opening/closing within the test, or use beforeEach to cy.open(id) and afterEach to close/reset, so each case runs from a known UI state.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 12 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 12 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

These new describe blocks are missing the usual go to section setup, making them order-dependent (they only work if a previous test already navigated to the correct docs section). To keep the suite robust and consistent with the rest of the file, add a cy.goToSection(...) in a beforeEach/first it for this block (and the other new blocks below).

Suggested change

	beforeEach(() => {
	cy.goToSection('Greek and Cyrillic search normalize');
	});

[Copilot]

This describe block also lacks a cy.goToSection(...) setup step, so it implicitly relies on prior tests having navigated to the section and rendered the selects. Add an explicit navigation step (or a shared beforeEach) so the block can run independently.

[Copilot]

This describe block also doesn't navigate to the docs section before interacting with the select. Add cy.goToSection('Greek and Cyrillic search normalize') (or a shared beforeEach) to avoid test-order coupling.

[Copilot]

filemanager-webpack-plugin@^10.0.1 declares engines.node >=22.14.0 (per lockfile) while the release workflow uses Node 20. This is likely to break npm ci/npm run build during publishing. Either keep filemanager-webpack-plugin on a Node-20-compatible major (e.g. v9) or bump the workflow Node version (and document the new minimum Node requirement).

Suggested change

	"filemanager-webpack-plugin": "^10.0.1",
	"filemanager-webpack-plugin": "^9.0.0",

[Copilot]

This function defines COMBINING_MARKS_REGEX inside normalizeString(), which will re-create the large RegExp on every call. Since docs search also runs per keystroke, consider hoisting the regex to module scope in this built asset to avoid unnecessary recompilation.