Properly handle multibyte characters in postcode templates for tax calculation #40304
Description
Preconditions and environment
- Magento 2.4.8-p3
Steps to reproduce
- Add product to cart
- Go to checkout as guest
- Enter "31000 Šibenik" as postcode
Expected result
There are available shipping methods and you can proceed to the next step.
Actual result
- Observe that "estimate-shipping-methods" ajax call returns errors 500 with message "Unable to serialize value. Error: Malformed UTF-8 characters, possibly incorrectly encoded"
- Check logs and observe that there is a MySQL DB error
SQLSTATE[HY000]: General error: 1271 Illegal mix of collations for operation 'in', query was: SELECT `main_table`.`tax_calculation_rate_id`, `main_table`.`tax_calculation_rule_id`, `main_table`.`customer_tax_class_id`, `main_table`.`product_tax_class_id`, `rule`.`priority`, `rule`.`position`, `rule`.`calculate_subtotal`, `rate`.`rate` AS `value`, `rate`.`tax_country_id`, `rate`.`tax_region_id`, `rate`.`tax_postcode`, `rate`.`tax_calculation_rate_id`, `rate`.`code`, IF(title_table.value IS NULL, rate.code, title_table.value) AS `title` FROM `tax_calculation` AS `main_table`
INNER JOIN `tax_calculation_rule` AS `rule` ON `rule`.`tax_calculation_rule_id` = main_table.tax_calculation_rule_id
INNER JOIN `tax_calculation_rate` AS `rate` ON rate.tax_calculation_rate_id = main_table.tax_calculation_rate_id
LEFT JOIN `tax_calculation_rate_title` AS `title_table` ON rate.tax_calculation_rate_id = title_table.tax_calculation_rate_id AND title_table.store_id = '1' WHERE (customer_tax_class_id = 10) AND (product_tax_class_id IN ('11')) AND (rate.tax_country_id = 'HR') AND (rate.tax_region_id IN(0, 0)) AND (rate.zip_is_range IS NULL) AND (rate.tax_postcode IS NULL OR rate.tax_postcode IN('*', '', '31000 šib', '31000 šib*', '31000 ši*', '31000 š*', '31000 �*', '31000 *', '31000*', '3100*', '310*', '31*', '3*')) ORDER BY `priority` ASC, `tax_calculation_rule_id` ASC, `tax_country_id` DESC, `tax_region_id` DESC, `tax_postcode` DESC, `value` DESC
Additional information
The issue comes from using "strlen" and "substr" in fields which do not handle multibyte characters properly.
Release note
No response
Triage and priority
- Severity: S0 - Affects critical data or functionality and leaves users without workaround.
- Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
- Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
- Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
- Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.