Fix PostgreSQL UTF-8 encoding errors from IMAP email data

[Copilot]

Scheduled email receiver was failing with SQLSTATE[22021]: invalid byte sequence for encoding "UTF8": 0xc3 0x20 when processing emails. IMAP data contains invalid UTF-8 sequences that PostgreSQL rejects.

Changes

• Added UTF-8 sanitization utilities (ThreadUtils.php)

  • sanitizeUtf8String() - replaces invalid byte sequences using mb_convert_encoding()
  • sanitizeUtf8Recursive() - handles nested arrays/objects

• Sanitize all text data before DB insertion (ThreadEmailDatabaseSaver.php)

  • Email addresses before queries
  • IMAP headers (JSON-encoded)
  • Attachment metadata (name, filename, filetype, location)
  • Error processing data

// Before
$all_emails = $email->getEmailAddresses($rawEmail);
$threads = Database::query("SELECT id, my_email FROM threads WHERE my_email IN (...)", $all_emails);

// After
$all_emails = $email->getEmailAddresses($rawEmail);
$all_emails = array_map('sanitizeUtf8String', $all_emails);
$threads = Database::query("SELECT id, my_email FROM threads WHERE my_email IN (...)", $all_emails);

Handles edge cases: empty strings, valid UTF-8 (including Norwegian characters), control characters, and the specific 0xc3 0x20 sequence from the error.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/PHPMailer/PHPMailer/zipball/bf74d75a1fde6beaa34a0ddae2ec5fce0f72a144
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/laminas/laminas-loader/zipball/c507d5eccb969f7208434e3980680a1f6c0b1d8d
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/laminas/laminas-mail/zipball/110e04497395123998220e244cceecb167cc6dda
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/laminas/laminas-mime/zipball/08cc544778829b7d68d27a097885bd6e7130135e
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/laminas/laminas-servicemanager/zipball/a8640182b892b99767d54404d19c5c3b3699f79b
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/laminas/laminas-stdlib/zipball/8974a1213be42c3e2f70b2c27b17f910291ab2f4
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/laminas/laminas-validator/zipball/b3ec5865e7aa60a0fbce211500f4a5c6a6e11a30
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/myclabs/DeepCopy/zipball/1720ddd719e16cf0db4eb1c6eca108031636d46c
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/nikic/PHP-Parser/zipball/ae59794362fe85e051a58ad36b289443f57be7a9
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/phar-io/manifest/zipball/54750ef60c58e43759730615a392c31c80e23176
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/phar-io/version/zipball/4f7fd7836c6f332bb2933569e566a0d6c4cbed74
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/php-fig/container/zipball/513e0666f7216c7459170d56df27dfcefe1689ea
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/php-fig/http-message/zipball/402d35bcb92c70c026d1a6a9883f06b2ead23d71
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/cli-parser/zipball/c34583b87e7b7a8055bf6c450c2c77ce32a24084
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/code-unit-reverse-lookup/zipball/5e3a687f7d8ae33fb362c5c0743794bbb2420a1d
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/code-unit/zipball/a81fee9eef0b7a76af11d121767abc44c104e503
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/comparator/zipball/a18251eb0b7a2dcd2f7aa3d6078b18545ef0558e
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/complexity/zipball/68ff824baeae169ec9f2137158ee529584553799
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/diff/zipball/c41e007b4b62af48218231d6c2275e4c9b975b2e
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/environment/zipball/8074dbcd93529b357029f5cc5058fd3e43666984
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/exporter/zipball/955288482d97c19a372d3f31006ab3f37da47adf
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/global-state/zipball/987bafff24ecc4c9ac418cab1145b96dd6e9cbd9
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/lines-of-code/zipball/856e7f6a75a84e339195d48c556f23be2ebf75d0
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/object-enumerator/zipball/202d0e344a580d7f7d04b3fafce6933e59dae906
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/object-reflector/zipball/24ed13d98130f0e7122df55d06c5c4942a577957
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/php-code-coverage/zipball/7e308268858ed6baedc8704a304727d20bc07c77
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/php-file-iterator/zipball/a95037b6d9e608ba092da1b23931e537cadc3c3c
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/php-invoker/zipball/f5e568ba02fa5ba0ddd0f618391d5a9ea50b06d7
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/php-text-template/zipball/0c7b06ff49e3d5072f057eb1fa59258bf287a748
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/php-timer/zipball/e2a2d67966e740530f4a3343fe2e030ffdc1161d
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/phpunit/zipball/8080be387a5be380dda48c6f41cee4a13aadab3d
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/recursion-context/zipball/05909fb5bc7df4c52992396d0116aed689f93712
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/type/zipball/462699a16464c3944eefc02ebdd77882bd3925bf
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/sebastianbergmann/version/zipball/c51fa83a5d8f43f1402e3f32a005e6262244ef17
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/symfony/polyfill-intl-idn/zipball/9614ac4d8061dc257ecc64cba1b140873dce8ad3
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/symfony/polyfill-intl-normalizer/zipball/3833d7255cc303546435cb650316bff708a1c75c
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/symfony/polyfill-mbstring/zipball/6d857f4d76bd4b343eac26d6b539585d2bc56493
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/theseer/tokenizer/zipball/737eda637ed5e28c3413cb1ebe8bb52cbf1ca7a2
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)
• https://api.github.com/repos/webmozarts/assert/zipball/11cb2199493b2f8a3b53e7f19068fc6aac760991
  • Triggering command: /usr/bin/php8.3 /usr/bin/php8.3 -n -c /tmp/8d2DYS /usr/bin/composer install --no-interaction (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Error in scheduled receiver of emails</issue_title>
<issue_description>```
An error occurred in the Offpost system:

Source: scheduled-email-receiver
Time: 2026-01-23 14:49:14 CET
Message: Error in email processing: Error processing folder: INBOX.964982031-snasa-kommune - Innsyn i haandskrevet opptellingsdata. Rolling back DB
transaction

Error Details:
success:
message: Error processing folder: INBOX.964982031-snasa-kommune - Innsyn i haandskrevet opptellingsdata. Rolling back DB transaction
folder_name: INBOX.964982031-snasa-kommune - Innsyn i haandskrevet opptellingsdata
thread_id: f36399b5-0bf4-40db-8a41-9f2cd74e2705
error: Rolling back DB transaction
debug: Error processing folder: INBOX.964982031-snasa-kommune - Innsyn i haandskrevet opptellingsdata. Rolling back DB transaction
Exception: Rolling back DB transaction
at ThreadEmailDatabaseSaver.saveThreadEmails(ThreadEmailDatabaseSaver.php:226)
at ThreadScheduledEmailReceiver.processNextFolder(ThreadScheduledEmailReceiver.php:106)
at require(scheduled-email-receiver.php:29)
at (main)(index.php:37)
Caused by: PDOException: SQLSTATE[22021]: Character not in repertoire: 7 ERROR: invalid byte sequence for encoding "UTF8": 0xc3 0x20
CONTEXT: unnamed portal parameter $3
at PDOStatement.execute(Database.php:61)
at Database.query(Database.php:61)
at ThreadEmailDatabaseSaver.saveThreadEmails(ThreadEmailDatabaseSaver.php:87)
at ThreadScheduledEmailReceiver.processNextFolder(ThreadScheduledEmailReceiver.php:106)
at require(scheduled-email-receiver.php:29)
at (main)(index.php:37)

stack_trace: Exception: Rolling back DB transaction
at ThreadEmailDatabaseSaver.saveThreadEmails(ThreadEmailDatabaseSaver.php:226)
at ThreadScheduledEmailReceiver.processNextFolder(ThreadScheduledEmailReceiver.php:106)
at require(scheduled-email-receiver.php:29)
at (main)(index.php:37)
Caused by: PDOException: SQLSTATE[22021]: Character not in repertoire: 7 ERROR: invalid byte sequence for encoding "UTF8": 0xc3 0x20
CONTEXT: unnamed portal parameter $3

## Comments on the Issue (you are @copilot in this section)

<comments>
</comments>

• Fixes Error in scheduled receiver of emails #129

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.