docs(unicode): add information about fixing UTF-8/Unicode problems

[Kristinita]

1. Summary

Adding the feature “Use Unicode UTF-8 for worldwide language support” resolves for me all Unicode/UTF-8 problems in AppVeyor that I don’t reproduced on my local Windows. I no longer have problems described, for example, in Stack Overflow question “UTF-8 in AppVeyor” and in issues #2107, #2129, #2226. I’m not detected new problems after enabling the feature.

I think, it would be nice if AppVeyor documentation will contain information about “Use Unicode UTF-8 for worldwide language support”.

2. Justification of the need for changes

I have written my first Stack Overflow question about Unicode problems on AppVeyor 8 years ago. I have written my first issue in this repository about Unicode problems 7 years ago; during these years, I have written 3 issues. In these times I still haven’t received the answer which would solve my problems. Based on this, I think that the solution of problems isn’t the obvious, and documentation changes could help peoples.

3. Solution

See comments for details.

appveyor/UnicodeUTF8Worldwide.reg:

Windows Registry Editor Version 5.00

; [OVERVIEW] “Beta: Use Unicode UTF-8 for worldwide language support” — support UTF-8 for Windows:
; https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page
;
; [INFO] Activate UTF-8 without GUI via “.reg” file:
; https://stackoverflow.com/a/68066008/5951529
;
; [REQUIRED] System restart after this registry changing:
; https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page#set-a-process-code-page-to-utf-8
;
; [INFO] The feature resolve for me all problems that I’m not reproduce on local Windows, for example:
; https://stackoverflow.com/questions/42644184/utf-8-in-appveyor-python-3-6
; https://github.com/appveyor/ci/issues/2107
; https://github.com/appveyor/ci/issues/2129
; https://github.com/appveyor/ci/issues/2226
;
; [INFO] Microsoft begin to support UTF-8 since 2019:
; https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#UTF-8
;
; [INFO] This feature set “chcp 65001” for Windows for the cmd.exe command prompt:
; https://stackoverflow.com/a/57134096/5951529
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage]
"ACP"="65001"
"OEMCP"="65001"
"MACCP"="65001"

The part of the appveyor.yml:

install:
# [INFO] Place import of the “.reg” file, “shutdown” and “timeout” commands inside “install”, not “init” section:
# https://github.com/appveyor/ci/issues/2459#issuecomment-456042331
- reg import appveyor/UnicodeUTF8Worldwide.reg

# [INFO] “/t” command-line argument — the time until system shutdown in seconds.
# I use the default value — 30 seconds.
- shutdown /r

# [INFO] Timeouts before and after restart required, otherwise AppVeyor will freeze:
# https://github.com/appveyor/ci/issues/2482#issuecomment-401995271
#
# [NOTE] AppVeyor hasn’t a documentation how to restart AppVeyor worker:
# https://github.com/appveyor/ci/issues/2483
#
# [NOTE] The timeout value of 10 second recommended on AppVeyor issue tracker aren’t enough for me:
# https://github.com/appveyor/ci/issues/2456#issuecomment-403299903
# Also, 30 seconds may be not enough:
# https://ci.appveyor.com/project/Kristinita/sashaappveyordebugging/builds/51589557
#
# [NOTE] Timeout value when I used in 2024 isn’t enough for 2025.
# [COMPARE] “timeout 30” doesn’t work for the same configuration for me in 2025:
# https://ci.appveyor.com/project/Kristinita/sashaappveyordebugging/builds/51589557
# https://ci.appveyor.com/project/Kristinita/sashaappveyordebugging/builds/48951962
# Therefore, I use the big value “60”.
#
# [NOTE] Timeout isn’t sustainable method.
# It would be nice if CLI commands in AppVeyor will run after some event on the system instead of timeout.
- timeout 60

4. Documentation suggestions

1. In my opinion, it would be nice if documentation will contain cross-terminal CLI commands that user can use even in cmd.exe command prompt. Not solely PowerShell-specific commands.
2. Changing the system locale may also resolve Unicode problems (I’m not tested it). Possibly, it would be nice if documentation will contain information about it.

5. Related suggestions

In my opinion, it would be nice if AppVeyor will contain at least documentation changes. But in general, I think, AppVeyor needs another changes.

5.1. “utf8” key

“Use Unicode UTF-8 for worldwide language support” requires the system restart. I think, it would be nice if the file appveyor.yml will contain the key as utf8. If utf8: true, AppVeyor build will run with the enabled option “Use Unicode UTF-8 for worldwide language support” without restart. Restarts isn’t the best solution:

1. For users, because restarts wasting their time
2. For AppVeyor team, because restarts wasting AppVeyor resources

5.2. Restart AppVeyor worker documentation

In 2018, Ilya Finkelshteyn created the issue #2483 “Document how to restart AppVeyor worker”. Unfortunately, it seems like the documentation still not created.

I don’t understand the correct way for restarting. Ilya Finkelshteyn wrote:

If you set sleep value to 10 seconds, it would be more than enough to eliminate its possibility and continue to restart at init step.

Owen McDonnell wrote:

What happens if you extend the sleep time (i.e. Start-Sleep -s 10)?

But in today’s tests, even 30 seconds wasn’t enough for me, see the build 1.0.383. timeout 60 resolves the problem, see the build 1.0.385.

I don’t know why it’s happen, and documentation for this case would be useful for me.

5.3. Resuming AppVeyor build after restart without timeout

Timeouts aren’t sustainable method. We cannot calculate the exact timeout time. In the beginning of 2024, the build 1.0.246 successfully worked for me with timeout 30, but the build 1.0.385 with the identical configuration failed today.

I think that resuming AppVeyor build based on system events, not timeout, would be better solution. Possibly, Windows registry keys Run and RunOnce would be help.

In ideal, it would be nice to have in the file appveyor.yml the key as restart. restart: true would solve without additional settings all problems with restarts like in example below.

install:
- reg import appveyor/UnicodeUTF8Worldwide.reg
- restart: true

Thanks.