Help with file encoding

[Agadoul]

Hello, I wrote a port for little cms and the devil library. I am close to send the PR but there is some issues with file encoding and I have a lot of warnings during compilation like:

warning C4828: The file contains a character starting at offset 0x248 that is illegal in the current source character set (codepage 65001)

It translates into an error in the case of devil:
line: IL_TEXT("format non supporté"),
error: 1>C:\Sources\Team Services\vcpkg\buildtrees\devil\src\devIL\DevIL\src-ILU\include\ilu_error/ilu_err-french.h(22): error C2059: syntax error: ')'
1>C:\Sources\Team Services\vcpkg\buildtrees\devil\src\devIL\DevIL\src-ILU\include\ilu_error/ilu_err-french.h(22): error C2061: syntax error: identifier 'eund'

This does not happen if the solution is generated from CMake.

Is there something to do related to file encoding ?

Edit:

removing /utf-8 from the c flags seems to fix it.

list(APPEND _csc_OPTIONS
    "-DCMAKE_CXX_FLAGS= /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc"
    "-DCMAKE_C_FLAGS= /DWIN32 /D_WINDOWS /W3"# /utf-8"
    "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON"
    "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON"
    "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON"
)

Regards,
Axel.

[ras0219-msft]

Glad to hear about the port!

According to the /utf-8 MSDN page [1], if no switch is supplied then MSVC will assume the user's current codepage (!) which sounds like a terrible recipe for works-on-my-machine issues. Therefore, the best way to fix this is probably to patch the CMakeLists.txt for devil to explicitly use the /source-charset: switch.

string(REPLACE "/utf-8" "" CMAKE_C_FLAGS "${CMAKE_C_FLAGS}")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} /source-charset:.1252")

.1252 is Latin-1, which is probably what you're using on your machine and therefore the value that works.

Alternatively, the ilu_err-french.h header could be re-encoded as UTF-8, however that seems significantly harder.

[1] https://msdn.microsoft.com/en-us/library/mt708821.aspx

[codicodi]

I looked at these headers and it looks like latin based ones (like french) are encoded in Latin-1. However there is also arabic as well as japanese translation, which are actually encoded as bom-less utf-8! So I wouldn't recommend changing source charset globally.

[ras0219-msft]

Hmm, that sounds like it is probably easier to pre-convert all the files to UTF-8 ahead of time with a (perhaps large) patch.

Are these internal-only headers, or are they installed for other code to use? If other code is going to consume them, that's a huge reason to standardize the encoding.

[codicodi]

They seem to be internal.

This does not happen if the solution is generated from CMake.

If compiler doesn't raise an error about utf-8 encoded japanese that pretends to be in local code page, you probably compile without UNICODE or _UNICODE #defined, the text macros doesn't expand to wide literals and encoding of the error messages is inconsistent.
I'd recommend

1. Patching the CMakeLists.txt with add_definitions(-DUNICODE -D_UNICODE)
2. Converting all the files in DevIL\src-ILU\include\ilu_error except the english, japanese and arabic ones to utf-8

As to how... is there a way do such conversion easily on the command line?

[Agadoul]

Thanks for all your feedback. LittleCMS does not come with a CMakeLists.txt so here I could just add:
string(REPLACE "/utf-8" "" CMAKE_C_FLAGS "${CMAKE_C_FLAGS}")

LittleCMS use different encoding for different *.c files. utf-8 is one of them.

My understanding is that visual studio is able to detect the character set used in that case on a per-file basis. Here is the warning displayed with /utf-8 globally enabled:
warning C4828: The file contains a character starting at offset 0x2673 that is illegal in the current source character set (codepage 65001).

For devil, it seems to be similar, multiple encoding files that visual studio is able to detect. I have never had to use the /utf-8 flags before. It is a bit weird that none of the previous ports had this issue because a solution made out of multiple files encoded differently seems like something you encounter quite often.

Anyway, I m going to patch the Devil repo directly and propose a PR to the guys over there, there is a few incompatibility with their CMakeLists.txt that I need to fix anyway.

In the meantime, you can expect a PR for the littleCMS library port.

Thanks.

[Lyrex]

I am currently trying to get a working DevIL port ready (issue #1973), but I'm stuck on the same problem you are.
If I change the encoding of the files manually, I get the whole thing sucessfully compiling and installing.. but I can't figure out how I can change the encoding of a whole file using a patch file.
Any ideas?

edit: I already fixed the encoding problem. Patching seemed to work fine.

@Agadoul Do you have your port completed already? If yes, would you mind sharing it?

[Agadoul]

@Lyrex I will have a look this weekend if I can finish it. I remember proposing something to devil

DentonW/DevIL#57

It was an issue where the cmake module path being overwritten by devil's CMakelists instead of appending to it. The author has not answered, nor merged the PR. :(

sorry for the late answer.

[Agadoul]

Hi @Lyrex , how did you fix the encoding finally?

I don't mind sharing my port but I haven't find a "clean" way for the encoding issue.

[Agadoul]

Hi I made a PR too: #2164

I guess we have 2 PRs now for the same port.

[ras0219-msft]

Sorry for the slow response, but Lyrex's pr has been merged!