chinese character's encoding problem

[toreal]

Thank you for your great project. I have some issues about chinese encoding problems.

(a) If the field type is setup as 'varchar'. the saved chinese text will become garbled text.
If I change "connection.ex"
case :unicode.characters_to_binary(value, :utf8, {:utf16 , :little}) do
to
case Codepagex.from_string(value, "VENDORS/MICSFT/WINDOWS/CP950") do
it can solve the problem.

(b) But if the field type is 'nvarchar', the chinese character becomes garbled text.

As I understanding, varchar's encoding is CP950.
nvarchar's encoding is unicode.

That is why have such problem.

(c) But, I face another problem. If the field type is varchar(1), then even english character can not be inserted. To solve this problem, I change "connection.ex"
defp param(value) when is_binary(value) do
{value, nil}
# case :unicode.characters_to_binary(value, :utf8, {:utf16, :little}) do
# {:error, _, _} -> {value, :binary}
# val -> {val, nil}
# end
end

Then, everything look fine including chinese character encoding problem.
My question is whether characters_to_binary is necessary or not. Could I ignore it?

[mjaric]

I think we have to convert bitstring to little endian binary representation, that is why we need that call. I'm not sure how SQL server managed to accept it without reversing char bits since for Chinese, if I'm not mistaken, each char takes 2 bytes.

Anyhow. NVARCHAR has to work with this function, I will write some tests since I never checked other codepages and let you know what I found.