Fix UTF-16LE line endings in dni/ini files

[mike-clark-8192]

• Three files had corrupted single-byte CRLF (0d 0a) mixed with UTF-16LE content, breaking git's working-tree-encoding conversion
• Corrected to proper UTF-16LE CRLF (0d 00 0a 00) so files round-trip correctly through git

[RamonUnch]

I cannot find the single word 0D0A in AltSnap.dni file nor in fr_FR.ini, However ko_KR.ini seems to be utf-8 encoded.

It seems you are actually converting to UTF-8 some files?

I do not understant what github is doing with encoding, maybe I should remove the .gitattributes flags, and treat those files like binary blobs.

[mike-clark-8192]

I think I ran into issues with EOL conversions interacting badly with the character encoding confusion, so the content of my pull request is probably not very helpful.

However, I did some more research, which led me to this recipe. I tested it and it appears to work. It may seem overly complex, but I wasn't able to simplify it further. I hope this helps.

#!/bin/bash
set -euo pipefail

# Fix for AltSnap repo encoding issues.
#
# We want to use a .gitattributes with working-tree-encoding=utf-16le-bom, as
# this enables git diffs on UTF-16 files, which is nice. Also it explicitly
# declares the required encoding for these files, also nice. However
# working-tree-encoding always requires that the blobs be stored as UTF-8 in
# the git repository, regardless of what encoding they will ultimately be
# checked out to. In this repo though, most of the blobs were committed as raw
# UTF-16LE — never converted to UTF-8. When the .gitattributes was added later,
# it confuses git. It results in warnings and results in working-tree-encoding
# not function properly. A broken .gitattributes makes it difficult to correct
# the encoding of the blobs to UTF-8, so we have to do it in multiple steps,
# the first of which is to temporarily remove .gitattributes (we will restore it
# later).
#
# This script:
#   1. Removes .gitattributes so git treats files as raw bytes
#   2. Converts the UTF-16LE blobs to UTF-8 (the format git expects)
#   3. Normalizes line endings to DOS (CRLF)
#   4. Restores .gitattributes (with eol=crlf added)
#   5. Lets git apply the encoding rules, producing proper UTF-16LE-BOM
#      working tree files from the now-correct UTF-8 blobs

REPO_URL="https://github.com/RamonUnch/AltSnap"
CLONE_DIR="AltSnap"

# Fresh clone (expect encoding errors — that's the bug we're fixing)
git clone "$REPO_URL" "$CLONE_DIR"
cd "$CLONE_DIR"

# Step 1: Remove .gitattributes and commit
git rm .gitattributes
git commit -m "remove .gitattributes"

# Re-checkout files as raw bytes (without .gitattributes filtering)
git rm --cached -r . > /dev/null
git reset HEAD > /dev/null 2>&1
git checkout -- .

# Step 2: Convert UTF-16 to UTF-8 and normalize line endings
# Using -f utf-16 (not utf-16le) so iconv reads and consumes the BOM.
# ko_KR.ini is already UTF-8 so skip the iconv step for it.
for f in AltSnap.dni Lang/*; do
    if [ "$f" != "Lang/ko_KR.ini" ]; then
        iconv -f utf-16 -t utf-8 "$f" > "$f.tmp"
        mv "$f.tmp" "$f"
    fi
    unix2dos -e "$f"
done

git add .
git commit -m "normalize git-internal encoding to UTF-8 and line endings to DOS"

# Step 3: Restore .gitattributes with an additional specifier of eol=crlf to 
# enforce Windows compatibility and avoid EOL conversion warnings.
cat > .gitattributes << 'EOF'
*.dni diff merge text eol=crlf linguist-language=ini working-tree-encoding=utf-16le-bom
Lang/* diff merge text eol=crlf linguist-language=ini working-tree-encoding=utf-16le-bom
EOF

git add .gitattributes
git commit -m "add back .gitattributes with eol=crlf"

# Step 4: Force git to re-checkout with the new .gitattributes rules
# Clearing the index is necessary — git won't re-apply working-tree-encoding
# if it thinks the checkout is already current.
git rm --cached -r . > /dev/null 2>&1
git reset --hard HEAD

# Step 5: Commit the re-encoded working tree
git add .
git commit -m "allow .gitattributes to take effect"

# Verify
echo ""
echo "=== Verification ==="
unix2dos -idumbteph AltSnap.dni Lang/*

[mike-clark-8192]

@RamonUnch I applied my fix to my feature branch (and thus to this PR).
Despite the large number of "changes" that GitHub Web UI reports, you can prove to yourself that the fix worked:

# You can clone my feature branch as a way to validate the effect of the PR, 
# since the PR's content is identical to the feature branch (it's literally symlinked by GitHub:)
git clone -b fix/utf16le-line-endings https://github.com/mike-clark-8192/AltSnap AS-MC # no warnings
git clone https://github.com/RamonUnch/AltSnap AS-RU # gives warnings

After making these clones, use a directory diff tool diffing ./AS-RU/ and ./AS-MC/
The only difference you should see between is the addition of eol=crlf to .gitattributes
Or at least, that's the result I got. So afaik you should be good to accept this PR if you want.