Bug: Characters disallowed by `PathCleanupSpec` in filesystems mounted via WSL2 are rendered as invalid codepoints.

[RokeJulianLockhart]

Description

I've some directories and files whose names are comprised of tabs ( ). Inside dolphin-25.04.2-1.fc42 (on cpe:/o:fedoraproject:fedora:42), these render as expected:

However, in FilesPreview, these render as UTF tofu:

To confirm, if I duplicate one of their filenames to my Windows clipboard, I see:

`{title ,id s22ai3}`{.JSON5}.dir

However, this should be, per my Linux installation:

`{title: "", id: "s22ai3"}`{.JSON5}.dir

...which can be confirmed with tree there:

.
└── \011`{title: "",\011id: "s22ai3"}`{.JSON5}.dir

This is a significant problem, because it means that Files is performing character replacement, thereby potentially making unintentionally modifying the filename trivial. I haven't confirmed this because I don't dare: I should have mounted this as ro.

Please note that although, in the browser, you're probably seeing hexagana if you're using Windows, this is font-dependent, so Files isn't doing anything special. It's merely not rendering the characters. As an example, explorer.exe "\\?\UNC\wsl.localhost\fedoraremix\mnt\rjlb_data\@{'Name'='Entities'}#.dir\@{'Status'='Commenced'; 'Name'='In Use'}#.dir\@{'Person'='Beedell'; 'Identifier'='S21AIB'}#.dir\@{'Name'='Entity-Specific'}#.dir\@{'Person'='Roke'; 'Identifier'='S1EV2L'}#.dir\@{'Name'='Temporary'}#.dir\. [ ]`{title Git,id s2ykra}`{.JSON5}.dir" also fails to, but renders another (incorrect) replacement character:

Steps To Reproduce

1. Using a real Linux OS, in a BTRFS filesystem, create a path longer than MAX_PATH that contains tab characters.

2. Create the undermentioned structure: ¹

   1. #!/usr/bin/env sh
      sudo lsblk -f

   2. NAME   FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
      sda
      sdb
      sdc                                           [SWAP]
      sdd                             954.8G     0% /mnt/wslg/distro
                                                    /
      sde
      ├─sde1
      ├─sde2
      ├─sde3
      └─sde4                            1.4T    24% /mnt/test
      sdf
      └─sdf1                          184.3G    90% /mnt/rjlb_data

   To do so:

   1. #!/usr/bin/env pwsh
      #Requires -PSEdition Core
      If ($IsWindows) {
      	sudo wsl --mount \\.\PHYSICALDRIVE1 --bare
      }

   2. #!/usr/bin/env sh
      sudo lsblk -f
      	# Utilise the output to confirm that the *device* has been mounted into `wsl`, then set its path as `$dev_id`, and define a mount point at `$path`.
      sudo mount -t btrfs /dev/$dev_id /mnt/$path

Files Version

3.9.10.0

Windows Version

10.0.26120

User ID

353b71b8-86f9-4fe0-ac0e-e1da114b9349

Log File

debug.log

Footnotes

1. files-community/Files/issues/17133 ↩

[RokeJulianLockhart]

#17205 (comment)

I've since realised that because this is replicated in explorer.exe, perhaps this is a bug in WSL? I've wondered this because a lot of things appear to break at this point. IDK whether it's due to being past MAX_PATH or the filenames themselves, since if I open a wt.exe instance (via FilesPreview, though) at Microsoft.PowerShell.Core\FileSystem::\\wsl.localhost\fedoraremix\mnt\rjlb_data\@{'Name'='Entities'}#.dir\@{'Status'='Commenced'; 'Name'='In Use'}#.dir\@{'Person'='Beedell'; 'Identifier'='S21AIB'}#.dir\@{'Name'='Entity-Specific'}#.dir\@{'Person'='Roke'; 'Identifier'='S1EV2L'}#.dir\@{'Name'='Temporary'}#.dir, then run:

1. Get-ChildItem, no names are returned, despite Windows disallowing $Null filenames:

   PS Microsoft.PowerShell.Core\FileSystem::\\wsl.localhost\fedoraremix\mnt\rjlb_data\@{'Name'='Entities'}#.dir\@{'Status'='Commenced'; 'Name'='In Use'}#.dir\@{'Person'='Beedell'; 'Identifier'='S21AIB'}#.dir\@{'Name'='Entity-Specific'}#.dir\@{'Person'='Roke'; 'Identifier'='S1EV2L'}#.dir\@{'Name'='Temporary'}#.dir> ls
   
       Directory: \\wsl.localhost\fedoraremix\mnt\rjlb_data\@{'Name'='Entities'}#.dir\@{'Status'='Commenced'; 'Name'='In
   Use'}#.dir\@{'Person'='Beedell'; 'Identifier'='S21AIB'}#.dir\@{'Name'='Entity-Specific'}#.dir\@{'Person'='Roke';
   'Identifier'='S1EV2L'}#.dir\@{'Name'='Temporary'}#.dir
   
   Mode                 LastWriteTime         Length Name
   ----                 -------------         ------ ----
   d----          2024-04-08    13:30
   d----          2023-09-25    23:51
   d----          2023-03-21    16:30
   d----          2023-12-16    03:10
   d----          2023-04-15    19:55
   d----          2022-10-23    22:23
   d----          2023-09-22    22:00
   d----          2023-09-22    22:01
   -----          2022-06-23    14:26     1011594814
   -----          2022-05-12    23:15         144430
   -----          2022-07-01    00:20            557
   -----          2022-01-07    00:50         373962
   -----          2023-09-22    22:02             67
   -----          2021-03-19    14:41          65647 @{'Status'='Completed'; 'Name'='Instagram Illustration';
                                                     'Identifier'='S1G0ZM'}#.JPEG
   -----          2022-05-12    23:15           3296
   -----          2022-05-26    13:21         440127
   -----          2022-05-26    13:25          18313
   -----          2022-06-30    09:13         141148
   -----          2022-06-30    09:14         250847
   -----          2022-07-01    17:35          35892
   -----          2022-07-02    01:14         165810
   -----          2022-07-02    01:14          48157
   -----          2022-07-03    14:10         376541
   -----          2021-12-24    15:18           9367
   -----          2022-03-23    14:59          25051
   -----          2023-07-04    23:41          34374

2. tree ., it spawns an OpenConsole instance that eventually appears to crash (per WinDbg):

   C:\__w\1\s\src\server\ConDrvDeviceComm.cpp(147)\OpenConsole.exe!00007FF7CF792894: (caller: 00007FF7CF7925C7) ReturnHr(1) tid(6020) 800700E9 No process is on the other end of the pipe.
   ntdll!NtTerminateProcess+0x14:
   00007ffc`1f7dfc34 c3              ret
   0:007> k
    # Child-SP          RetAddr               Call Site
   00 0000002b`3a7ffc38 00007ffc`1f75fd8e     ntdll!NtTerminateProcess+0x14
   01 0000002b`3a7ffc40 00007ffc`1e9218ab     ntdll!RtlExitUserProcess+0x11e
   02 0000002b`3a7ffc70 00007ff7`cf70a4b9     KERNEL32!ExitProcessImplementation+0xb
   03 0000002b`3a7ffca0 00007ff7`cf72eb47     OpenConsole+0xa4b9
   04 0000002b`3a7ffcd0 00007ffc`1e90e8d7     OpenConsole+0x2eb47
   05 0000002b`3a7fff50 00007ffc`1f75fbcc     KERNEL32!BaseThreadInitThunk+0x17
   06 0000002b`3a7fff80 00000000`00000000     ntdll!RtlUserThreadStart+0x2c

   Originally, I thought it was designed to close on completion, but tree, in all other circumstances, doesn't spawn a new window. It wouldn't make much sense for a standalone window, anyway. (I've a .dump /ma, if of use, although that would probably be best provided to the Windows Terminal team, if this is a bug.)

[0x5bfa]

but renders another (incorrect) replacement character:

The reason why there's difference between File Explorer and Files is most likely because they use different fonts (FE: Segoe UI, Files: Segoe UI Variable Text).

Characters disallowed by Win32 and WinRT in filenames are rendered as squares

As File Explorer (more specifically DWrite I believe) does the char replace, this is expected behavior also in Files.

[RokeJulianLockhart]

but renders another (incorrect) replacement character:

The reason why there's difference between File Explorer and Files is most likely because they use different fonts (FE: Segoe UI, Files: Segoe UI Variable Text).

@0x5bfa, regardless, Files and explorer.exe should be utilising UTF “Replacement Characters” rather than whatever box character it's just decided to use. It's not semantic otherwise. ¹ I don't believe that the font decides this.

As File Explorer (more specifically DWrite I believe) does the char replace, this is expected behavior also in Files.

...Even though you can see that if someone were to accidentally modify such a filename, it could well corrupt it? Files is designed to be better than Explorer – it's its raison d’être, so perhaps it could make such objects at least ro (or warn the user) when its "are all characters real Unicode characters" sanity check fails?

Irrespective, since WSL's btrfs passthrough driver appears to be the one supplying these values, rather than Files or explorer.exe interpreting them incorrectly, I'll try reporting it to them, after going through PowerShell's developers to confirm.

Footnotes

1. english.stackexchange.com/revisions/233906/1 ↩

[0x5bfa]

regardless, Files and explorer.exe should be utilising UTF “Replacement Characters” rather than whatever box character it's just decided to use. It's not semantic otherwise. 1 I don't believe that the font decides this.

I didnt say fonts decide what char for missing glyphs should be used. The graphic redner (on Windows, most likely DWrite) decides.

...Even though you can see that if someone were to accidentally modify such a filename, it could well corrupt it? Files is designed to be better than Explorer – it's its raison d’être, so perhaps it could make such objects at least ro (or warn the user) when its "are all characters real Unicode characters" sanity check fails?

I'm not sure what kind of prompt is suitable here. CC @yaira2

[RokeJulianLockhart]

Per #17205 (comment), I've submitted this to microsoft/WSL#13166 (comment), since it affects explorer.exe. However, per PowerShell/PowerShell#25680 (comment), this can be handled by the consumer right now, regardless.

[yaira2]

There seems to be components here

1. Files should display the tab spaces properly
2. Files should be able to rename these items properly without replacing the tabs

Can you confirm whether I got this right?

[RokeJulianLockhart]

Files should display the tab spaces properly

@yaira2Y, yes. At the very least, utilise semantic replacement characters, but actual filename should render! Win32's and NTFS's disallowed characters have no bearing on BTRFS.

Files should be able to rename these items properly without replacing the tabs

I'm unable to feasibly confirm whether modifying the name causes it to be changed, because I don't have a test filesystem available to me RN, and don't want to corrupt my actual data drive.

However, considering that your file rename code almost definitely works on characters (rather than bytestreams) and likely contains basic sanity checks to remove non-UTF bytes (to prevent filesystem corruption), the behaviour on rename needs to be confirmed to ensure that it is as expected.

I'll attempt to get around to confirming this when I've next a chance to mount another BTRFS drive. It'll also allow me to ascertain what tangential aspects are due to MAX_PATH versus due to disallowed characters.