fsspec path handling

[fjansson]

Proposed as fix for #3201, as an alternative to the one in #3302.

Some filesystems need the scheme as part of the path, while others don't. FsspecStore.from_url() throws an exception if the scheme is left in the path, for any filesystem except http and https. However, the swift fs also needs the scheme in the path. This commit removes the path check, rather than adding more special cases which are difficult to unit test (discussed in #3302).

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.84%. Comparing base (61ca06b) to head (1dbf4f6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3343      +/-   ##
==========================================
- Coverage   61.85%   61.84%   -0.01%     
==========================================
  Files          85       85              
  Lines       10150    10145       -5     
==========================================
- Hits         6278     6274       -4     
+ Misses       3872     3871       -1     

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	68.20% <ø> (-0.34%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

@jhamman could you have a look

[d-v-b]

I would really like the path attribute of FSSpecStore to be consistent with the normal definition of a path in the context of a URL. This means, given the URL "https://mydata.zarr", the FSSpecStore.path should be "mydata.zarr", not "https://mydata.zarr", because that's not the path of the URL!

I understand that fsspec has some design choices that make this tricky, but we should see what we can do to hide that stuff instead of propagating forward

[d-v-b]

@fjansson take a look at https://github.com/d-v-b/zarr-python/tree/remove-fsspec-path-check, would that work for you? Basically I ensure that the Fsspecstore.path is always a valid URL path, but internally we give fsspec the full URL for operations that need it. We need to test this on lots of different fsspec backends though.

[fjansson]

I like the idea: zarr internally has a path without scheme, and passes to fsspec what it wants to see.

I already made a comment on self._url = self.fs.to_dict()["protocol"] + "://" + path - I think this should be self.path where the scheme has been removed.

Then I'm confused: we now pass a full url where it previously was a path. But some of the fsspec file systems cannot handle a scheme in their paths, others demand it. This has previously been handled by calling fs._strip_protocol(path) for each file system, to let them remove the scheme or not.

[fjansson]

@d-v-b Would it be an idea to merge the two commits needed for SWIFT access to work, before the larger path handling effort? My two commits only remove code, fixing the path handling seems a lot more complex.

[d-v-b]

I'm inclined to merge this in order to unblock SWIFT access.