Deadlock when using http kvstore within virtual_chunked

[sjperkins]

Environment Details

$ uname -o -m -s -v -r
Linux 6.8.0-85-generic #85-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 18 15:26:59 UTC 2025 x86_64 GNU/Linux
$ python --version
Python 3.11.13
$ uv pip freeze | grep tensorstore
tensorstore==0.1.78

I'm accessing some npy files on a http/s3 store with the following simplified pattern:

import tensorstore as ts

def virtual_store_factory(endpoint: str, chunk_info: Dict):
  http_spec = { "driver": "http", "base_url": endpoint }
  http_store = ts.KvStore.open(http_spec).result()
  
  def read(domain, array, params):
    key_bits = [f"{o:05}" for o in domain.origin]
    key = f"prefix/{'_'.join(key_bits)}.npy"
    read_result = http_store.read(key).result()
    data = np.frombuffer(read_result.value[header_len:], dtype=dtype)   # Simplification
    tensor_data = ts.array(data)
    array[...] = tensor_data[domain.translate_backward_by[domain.origin]]
    return read_result.stamp

  return ts.virtual_chunked(
    read_function=read,  
    rank=len(chunk_info["chunks"]),
    shape=tuple(sum(dc) for dc in chunk_info["chunks"]),
    dtype=chunk_info["dtype"],
    chunk_layout=ts.ChunkLayout(chunk_shape=[c[0] for c in chunk_info["chunks"])
   )

If I then try use the store:

virtual_store = virtual_store_factory(...)
data = virtual_store[index].read().result()

I see from prints that all read_result = http_store.read(key).result() futures are issued and completed, but for some reason data = virtual_store[index].read().result() does not complete.

Question

• Is using the http store within the virtual store conceptually incorrect in some way?
• although, I imagine the http_store reference is retained within read's closure variable as long as virtual_store is alive.
• Is there perhaps some more appropriate way of achieving the same result?

[sjperkins]

Probably the reason is given in the documentation:

Neither read_function nor write_function should block synchronously while waiting for another TensorStore operation; blocking on another operation that uses the same Context.data_copy_concurrency resource may result in deadlock. Instead, it is better to specify a coroutine function for read_function and write_function and use await to wait for the result of other TensorStore operations.

I'll make the necessary changes and close the issue if the above advice succeeds.

[jbms]

Another option is to specify a separate context with explicit data_copy_concurrency limit for the virtual_chunked object.

[sjperkins]

Another option is to specify a separate context with explicit data_copy_concurrency limit for the virtual_chunked object.

Thanks, this worked.

[sjperkins]

In the configuration described above #253 (comment), is there any benefit to replacing http_store.read(key).result() with asyncio.run_coroutine_threadsafe(async_read_wrapper, loop) while keeping the virtual_chunked read_function synchronous? I've got this working and my thinking is that it's better to block on the completion of async I/O coroutines than futures encapsulating I/O threads.

Managing an async event loop comes with it's own complexities though.

[sjperkins]

is there any benefit to replacing

Perhaps the http kvstore uses threads for concurrency internally, while presenting an async interface at the Python level?

[jbms]

The http kvstore does not use the data_copy_concurrency thread pool. You can determine this from the fact that the spec does not list a DataCopyConcurrency resource: https://google.github.io/tensorstore/kvstore/http/index.html#http-key-value-store-driver

As a side note, in fact the http kvstore currently does use a separate thread pool for requests, even though it ultimately dispatches to our async http request machinery in tensorstore, because unlike the gcs and s3 drivers it hasn't been updated to be truly async.

The problem is actually the ts.array call --- even though ts.array operations are synchronous, all TensorStore objects need an associated executor that is used for read and write operations, and ts.array uses the default context if none is specified (https://google.github.io/tensorstore/python/api/tensorstore.array.html#)

Therefore reads from tensor_data are blocking on the same thread pool as is used for virtual_chunked.

As far as using asyncio.run_coroutine_threadsafe, I would have to see the modified program to give a proper assessment, but if you kept the call to ts.array within the sync virtual_chunked read function, then it may have only worked by accident, e.g. due to timing.

In general I would say it is probably easiest to either use asyncio entirely or not at all --- mixing may be more confusing and I don't know that it will provide any benefit.

[sjperkins]

Thank you for summarising that, that's useful to know.

Therefore reads from tensor_data are blocking on the same thread pool as is used for virtual_chunked.

This is interesting, I hadn't fully internalised that a ts.array was a tensorstore but on reflection this makes sense in the context of tensorstore's Expressions framework, which I surmise uses multiple threads to accelerate individual operations.

One operation that we need to perform (and is omitted from the original example) is a reshape and transpose of the data:

data = http_store.read(key).result()
# data shape is (time, frequency, corrprod)
data = np.frombuffer(read_result.value[header_len:], dtype=dtype)

tensor_data = ts.array(data)[..., cp_argsort].reshape(time, frequency, baseline, polarization).transpose(0, 2, 1, 3)

I assume doing this via a tensorstore Expression would be faster than using NumPy, due to the multi-threaded data copy ops? However, this would depend on implementation of the reshape functionality mentioned here:

• Add TIFF Key-Value Store Implementation (take 2) #230 (comment)

This isn't critical NumPy is fine for our current exploratory work.