questions on the longterm roadmap

[tswast]

Hello @xando ! Thanks for creating this project. As one of the original developers on the google-cloud-bigquery-storage package, this caught my eye. I'm interested in contributing as find capacity, and had a few questions about if you're open to some larger changes to this project?

Interface

I'm curious if you've explored the pyarrow Tabular Dataset interface? It provides some really handy features that seem like they'd be a good fit with the BigQuery Storage Read API interface:

A unified interface that supports different sources and file formats and different file systems (local, cloud).
-Discovery of sources (crawling directories, handle directory based partitioned datasets, basic schema normalization,)
Optimized reading with predicate pushdown (filtering rows), projection (selecting and deriving columns), and optionally parallel reading.

That said, I'm not sure how much could be done in this package versus would have to be done in the Arrow C++ package for that. as I'm not seeing extension information regarding Dataset in the official pyarrow docs.

Performance

I ran some code using pyarrow-bigquery through the p-spy sampling profiler and discovered about 1/3 of the time (estimated) is spent writing feather data to disk for worker communication:

I also see a lot of "grpc spin". I think that means just waiting around for data, which might imply some other bottlenecks preventing grpc from reading data. Some of this may be GIL, since the "process" worker does have a speedup in my tests.

Any thoughts on creating a Cython (wrapping the C++ client) or PyO3 (wrapping one of the community Rust clients) version that could potentially avoid some of the slowness Python introduces, especially around sharing data across the workers?

Thanks for your consideration,

Tim

[tswast]

Re: Dataset interface, it looks like the Arrow project may be open to a contribution there: apache/arrow#21199 though the last discussion was in 2019.

[xando]

Hey @tswast — thanks for taking the time to look at the project, and sorry for the long silence.

1. Interface
   I’ll look into this. I’m generally open to changing the API to better fit the dataset. My primary goal was to provide a cleaner way to download data directly from BigQuery into PyArrow; the API was secondary.

2. Performance
   There are a couple of things here:

   • I recently changed how workers exchange memory with the main process. Using Feather for threading wasn’t great, so I introduced the concept of exchanges. For threads this is just in-process (exchange/memory.py). exchange/feather.py is the default for multiprocessing, although there’s also a sharedmemory exchange. You can swap them like this:

     source = "bigquery-public-data.hacker_news.full"
     
     _t = time.time()
     table = bq.read_table(
         project=project,
         source=source,
         worker_type=multiprocessing.Process,  # Use multiprocessing for parallel reads
         ipc_exchange=SharedMemory(),          # Use SharedMemory for IPC
         batch_size=2000,
     )
     print(f"Time taken: {time.time() - _t:.2f} seconds")

   • I don’t have much experience with Rust or Cython, but I suspect (especially with Rust) it would be fairly straightforward to implement things (with some LLM help)

Question:
When I was working on this, Given your experience with Google products. it was important to support writing without a fixed Protobuf schema. From what I can tell, the usual Python approach is to manually define .proto files, and there’s no built-in way to compile Protobufs at runtime through the Python API. Is that correct, or am I missing an option?

[tswast]

I was not able to get proto schemas generated at runtime in Python. Thankfully the Write API recently added Arrow support, which should work more nicely with Python.

[xando]

Thanks @tswast

I was unaware that this was has been released, I looked into this. Basic example with pyarrow and does what I wanted.