Add arrow adapter to stream record batches zero-copy into csp #519
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ptomecek
reviewed
Apr 7, 2025
| return None | ||
|
|
||
|
|
||
| ArrowHistoricalAdapter = py_pull_adapter_def( |
There was a problem hiding this comment.
Does the docstring for this adapter automatically pull the docstring for the adapter impl? We should make sure it has good documentation
Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
force-pushed
the
ac/arrow_adapter
branch
3 times, most recently
from
5ac31a4 to
7ebce5c
Compare
…pter Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
…rquetReader Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
force-pushed
the
ac/arrow_adapter
branch
from
7ebce5c to
01efd9f
Compare
Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
force-pushed
the
ac/arrow_adapter
branch
from
7082673 to
abb25ed
Compare
Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
…_to_record_batches Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
force-pushed
the
ac/arrow_adapter
branch
from
df34e91 to
92e5ff9
Compare
Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
commented
Apr 30, 2025
| @@ -192,9 +192,17 @@ void SingleTableParquetReader::setColumnAdaptersFromCurrentTable() | |||
| columnAdapter = createColumnAdapter( *this, *field, getCurFileOrTableName(), &getStructColumnMeta() ); | |||
| auto &fieldInfo = fieldsInfo[ index ]; | |||
|
|
|||
| for( std::size_t i = 0; i < fieldInfo.m_width; ++i ) | |||
| if( isArrowIPC() ) | |||
There was a problem hiding this comment.
This is way to check if we are using the InMemoryTableParquetReader
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partially resolves #297