This repository was archived by the owner on Jan 12, 2026. It is now read-only.
Merged
Conversation
…o schema checking (apache#8944) # Which issue does this PR close? / Rationale for this change Solves an issue discovered during apache#8790, namely that `ArrowArrayStreamReader` does not correctly expose schema-level metadata and does not check whether the `StructArray` constructed from the FFI stream actually in general corresponds to the expected schema. # What changes are included in this PR? - Change how `RecordBatch` is construted inside `ArrowArrayStreamReader` such that it holds metadata and schema validity checks are done. - Augment FFI tests with schema- and column-level metadata. # Are these changes tested? Yes, both `_test_round_trip_export` and `_test_round_trip_import` now test for metadata on schema- and column-level. # Are there any user-facing changes? Yes, `ArrowArrayStreamReader` now is able to export `RecordBatch` with schema-level metadata, and the interface has increased security since it actually checks for schema validity.
The previous code called Buffer::from to get typed_data, which allocates a new MutableBuffer. # Which issue does this PR close? I believe this PR is small enough to not require an issue. # Rationale for this change Performance. This PR reduces allocations # What changes are included in this PR? See commit # Are these changes tested? By pre-existing tests # Are there any user-facing changes? No Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
# Which issue does this PR close? Part of apache#279, related to apache#8879 # Rationale for this change ``` take check bounds i32 512 time: [338.84 ns 339.71 ns 340.89 ns] change: [−55.056% −54.843% −54.650%] (p = 0.00 < 0.05) Performance has improved. take check bounds i32 1024 time: [548.47 ns 551.23 ns 554.47 ns] change: [−63.999% −63.813% −63.608%] (p = 0.00 < 0.05) Performance has improved. take check bounds i32 8192 time: [3.4934 µs 3.5004 µs 3.5116 µs] change: [−68.252% −68.158% −68.050%] (p = 0.00 < 0.05) Performance has improved. ``` # What changes are included in this PR? Instead of checking index by index, and error-branching individually on them, check all indices for the presence of *any* out-of-bounds index, and only then, in the slow-path, find the exact out-of-bounds index Note how LLVM vectorizes the happy path into a very tight loop: <img width="634" height="155" alt="image" src="https://github.com/user-attachments/assets/64006323-ea7c-4fce-8c74-acc32fc5e7d4" /> https://rust.godbolt.org/z/MhY4cP6WG # Are these changes tested? Existing tests already cover the changed function # Are there any user-facing changes? No
# Which issue does this PR close? - Closes apache#8968. # Rationale for this change - Add `i256::trailing_zeros` to align with other integer types - e.g. [`i128::trailing_zeros`](https://doc.rust-lang.org/std/primitive.i128.html#method.trailing_zeros) - Add function documentation - Add unit tests # What changes are included in this PR? Function, docs, unit tests # Are these changes tested? New unit tests # Are there any user-facing changes? New function in `i256` struct
# Which issue does this PR close? - Related to apache#8940 # Rationale for this change When reviewing apache#8940 it wasn't clear you could create a `VariantPath` from a string Thus let's add an example to the documentation # What changes are included in this PR? Add an example to the documentation # Are these changes tested? Yes, by CI # Are there any user-facing changes? Docs only, no functional change
# Which issue does this PR close? - Closes apache#8965. # Rationale for this change `leading_zeros` exists for other integer types, and in wide `i128` as well. This PR makes it public and tested. It's worth having it as a public part of the struct. # What changes are included in this PR? - Make the function public - Unit tests - Function documentatioon # Are these changes tested? Yes, via unit tests. # Are there any user-facing changes? No Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
# Which issue does this PR close? - Closes apache#8906. # Rationale for this change Support `GenericListViewArray::from_iter_primitive` # What changes are included in this PR? add `GenericListViewArray::from_iter_primitive`, this is basically a copy from `GenericListArray::from_iter_primitive`: https://github.com/apache/arrow-rs/blob/49ed6ef804796ed56512512dc52da0005dd59d75/arrow-array/src/array/list_array.rs#L416-L420 # Are these changes tested? Yes # Are there any user-facing changes? New method added
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.