[WIP] Support Shredded Lists/Array in variant_get
#8354
Conversation
| let main_struct = crate::variant_array::StructArrayBuilder::new() | ||
| .with_field("metadata", Arc::new(metadata_array)) | ||
| .with_field("value", Arc::new(value_array)) | ||
| .with_field("typed_value", Arc::new(typed_value_array)) |
There was a problem hiding this comment.
Check the variant shredding spec for arrays -- the typed_value for a shredded variant array is a non-nullable group called element, with child fields typed_value and value for shredded and unshredded list elements, respectively.
And then we'll need to build an appropriate GenericListArray out of this string array you built, which gives the offsets for each sub-list.
There was a problem hiding this comment.
Thanks for this too, I was under the wrong impression that the metadata encoding stores the offsets for the actual values. Reading your #8359 and rereading the Variant Encoding spec I see that the values offsets are within the value encoding itself.
So the outermost typed_value should be an GenericListArray of element - VariantObjects with {value and typed_value fields}?
There was a problem hiding this comment.
yes, exactly! And element is non-nullable (**), while the two children are nullable.
(**) As always, in arrow, it can still have null entries, but only if its parent is already NULL for the same row (so nobody can ever observe a non-null element)
…ed_list_support
Could you elaborate please? I am currently trying to build just the Shredded |
No worries -- the current iteration does look it produces a correct shredded variant containing a list, so I should probably just be patient and let you finish! |
|
My question is: does The reason I am asking is that since we use the output of The only way to work with list arrays I came up with so far, is to build new arrays with
|
| // Build the list of indices to take | ||
| let mut take_indices = Vec::with_capacity(list_len); | ||
| for i in 0..list_len { | ||
| let start = offsets[i] as usize; | ||
| let end = offsets[i + 1] as usize; | ||
| let len = end - start; | ||
|
|
||
| if *index < len { | ||
| take_indices.push(Some((start + index) as u32)); | ||
| } else { | ||
| take_indices.push(None); | ||
| } | ||
| } | ||
|
|
||
| let index_array = UInt32Array::from(take_indices); | ||
|
|
||
| // Use Arrow compute kernel to gather elements | ||
| let taken = take(field_array, &index_array, None)?; |
There was a problem hiding this comment.
You can see the basic idea here
…ed_list_support
…ed_list_support
|
Hey @scovich I made it work for a one of the simple tests and it doesn't go through with the second one because Variant to Arrow does not support utf8 yet. Do we have an issue tracking variant_to_arrow types support? If not, I can make one. |
I'm not sure we have a tracking issue for utf8 support in variant_to_arrow, but I've also noticed that it's an annoying gap for unit testing (we all seem to reach for string values...) |
Which issue does this PR close?
variant_get#8082.Rationale for this change
We should be able to read lists using
variant_getWhat changes are included in this PR?
Are these changes tested?
I'm trying to start with some basic tests to do some TDD.
Are there any user-facing changes?