[lake/paimon] Support Array types for tiering paimon #2166

binary-signal · 2025-12-13T12:24:55Z

Purpose

Linked Issue: #2250

This PR adds support for ARRAY data types in the Fluss Lake Paimon integration. Previously, attempting to use array types when tiering data to Paimon would throw an UnsupportedOperationException. This change enables proper conversion between Fluss and Paimon array types in both directions.

Brief change log

Added FlussArrayAsPaimonArray: Adapter class that wraps org.apache.fluss.row.InternalArray and implements org.apache.paimon.data.InternalArray, enabling Fluss arrays to be read as Paimon arrays.
Added PaimonArrayAsFlussArray: Adapter class that wraps org.apache.paimon.data.InternalArray and implements org.apache.fluss.row.InternalArray, enabling Paimon arrays to be read as Fluss arrays.
Updated FlussRowAsPaimonRow.getArray(): Now returns a FlussArrayAsPaimonArray wrapper instead of throwing UnsupportedOperationException.
Updated PaimonRowAsFlussRow.getArray(): Now returns a PaimonArrayAsFlussArray wrapper instead of throwing UnsupportedOperationException.

Tests

FlussRowAsPaimonRowTest (10 new test methods):

testArrayTypeWithIntElements() - Verifies array of integers can be read correctly
testArrayTypeWithStringElements() - Verifies array of strings can be read correctly
testArrayTypeWithNullableElements() - Verifies arrays with null elements are handled properly
testNullArray() - Verifies null array fields return isNullAt() == true
testNestedArrayType() - Verifies nested arrays (ARRAY<ARRAY<INT>>) work correctly
testEmptyArray() - Verifies empty arrays are handled correctly
testArrayWithAllPrimitiveTypes() - Verifies all primitive array types and toXArray() conversions
testArrayWithDecimalElements() - Verifies ARRAY<DECIMAL> type conversion
testArrayWithTimestampElements() - Verifies ARRAY<TIMESTAMP> type conversion
testArrayWithBinaryElements() - Verifies ARRAY<VARBINARY> type conversion

PaimonRowAsFlussRowTest (new test class - 15 test methods):

testArrayTypeWithIntElements() - Verifies Paimon int arrays convert to Fluss correctly
testArrayTypeWithStringElements() - Verifies Paimon string arrays convert to Fluss correctly
testArrayTypeWithNullableElements() - Verifies arrays with null elements are handled
testNullArray() - Verifies null array fields are handled
testNestedArrayType() - Verifies nested arrays work in Paimon→Fluss direction
testEmptyArray() - Verifies empty arrays are handled
testReplaceRow() - Verifies replaceRow() works correctly with array fields
testArrayWithLongElements() - Verifies ARRAY<BIGINT> conversion
testArrayWithDoubleElements() - Verifies ARRAY<DOUBLE> conversion
testArrayWithBooleanElements() - Verifies ARRAY<BOOLEAN> conversion
testArrayWithAllPrimitiveTypes() - Verifies all primitive types and toXArray() methods
testArrayWithDecimalElements() - Verifies ARRAY<DECIMAL> conversion
testArrayWithTimestampElements() - Verifies both getTimestampNtz() and getTimestampLtz()
testArrayWithBinaryElements() - Verifies both getBinary() and getBytes() methods
testArrayWithCharElements() - Verifies both getChar() and getString() methods

FlussRecordAsPaimonRowTest (10 new test methods):

testArrayTypeWithIntElements() - Verifies int arrays work with system columns in tiering context
testArrayTypeWithStringElements() - Verifies string arrays work in tiering context
testNestedArrayType() - Verifies nested arrays work in tiering context
testArrayWithAllPrimitiveTypes() - Verifies all primitive types and toXArray() methods with system columns
testArrayWithDecimalElements() - Verifies ARRAY<DECIMAL> in tiering context
testArrayWithTimestampElements() - Verifies ARRAY<TIMESTAMP> in tiering context
testArrayWithBinaryElements() - Verifies ARRAY<VARBINARY> in tiering context
testNullArray() - Verifies null arrays in tiering context
testArrayWithNullableElements() - Verifies arrays with null elements in tiering context
testEmptyArray() - Verifies empty arrays in tiering context

API and Format

No. This change is an internal implementation that enables existing array type support to work correctly with Paimon tiering. No public API changes.

Documentation

No documentation changes required. Array types were already documented as a supported data type; this PR fixes the implementation to actually support them in the Paimon lake integration.

binary-signal · 2025-12-13T17:07:25Z

added few mores tests cases hoping coverage will be okay

binary-signal · 2025-12-13T17:33:48Z

Adding also 2 PR's related to fluss arrays for visibility
#2165

#2158

Copilot

Pull request overview

This PR implements support for ARRAY and nested ARRAY data types in the Fluss Lake Paimon tiering service, addressing issue #1980. Previously, attempting to use array types would result in an UnsupportedOperationException. The implementation provides bidirectional conversion between Fluss and Paimon array representations through adapter classes.

Introduced two adapter classes to enable seamless array type conversion between Fluss and Paimon
Updated row conversion methods to return wrapped array instances instead of throwing exceptions
Added comprehensive test coverage with 35 new test methods across three test files

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
FlussArrayAsPaimonArray.java	New adapter class that wraps Fluss InternalArray to implement Paimon InternalArray interface, enabling Fluss→Paimon array conversion with support for all primitive types, decimals, timestamps, and nested arrays
PaimonArrayAsFlussArray.java	New adapter class that wraps Paimon InternalArray to implement Fluss InternalArray interface, enabling Paimon→Fluss array conversion with proper handling of both TimestampNtz and TimestampLtz types
FlussRowAsPaimonRow.java	Updated getArray() method to return FlussArrayAsPaimonArray wrapper instead of throwing UnsupportedOperationException
PaimonRowAsFlussRow.java	Updated getArray() method to return PaimonArrayAsFlussArray wrapper instead of throwing UnsupportedOperationException
PaimonRowAsFlussRowTest.java	New test class with 15 comprehensive test methods validating Paimon→Fluss array conversion including primitives, strings, decimals, timestamps, null handling, and nested arrays
FlussRecordAsPaimonRowTest.java	Added 10 test methods verifying array support in the tiering context with system columns, covering all primitive types and edge cases
FlussRowAsPaimonRowTest.java	Added 10 test methods validating Fluss→Paimon array conversion for various data types including nested arrays, null values, and empty arrays

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-15T06:42:22Z

...ke-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/FlussRecordAsPaimonRowTest.java

+        InternalArray innerArray2 = outerArray.getArray(1);
+        assertThat(innerArray2.size()).isEqualTo(3);
+        assertThat(innerArray2.getInt(0)).isEqualTo(3);
+    }


The test assertions for innerArray2 are incomplete. The array contains 3 elements (3, 4, 5) based on the test setup at line 266, but only the first element is verified. The assertions for elements at indices 1 and 2 are missing, making this test less comprehensive than the equivalent tests in PaimonRowAsFlussRowTest and FlussRowAsPaimonRowTest where all elements are verified.

luoyuxia

@binary-signal Thanks for the pr.
Also add IT to vierfy array type can be tiered to paimon and unioin read by flink.
Maybe you can add the it in FlinkUnionReadPrimaryKeyTableITCase#testUnionReadFullType

luoyuxia · 2025-12-15T06:53:41Z

...s-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/source/FlussRowAsPaimonRowTest.java

+    }
+
+    @Test
+    void testArrayWithAllPrimitiveTypes() {


is it possible to put test for all types in array in this one test method instead having testArrayWithLongElements, testArrayWithDoubleElements, etc...

Yes that make sense I am on it

luoyuxia · 2025-12-15T06:54:34Z

...ke-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/FlussRecordAsPaimonRowTest.java

+    }
+
+    @Test
+    void testArrayWithAllPrimitiveTypes() {


dito. Is it possible to test in one single test method instead of introduing many other test method

luoyuxia · 2025-12-15T06:55:13Z

...ss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/utils/PaimonRowAsFlussRowTest.java

+    }
+
+    @Test
+    void testArrayWithAllPrimitiveTypes() {


dito.
Is it possible to reduce test methods?

wuchong · 2025-12-21T11:15:05Z

Hi @binary-signal , do you have time to update the pull request? If you're busy, we’d be happy to help with the changes and get it merged.

wuchong · 2025-12-24T10:10:01Z

@luoyuxia could you help to update the PR and I will make a final review.

luoyuxia · 2025-12-25T02:27:51Z

@wuchong Sure. I'll help update the pr

Signed-off-by: binary-signal <binary-signal@github.noreply.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/source/FlussArrayAsPaimonArray.java

...ss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/utils/PaimonRowAsFlussRowTest.java

...s-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/source/FlussRowAsPaimonRowTest.java

luoyuxia · 2025-12-25T07:22:34Z

@wuchong Hi, i already updated the pr. Please help review when you got some time

wuchong requested a review from luoyuxia December 13, 2025 14:48

binary-signal changed the title ~~[lake/paimon] Add ARRAY type support for Fluss-Paimon row conversion~~ [lake/paimon] Support Array types in tiering service Dec 13, 2025

binary-signal changed the title ~~[lake/paimon] Support Array types in tiering service~~ [lake/paimon] Support Array/Nested Array types in tiering service Dec 13, 2025

luoyuxia requested a review from Copilot December 15, 2025 06:33

Copilot started reviewing on behalf of luoyuxia December 15, 2025 06:34 View session

Copilot AI reviewed Dec 15, 2025

View reviewed changes

luoyuxia reviewed Dec 15, 2025

View reviewed changes

binary-signal added 3 commits December 25, 2025 11:58

add adapters for converting fluss arrays to paimon arrays and vice versa

1b2580c

Signed-off-by: binary-signal <binary-signal@github.noreply.com>

add more tests with array primitives

51a1a8f

Signed-off-by: binary-signal <binary-signal@github.noreply.com>

more tests for fluss record paimon rows in tiering

cd7f615

Signed-off-by: binary-signal <binary-signal@github.noreply.com>

luoyuxia force-pushed the fluss-paimon-array-adapter branch from bb18842 to c22e6b7 Compare December 25, 2025 04:01

luoyuxia requested a review from Copilot December 25, 2025 04:02

Copilot started reviewing on behalf of luoyuxia December 25, 2025 04:02 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

improve test

245e87d

luoyuxia force-pushed the fluss-paimon-array-adapter branch from c22e6b7 to 245e87d Compare December 25, 2025 04:48

luoyuxia changed the title ~~[lake/paimon] Support Array/Nested Array types in tiering service~~ [lake/paimon] Support Array types for tiering paimon Dec 25, 2025

XuQianJin-Stars mentioned this pull request Dec 25, 2025

Support tier array type for iceberg #2252

Open

2 tasks

[lake/paimon] Support Array types for tiering paimon #2166

Are you sure you want to change the base?

[lake/paimon] Support Array types for tiering paimon #2166

Conversation

binary-signal commented Dec 13, 2025 • edited by luoyuxia Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

binary-signal commented Dec 13, 2025

Uh oh!

binary-signal commented Dec 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

luoyuxia Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

binary-signal Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

luoyuxia Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

luoyuxia Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

wuchong commented Dec 21, 2025

Uh oh!

wuchong commented Dec 24, 2025

Uh oh!

luoyuxia commented Dec 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luoyuxia commented Dec 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

binary-signal commented Dec 13, 2025 •

edited by luoyuxia

Loading