Skip to content

Conversation

nathaniel-d-ef
Copy link
Contributor

Which issue does this PR close?

Rationale for this change

This change adds support for Arrow DataType::Duration to the arrow-avro crate. Previous versions of this code return a NotYetImplemented error encouraging users to manually cast to Interval(MonthDayNano). With this change, Duration will now be represented as a long, with a metadata value of arrowDurationType which corresponds to the TimeUnit. This retains the Arrow paradigm for Duration of clock time, rather than the Avro approach of calendar time. Because of this disconnect between the two specs, any attempt to map the value (e.g. seconds into days) would be lossy, as days and months can be of varying length.

What changes are included in this PR?

Expands the arrow-avro crate to handle Arrow Duration types with various TimeUnit variants (Second, Millisecond, Microsecond, Nanosecond). Includes:

  • Additions in encoder/decoder to support Duration types.
  • Updates to schema handling for duration-specific metadata.
  • Comprehensive unit tests for reading and writing Duration types.

Are these changes tested?

Yes, existing tests are all passing, and tests have been added to validate the encoding and decoding of Duration.

Are there any user-facing changes?

  • Crate is not yet public

@github-actions github-actions bot added arrow Changes to the arrow crate arrow-avro arrow-avro crate labels Sep 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate arrow-avro arrow-avro crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant