Skip to content

Implement val_temporal_unit for deciding how datetimes and dates timestamps get validated. #1751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

ollz272
Copy link
Contributor

@ollz272 ollz272 commented Jun 30, 2025

Change Summary

Follow up to #1743 and a part of pydantic/pydantic#11504 implementation - allows the config switch val_temporal_unit to be used to determine how dates and datetimes timestamps are validated - either as seconds, milliseconds, or inferred.

NOTE - the original ticket makes mention of this also working for time and duration. As far as i can see, we do not currently infer the value of these, they're just assumed to be seconds. Theres no support for this in speedate anyway. So I've implemented what is currently possible. I see 3 possible outcomes from this:

  1. We decide that this is fine for all time and this only supports date/datetime
  2. We decide that it should support time/duration, but its fine to get this part done first and add support for those later
  3. We decide that it should support time/duration, and it needs to go in at the same time.

Related issue number

Checklist

  • Unit tests for the changes exist
  • Documentation reflects the changes where applicable
  • Pydantic tests pass with this pydantic-core (except for expected changes)
  • My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @davidhewitt

Copy link

codecov bot commented Jun 30, 2025

Copy link

codspeed-hq bot commented Jun 30, 2025

CodSpeed Performance Report

Merging #1751 will not alter performance

Comparing ollz272:val_temporal_unit_2 (e984613) with main (d523cf5)

Summary

✅ 157 untouched benchmarks

@ollz272
Copy link
Contributor Author

ollz272 commented Jul 1, 2025

Note - added a test here as this is the pr I discovered the issue with fractional millisecond timestamps (pydantic/pydantic#12027)

@ollz272
Copy link
Contributor Author

ollz272 commented Jul 1, 2025

Please review

@ollz272
Copy link
Contributor Author

ollz272 commented Jul 14, 2025

Note - i've also added a fix for: #1750 in this commit: 67fcdc3

Copy link
Contributor

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, and sorry for the slow review.

Overall looks great, with just a couple thoughts.

We decide that it should support time/duration, but its fine to get this part done first and add support for those later

This feels ok to me, let's just sense check:

  • presumably for milliseconds time values these can never overlap with seconds time values due to the limit of time being 24 hours. So this is safe to go in later as a new feature (would make values previously not accepted be valid).
  • for duration, the scale is presumably the same as datetime so we should consider if it makes sense to have inference there at all. IIRC the milliseconds scale kicks in for a timestamp which is on the order of hundreds of years? (we should sense check this). So it's unlikely but possible that this changes the meaning of valid values. In which case implementing this might be a separate issue we should consider for v3?

Comment on lines 372 to +373
microseconds_overflow_behavior: MicrosecondsPrecisionOverflowBehavior,
mode: TemporalUnitMode,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if it makes sense to just pass &DateTimeConfig here to simplify things?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think i chose not to as i wasn't a fan of using something from the serialization module in the validation logic. Happy to be overridden though!

Comment on lines +448 to +450
// Use the same watershed from speedate to determine if we treat the float as seconds or milliseconds.
// TODO: should we expose this from speedate?
if timestamp.abs() <= 20_000_000_000.0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially worth adding a utility function in speedate DateTime::from_float_with_config?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ollz272
Copy link
Contributor Author

ollz272 commented Jul 16, 2025

Thanks for working on this, and sorry for the slow review.

Overall looks great, with just a couple thoughts.

We decide that it should support time/duration, but its fine to get this part done first and add support for those later

This feels ok to me, let's just sense check:

  • presumably for milliseconds time values these can never overlap with seconds time values due to the limit of time being 24 hours. So this is safe to go in later as a new feature (would make values previously not accepted be valid).
  • for duration, the scale is presumably the same as datetime so we should consider if it makes sense to have inference there at all. IIRC the milliseconds scale kicks in for a timestamp which is on the order of hundreds of years? (we should sense check this). So it's unlikely but possible that this changes the meaning of valid values. In which case implementing this might be a separate issue we should consider for v3?

Both make sense to me 👍🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants