Skip to content

Add AusTemp SST dataset configuration and refactor regex filtering#251

Closed
Copilot wants to merge 1 commit intoAustempfrom
copilot/sub-pr-250
Closed

Add AusTemp SST dataset configuration and refactor regex filtering#251
Copilot wants to merge 1 commit intoAustempfrom
copilot/sub-pr-250

Conversation

Copy link

Copilot AI commented Feb 16, 2026

Adds support for the AusTemp 8-day Sea Surface Temperature dataset with cloud-optimized Zarr format, including schema definitions for SST variables (sst, sst_mosaic, sst_mosaic_age, sst_anom, sst_anom_mosaic).

Changes

  • New dataset configuration: satellite_austemp_sst_8day.json with complete schema and AWS OpenData Registry metadata
  • Regex filter refactoring: Simplified file filtering in generic_cloud_optimised_creation.py from loop-based to single-pattern approach, leveraging existing field validator that converts filter lists to strings
  • Filter pattern updates: Updated regex patterns in austemp and himawari8 configs (removed $ anchor for consistency with other satellite configs)
  • Worker optimization: Reduced himawari8 nthreads from 4 to 1 for better memory management
  • Documentation: Added/updated Jupyter notebooks for dataset access examples

Example

Before (looped over filter list):

for pattern in path_cfg.filter or []:
    regex = re.compile(pattern)
    matching_files = [f for f in matching_files if regex.search(f)]

After (single pattern, validator ensures string type):

pattern: str = path_cfg.filter or ".*"
regex = re.compile(pattern)
matching_files = [f for f in matching_files if regex.search(f)]

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI mentioned this pull request Feb 16, 2026
Copilot AI changed the title [WIP] Austemp code changes review request Add AusTemp SST dataset configuration and refactor regex filtering Feb 16, 2026
Copilot AI requested a review from lbesnard February 16, 2026 23:21
@lbesnard lbesnard closed this Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants