feat(templating): add ${VAR} template variable substitution for YAML …

.envrc

@@ -0,0 +1,3 @@
+
+
+source .venv/bin/activate

docs/templating.md

@@ -0,0 +1,231 @@
+# Template Variable Substitution
+
+Rompy supports template variable substitution in YAML configuration files using `${VAR}` syntax. This allows you to:
+
+- Use environment variables in configs
+- Set default values for missing variables
+- Process datetime values with filters
+- Share configs across different environments
+
+## Syntax
+
+### Basic Substitution
+
+```yaml
+output_dir: "${OUTPUT_ROOT}/my_run"
+run_id: "${RUN_ID}"
+```
+
+### Default Values
+
+Provide fallback values when variables are not set:
+
+```yaml
+output_dir: "${OUTPUT_ROOT:-./output}/my_run"
+timeout: "${JOB_TIMEOUT:-3600}"
+threads: "${NUM_THREADS:-4}"
+```
+
+### Type Conversion
+
+When a YAML value is **exactly** one template expression, type conversion is automatic:
+
+```yaml
+timeout: "${TIMEOUT}"          # "3600" → 3600 (int)
+debug: "${DEBUG}"               # "true" → True (bool)
+pi: "${PI}"                     # "3.14" → 3.14 (float)
+```
+
+Embedded templates always produce strings:
+
+```yaml
+path: "/data/${USER}/file"     # Always string
+```
+
+## Datetime Filters
+
+### Available Filters
+
+| Filter | Description | Example |
+|--------|-------------|---------|
+| `as_datetime` | Parse ISO-8601 datetime | `${CYCLE\|as_datetime}` |
+| `strftime` | Format datetime | `${CYCLE\|strftime:%Y%m%d}` |
+| `shift` | Add/subtract time | `${CYCLE\|shift:-1d}` |
+
+### Filter Chaining
+
+Combine filters with `|`:
+
+```yaml
+previous_day: "${CYCLE|as_datetime|shift:-1d|strftime:%Y-%m-%d}"
+```
+
+### Datetime Examples
+
+```yaml
+cycle_date: "${CYCLE|as_datetime}"
+filename: "wind_${CYCLE|strftime:%Y%m%d}.nc"
+prev_cycle: "${CYCLE|as_datetime|shift:-1d}"
+end_time: "${CYCLE|as_datetime|shift:+24h}"
+```
+
+### Shift Syntax
+
+Time deltas use: `[+|-]<number><unit>`
+
+Units:
+- `d` = days
+- `h` = hours
+- `m` = minutes
+- `s` = seconds
+
+Examples:
+- `+1d` = add 1 day
+- `-6h` = subtract 6 hours
+- `+30m` = add 30 minutes
+
+## Complete Examples
+
+### Basic Config
+
+```yaml
+run_id: "cycle_${CYCLE|strftime:%Y%m%d}"
+
+period:
+  start: "${CYCLE}"
+  end: "${CYCLE|as_datetime|shift:+1d}"
+  interval: "1H"
+
+output_dir: "${OUTPUT_ROOT:-./output}/cycle_${CYCLE|strftime:%Y%m%d}"
+
+input_files:
+  wind: "${DATA_ROOT}/wind/wind_${CYCLE|strftime:%Y%m%d}.nc"
+  wave: "${DATA_ROOT}/wave/wave_${CYCLE|strftime:%Y%m%d}.nc"
+```
+
+Usage:
+```bash
+export CYCLE=2023-01-01T00:00:00
+export DATA_ROOT=/scratch/data
+rompy generate config.yml
+```
+
+### Backend Config
+
+```yaml
+type: local
+timeout: "${JOB_TIMEOUT:-3600}"
+command: "python run_model.py"
+
+env_vars:
+  OMP_NUM_THREADS: "${NUM_THREADS:-4}"
+  WORK_DIR: "${WORK_DIR}"
+```
+
+Usage:
+```bash
+export WORK_DIR=/scratch/my_job
+export NUM_THREADS=8
+rompy run config.yml --backend-config backend.yml
+```
+
+### Lookback Pattern
+
+Access previous time periods:
+
+```yaml
+input_files:
+  current: "${DATA_ROOT}/data_${CYCLE|strftime:%Y%m%d}.nc"
+  previous: "${DATA_ROOT}/data_${CYCLE|as_datetime|shift:-1d|strftime:%Y%m%d}.nc"
+  week_ago: "${DATA_ROOT}/data_${CYCLE|as_datetime|shift:-7d|strftime:%Y%m%d}.nc"
+```
+
+### Nested Directory Structures
+
+```yaml
+output_dir: "${DATA_ROOT}/output/${CYCLE|strftime:%Y/%m/%d}"
+```
+
+With `CYCLE=2023-01-15T00:00:00` → `/data/output/2023/01/15`
+
+## How It Works
+
+Template rendering happens **after YAML parsing** but **before Pydantic validation**:
+
+```
+1. Load YAML file → dict
+2. Render templates: ${VAR} → actual values
+3. Pydantic validation: dict → ModelRun object
+```
+
+This ensures:
+- Type safety (Pydantic sees resolved values)
+- Clear error messages (template errors before validation errors)
+- Datetime objects work with Pydantic models
+
+## Error Handling
+
+### Missing Variables
+
+By default, missing variables cause an error:
+
+```yaml
+path: "${MISSING_VAR}"  # Error: Variable 'MISSING_VAR' not found
+```
+
+Use defaults to make variables optional:
+
+```yaml
+path: "${OPTIONAL_VAR:-/default/path}"  # OK if OPTIONAL_VAR not set
+```
+
+### Invalid Filters
+
+Unknown filters produce clear errors:
+
+```yaml
+date: "${CYCLE|unknown_filter}"  # Error: Unknown filter 'unknown_filter'
+```
+
+### Datetime Parsing
+
+Invalid datetime strings fail early:
+
+```yaml
+date: "${CYCLE|as_datetime}"  # Error if CYCLE is not ISO-8601 format
+```
+
+## Tips
+
+### Quote Values with `:-`
+
+YAML interprets `:` as mapping syntax. Quote defaults containing colons:
+
+```yaml
+path: "${VAR:-/path/with:colon}"  # GOOD - quoted
+path: ${VAR:-/path/with:colon}    # BAD - YAML parse error
+```
+
+### Environment Variables
+
+Templates use `os.environ` by default:
+
+```bash
+export MY_VAR=value
+rompy generate config.yml  # ${MY_VAR} resolved automatically
+```
+
+### Separation from Jinja2
+
+Don't confuse with rompy's existing Jinja2 templates (used for model control files):
+
+- `${VAR}` = **Config templating** (pre-load, env vars)
+- `{{runtime.var}}` = **File templating** (post-load, Python objects)
+
+They serve different purposes and run at different times.
+
+## See Also
+
+- Example configs: `examples/configs/templated_*.yml`
+- Tests: `tests/test_templating.py`
+- Implementation: `src/rompy/templating.py`

examples/configs/templated_advanced.yml

@@ -0,0 +1,35 @@
+# Example: Advanced Templated Configuration
+# Shows datetime processing and complex filter chains
+
+# Environment variables:
+# - CYCLE: ISO datetime (e.g., 2023-01-15T00:00:00)
+# - DATA_ROOT: Base data directory
+# - FORECAST_HOURS: Hours to forecast (defaults to 24)
+
+run_id: "forecast_${CYCLE|strftime:%Y%m%d_%H%M}"
+
+period:
+  start: "${CYCLE}"
+  end: "${CYCLE|as_datetime|shift:+${FORECAST_HOURS:-24}h}"
+  interval: "1H"
+
+output_dir: "${DATA_ROOT}/output/${CYCLE|strftime:%Y/%m/%d}"
+
+input_files:
+  wind:
+    current: "${DATA_ROOT}/wind/wind_${CYCLE|strftime:%Y%m%d}.nc"
+    previous: "${DATA_ROOT}/wind/wind_${CYCLE|as_datetime|shift:-1d|strftime:%Y%m%d}.nc"
+
+  wave:
+    current: "${DATA_ROOT}/wave/wave_${CYCLE|strftime:%Y%m%d}.nc"
+    forecast: "${DATA_ROOT}/wave/wave_${CYCLE|as_datetime|shift:+1d|strftime:%Y%m%d}.nc"
+
+config:
+  lookback_days: 3
+  lookback_start: "${CYCLE|as_datetime|shift:-3d}"
+
+# Usage with datetime arithmetic:
+# export CYCLE=2023-01-15T12:00:00
+# export DATA_ROOT=/data/ocean
+# export FORECAST_HOURS=48
+# rompy generate examples/configs/templated_advanced.yml

examples/configs/templated_backend.yml

@@ -0,0 +1,23 @@
+# Example: Templated Backend Configuration
+# Demonstrates template variable substitution in backend configs
+
+# Environment variables:
+# - NUM_THREADS: Number of CPU threads (defaults to 4)
+# - JOB_TIMEOUT: Timeout in seconds (defaults to 3600)
+# - WORK_DIR: Working directory path
+
+type: local
+
+timeout: "${JOB_TIMEOUT:-3600}"
+
+command: "python run_model.py"
+
+env_vars:
+  OMP_NUM_THREADS: "${NUM_THREADS:-4}"
+  MODEL_TYPE: "production"
+  WORK_DIR: "${WORK_DIR}"
+
+# Usage:
+# export WORK_DIR=/scratch/my_job
+# export NUM_THREADS=8
+# rompy run config.yml --backend-config templated_backend.yml

examples/configs/templated_modelrun.yml

@@ -0,0 +1,28 @@
+# Example: Templated ModelRun Configuration
+# This demonstrates template variable substitution in rompy configs
+
+# Environment variables required:
+# - CYCLE: ISO datetime string (e.g., 2023-01-01T00:00:00)
+# - DATA_ROOT: Path to input data directory
+# - OUTPUT_ROOT: (optional) Path to output directory (defaults to ./output)
+
+run_id: "cycle_${CYCLE|strftime:%Y%m%d}"
+
+period:
+  start: "${CYCLE}"
+  end: "${CYCLE|as_datetime|shift:+1d}"
+  interval: "1H"
+
+output_dir: "${OUTPUT_ROOT:-./output}/cycle_${CYCLE|strftime:%Y%m%d}"
+delete_existing: true
+
+# Example input files with templated paths
+input_files:
+  wind: "${DATA_ROOT}/wind/wind_${CYCLE|strftime:%Y%m%d}.nc"
+  wave: "${DATA_ROOT}/wave/wave_${CYCLE|strftime:%Y%m%d}.nc"
+
+# Usage:
+# export CYCLE=2023-01-01T00:00:00
+# export DATA_ROOT=/scratch/data
+# export OUTPUT_ROOT=/scratch/output  # Optional
+# rompy generate examples/configs/templated_modelrun.yml -v

src/rompy/cli.py

@@ -21,6 +21,7 @@
 from rompy.backends import DockerConfig, LocalConfig, SlurmConfig
 from rompy.logging import LogFormat, LoggingConfig, LogLevel, get_logger
 from rompy.model import PIPELINE_BACKENDS, POSTPROCESSORS, RUN_BACKENDS, ModelRun
+from rompy.templating import render_templates
 
 # Initialize the logger
 logger = get_logger(__name__)
@@ -165,11 +166,20 @@ def load_config(
     try:
         config = yaml.safe_load(content)
         logger.info("Parsed config as YAML")
-        return config
     except yaml.YAMLError as e:
         logger.error(f"Failed to parse config as JSON or YAML: {e}")
         raise click.UsageError("Config file is not valid JSON or YAML")
 
+    # Render template variables in config
+    try:
+        config = render_templates(config, context=dict(os.environ), strict=True)
+        logger.debug("Template variables rendered successfully")
+    except Exception as e:
+        logger.error(f"Failed to render template variables: {e}")
+        raise click.UsageError(f"Template rendering error: {e}")
+
+    return config
+
 
 def print_version(ctx, param, value):
     """Callback to print version and exit."""

src/rompy/core/config.py

@@ -1,6 +1,6 @@
 import logging
 from pathlib import Path
-from typing import Literal, Optional, Any
+from typing import Literal, Optional
 
 from pydantic import Field
 

src/rompy/core/source.py

@@ -1,6 +1,5 @@
 """Rompy source objects."""
 
-import logging
 from abc import ABC, abstractmethod
 from functools import cached_property
 from pathlib import Path

src/rompy/run/docker.py

@@ -8,7 +8,6 @@
 import json
 import logging
 import pathlib
-import subprocess
 import time
 from typing import TYPE_CHECKING, Dict, List, Optional
 
@@ -289,7 +288,7 @@ def _run_container(
             # Note: When remove=True, client.containers.run() returns None
             # If you need to capture output, you'd need to set remove=False and manually remove
             client.containers.run(**container_config)
-            
+
             logger.info("Model run completed successfully")
             return True