Cc schemas by slawler · Pull Request #12 · fema-ffrd/specs

slawler · 2025-09-20T18:59:17Z

Summary

This PR implements a comprehensive schema documentation system and consolidates JSON schemas for the FFRD container specifications. The main focus is on creating automated documentation generation for JSON schemas and establishing a solid foundation for base image specifications.

Key Features Added

Automated Schema Documentation Generation

Created json_to_md.py script for converting JSON schemas to markdown documentation
Multi-directory processing - supports processing multiple schema/doc directory pairs
Reference resolution - automatically resolves $ref links between schemas
AllOf schema merging - properly handles complex schema inheritance
Clean table formatting - separates examples from property tables for better readability
Nested properties support - displays object properties with proper hierarchy

Schema Structure Improvements

Consolidated base schemas in reference/base/schemas/:
- action.json - Base action schema with common properties
- data.json - Data path and source definitions
- store.json - Storage configuration schema
RAS-specific schemas in reference/ras/v660/schemas/:
- action.ras.run_unsteady_simulation.json - Extends base action for RAS workflows

Documentation Enhancements

Auto-generated markdown with clean table formatting:
- Property tables without cluttered examples column
- Dedicated examples sections below tables
- Explicit references section showing schema inheritance
Cross-references between schemas with clickable links
MkDocs integration with proper include paths

Infrastructure Updates

Base image Docker configuration and build scripts
RAS v6.6.0 container updates with new schema implementation
Validation scripts for schema compliance
Example configurations for different deployment scenarios

Schema Documentation Links

The generated documentation now includes:

Base Image Schemas: docs/draft/base_image/
RAS Simulation Schemas: docs/draft/ras_sim/
Automatic cross-linking between related schemas

Files Changed

New: Schema documentation generation system
Enhanced: Base and RAS container implementations
Improved: MkDocs configuration for better schema presentation
Added: Comprehensive validation and example configurations

Testing

MkDocs builds successfully with new schema documentation
Schema validation passes for all JSON files
Cross-references between schemas work correctly
Examples render properly in generated documentation

Ras container

Copilot

Pull Request Overview

This PR implements a comprehensive schema documentation system and consolidates JSON schemas for the FFRD container specifications, with a focus on creating automated documentation generation for JSON schemas and establishing base image specifications.

Creates automated schema documentation generation with cross-reference support
Implements new base schemas for actions, data sources, and storage configurations
Adds RAS v6.6.0 container implementation with unsteady simulation support

Reviewed Changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
reference/ras/v660/schemas/action.ras.run_unsteady_simulation.json	New schema defining RAS unsteady simulation action with input/output validation
reference/ras/v660/run_unsteady.py	Main execution script for RAS unsteady simulations with S3/FS support
reference/ras/v660/parsers/unsteady_simulation_config.py	Configuration parser with attribute substitution and dataclass definitions
reference/json_to_md.py	Schema documentation generator with reference resolution and markdown output
reference/base/validate.py	Updated JSON schema validator supporting Draft 2020-12 specification
reference/base/schemas/*.json	Base schema definitions for actions, data sources, and storage

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-20T19:00:49Z

+  "description": "Schema for an unsteady HEC-RAS simulation action. Combines action metadata, attribute substitutions, input data sources, outputs, and store configuration.",
+  "allOf": [
+    {
+      "$ref": "./action.json#/$defs/Action"


The reference path './action.json#/$defs/Action' may not resolve correctly from this location. Based on the file structure, this should likely reference '../../base/schemas/action.json#/$defs/Action' to point to the base schemas directory.

Suggested change

"$ref": "./action.json#/$defs/Action"

"$ref": "../../base/schemas/action.json#/$defs/Action"

Copilot · 2025-09-20T19:00:50Z

+          "type": "array",
+          "description": "Input data sources required for the unsteady simulation.",
+          "items": {
+            "$ref": "./data.json#/$defs/DataSource"


The reference path './data.json#/$defs/DataSource' may not resolve correctly from this location. Based on the file structure, this should likely reference '../../base/schemas/data.json#/$defs/DataSource' to point to the base schemas directory.

Suggested change

"$ref": "./data.json#/$defs/DataSource"

"$ref": "../../base/schemas/data.json#/$defs/DataSource"

Copilot · 2025-09-20T19:00:50Z

+          "type": "array",
+          "description": "Output data sources produced by the unsteady simulation.",
+          "items": {
+            "$ref": "./data.json#/$defs/DataSource"


The reference path './data.json#/$defs/DataSource' may not resolve correctly from this location. Based on the file structure, this should likely reference '../../base/schemas/data.json#/$defs/DataSource' to point to the base schemas directory.

Copilot · 2025-09-20T19:00:50Z

+#         local_paths = output.create_local_output_paths(local_source)
+
+#         # Get the file types dynamically from the OutputPaths object
+#         file_types = [field.name for field in fields(output.paths) if getattr(output.paths, field.name)]
+
+#         # Process the files for upload
+#         try:
+#             for file_type in file_types:
+#                 local_path = local_paths[file_type]
+#                 if not local_path or not os.path.exists(local_path):
+#                     print(f"❌ Local file not found: {local_path}")
+#                     return False
+#                 s3_path = f"s3://{output.store_root}/{getattr(output.paths, file_type)}"
+#                 subprocess.call([UPLOAD, s3_path, "--path", local_path])
+
+#         except Exception as e:
+#             print(f"❌ Error uploading files for output '{output_name}': {e}")
+#             return False
+
+#     return True
+
+


Large block of commented code should be removed rather than left in the codebase. If this functionality might be needed later, consider using version control history or creating a separate branch.

Suggested change

# local_paths = output.create_local_output_paths(local_source)

# # Get the file types dynamically from the OutputPaths object

# file_types = [field.name for field in fields(output.paths) if getattr(output.paths, field.name)]

# # Process the files for upload

# try:

# for file_type in file_types:

# local_path = local_paths[file_type]

# if not local_path or not os.path.exists(local_path):

# print(f"❌ Local file not found: {local_path}")

# return False

# s3_path = f"s3://{output.store_root}/{getattr(output.paths, file_type)}"

# subprocess.call([UPLOAD, s3_path, "--path", local_path])

# except Exception as e:

# print(f"❌ Error uploading files for output '{output_name}': {e}")

# return False

# return True

Copilot · 2025-09-20T19:00:50Z

 #!/usr/bin/env python3
+"""
+JSON Schema Validator for RAS simulation schemas
+Supports JSON Schema Draft 2020-12 with latest jsonschema librarys


Spelling error: 'librarys' should be 'libraries'.

Suggested change

Supports JSON Schema Draft 2020-12 with latest jsonschema librarys

Supports JSON Schema Draft 2020-12 with latest jsonschema libraries

Copilot · 2025-09-20T19:00:51Z

+  "$id": "./action.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Action",
+  "description": "A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.",


Multiple spelling errors in the description: 'simulatuion' should be 'simulation' and 'attrubutes' should be 'attributes'.

Suggested change

"description": "A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.",

"description": "A generic action (e.g. model simulation, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attributes, and parameters.",

Copilot · 2025-09-20T19:00:51Z

+          "description": "Map of domain keys to in-file selectors (optional unless required by the specific action)."
+        },
+        "store": {
+          "$ref": "file:///app/schemas/store.json",


The absolute file path reference 'file:///app/schemas/store.json' is likely incorrect and will not resolve properly. This should use a relative reference like './store.json#/$defs/Store' to match the pattern used elsewhere.

Suggested change

"$ref": "file:///app/schemas/store.json",

"$ref": "./store.json#/$defs/Store",

Copilot · 2025-09-20T19:00:51Z

@@ -0,0 +1,25 @@
+# Action
+
+A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.


Multiple spelling errors in the description: 'simulatuion' should be 'simulation' and 'attrubutes' should be 'attributes'.

Suggested change

A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.

A generic action (e.g. model simulation, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attributes, and parameters.

Co-authored-by: Brendan Barnes <brendan.barnes856@gmail.com>

…cc-schemas

Add GitHub Actions workflows for building and testing FFRD images

…entation

lalekost and others added 16 commits August 25, 2025 10:43

Updated RAS V610 and V660

0efdb58

remove duplicate 610 code

e0b1f17

update config and add error handling to rasUnsteady

06fe2d2

add first pass schemas from cc manifests

4289cdf

add description to ras files

556ee79

update markdown files

5ff975e

Merge pull request #11 from fema-ffrd/ras-container

43caff5

Ras container

port cc schemas to base image for testing and eval

bddfd23

update ras 660 scripts for testing with ccschema

07a1316

add vscode to gitignore

139ef94

add env replacement for attrs in ras action

cf3259c

update ras schemas during testing

0176906

update model run routine using base image and new config

ed68cdc

move and consolidate reference schemas

1b91092

update ras test case for schema implementation

35ed7a3

update docs

5dbde81

slawler requested review from Copilot and thwllms September 20, 2025 18:59

Copilot AI reviewed Sep 20, 2025

View reviewed changes

slawler added 3 commits September 26, 2025 14:33

move parser to base image

6a41214

move parser to base image and call in ras image

f22456a

reformat (manually) md files

5315100

slawler changed the base branch from main to orchestration September 26, 2025 19:11

slawler requested a review from barne856 September 26, 2025 19:12

slawler added 4 commits September 30, 2025 14:56

update parser to remove snakecase

186fe6e

update ras configs

5782618

add example hms schema

c39e830

add descriptions to hms schema

5e37c1a

daniel-aragon-MBI changed the base branch from orchestration to main October 1, 2025 15:27

first step incorporating json validate into hms-go-runner

c945145

update entrypoint with json string input

2bb6300

barne856 reviewed Oct 2, 2025

View reviewed changes

Comment thread reference/ras/v660/entrypoint.sh Outdated

slawler and others added 16 commits October 2, 2025 11:35

Update reference/ras/v660/entrypoint.sh

5f3c974

Co-authored-by: Brendan Barnes <brendan.barnes856@gmail.com>

calibration passes, model starts, but datasource error

bdc322a

Merge branch 'cc-schemas' of https://github.com/fema-ffrd/specs into …

f9b0d2b

…cc-schemas

Merge branch 'main' into cc-schemas

1888a72

resolve go build issues after merging main

918a0e5

Add GitHub Actions workflows for building and testing FFRD images

3829d17

Remove old deploy-hms workflow

3fce0fe

fix test build

0fd9a72

use a single job to test builds

f7a452a

Add configuration for Docker Buildx setup

3d17910

Update base image tag to use 'ffrd_base:latest' in test build workflow

144f9e0

Merge pull request #15 from fema-ffrd/build-and-push-workflow

2ea8d61

Add GitHub Actions workflows for building and testing FFRD images

local FS example updates, components working for S3 store type implem…

1fc5165

…entation

working except for non-default output locations

8ed8dc3

fix markdown formatting

b4e5c1b

remove old hms runner test builds

f997568

thwllms approved these changes Oct 9, 2025

View reviewed changes

thwllms merged commit f3a213e into main Oct 9, 2025
7 checks passed

thwllms deleted the cc-schemas branch October 9, 2025 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cc schemas#12

Cc schemas#12
thwllms merged 41 commits intomainfrom
cc-schemas

slawler commented Sep 20, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Copilot AI Sep 20, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	"$ref": "./action.json#/$defs/Action"
	"$ref": "../../base/schemas/action.json#/$defs/Action"

	"$ref": "./data.json#/$defs/DataSource"
	"$ref": "../../base/schemas/data.json#/$defs/DataSource"

	Supports JSON Schema Draft 2020-12 with latest jsonschema librarys
	Supports JSON Schema Draft 2020-12 with latest jsonschema libraries

	"description": "A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.",
	"description": "A generic action (e.g. model simulation, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attributes, and parameters.",

	"$ref": "file:///app/schemas/store.json",
	"$ref": "./store.json#/$defs/Store",

		@@ -0,0 +1,25 @@
		# Action

		A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.

	A generic action (e.g. model simulatuion, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attrubutes, and parameters.
	A generic action (e.g. model simulation, data or transformation step) with typed inputs/outputs, parameterized paths, and optional operator parameters. See specific action schemas for specialized actions, attributes, and parameters.

Conversation

slawler commented Sep 20, 2025

Summary

Key Features Added

Automated Schema Documentation Generation

Schema Structure Improvements

Documentation Enhancements

Infrastructure Updates

Schema Documentation Links

Files Changed

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants