feat: Add support for !ignore tag and !reference/!reference-all scalar shorthand (spec v0.2.8-1)

README.md

@@ -1,6 +1,6 @@
 # yaml-reference
 
-Using `ruamel.yaml`, support cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, and `!merge`.
+Using `ruamel.yaml`, yaml-reference supports cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, `!merge`, and `!ignore`.
 
 Install the package from PyPI with:
 
@@ -14,25 +14,23 @@ uv add yaml-reference
 ```
 
 ## Spec
-![Spec Status](https://img.shields.io/badge/spec%20v0.2.6--4-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.6-4)
+![Spec Status](https://img.shields.io/badge/spec%20v0.2.8--1-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.8-1)
 
-This Python library implements the YAML specification for cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, and `!merge` as defined in the [yaml-reference-specs project](https://github.com/dsillman2000/yaml-reference-specs).
+This Python library implements the YAML specification for cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, `!merge`, and `!ignore` as defined in the [yaml-reference-specs project](https://github.com/dsillman2000/yaml-reference-specs).
 
 ## Example
 
 ```yaml
 # root.yaml
 version: "3.1"
 services:
-  - !reference
-    path: "services/website.yaml"
+  - !reference "services/website.yaml"
 
   - !reference
     path: "services/database.yaml"
 
 networkConfigs:
-  !reference-all
-  glob: "networks/*.yaml"
+  !reference-all "networks/*.yaml"
 
 tags: !flatten
   - !reference { path: "common/tags.yaml" }
@@ -43,6 +41,14 @@ config: !merge
   - !reference { path: "config/defaults.yaml" }
   - !reference { path: "config/overrides.yaml" }
 
+.anchors: !ignore
+  commonTags: &commonTags
+    - common:http
+    - common:security
+  dbDefaults: &dbDefaults
+    host: localhost
+    port: 5432
+
 ```
 
 Supposing there are `services/website.yaml` and `services/database.yaml` files in the same directory as `root.yaml`, and a `networks` directory with YAML files, the above will be expanded to account for the referenced files with the following Python code:
@@ -73,6 +79,48 @@ print(data["networkConfigs"])
 data = parse_yaml_with_references("root.yaml", allow_paths=["/allowed/path"])
 ```
 
+For `!reference` and `!reference-all`, both mapping and scalar shorthand forms are supported. These are equivalent:
+
+```yaml
+# Scalar shorthand
+service: !reference "services/api.yaml"
+
+# Mapping form
+service: !reference { path: "services/api.yaml" }
+
+# Scalar shorthand
+networks: !reference-all "networks/*.yaml"
+
+# Mapping form
+networks: !reference-all { glob: "networks/*.yaml" }
+```
+
+Use the mapping form when you need optional arguments such as `anchor`; use the scalar shorthand when you only need `path` or `glob`.
+
+### The `!ignore` Tag
+
+The `!ignore` tag marks YAML content that should be parsed but omitted from the final resolved output. The most common use case is a hidden section of reusable anchors that should remain available for aliases elsewhere in the document without being emitted in the resolved result.
+
+```yaml
+.anchors: !ignore
+  commonLabels: &commonLabels
+    app: payments
+    team: platform
+  defaultResources: &defaultResources
+    requests:
+      cpu: "100m"
+      memory: "128Mi"
+
+service:
+  metadata:
+    labels: *commonLabels
+  resources: *defaultResources
+```
+
+When loaded with `load_yaml_with_references`, the `.anchors` key is removed entirely, but the anchors it defined remain usable by aliases elsewhere in the document.
+
+Ignored items are also pruned before `!flatten` and `!merge` are evaluated, so an ignored sequence entry inside either tag is simply omitted from the flattened or merged result.
+
 ### The `!merge` Tag
 
 The `!merge` tag combines multiple YAML mappings (dictionaries) into a single mapping. This is useful for composing configuration from multiple sources or applying overrides. When you use `!merge`, you provide a sequence of mappings that will be merged together, with later mappings overriding keys from earlier ones.
@@ -151,20 +199,25 @@ Note that the `app_name` and `cache_settings` fields from `config.yaml` are not
 
 ### VSCode squigglies
 
-To get rid of red squigglies in VSCode when using the `!reference`, `!reference-all`, `!flatten`, and `!merge` tags, you can add the following to your `settings.json` file:
+To get rid of red squigglies in VSCode when using the `!reference`, `!reference-all`, `!flatten`, `!merge`, and `!ignore` tags, you can add the following to your `settings.json` file:
 
 ```json
     "yaml.customTags": [
+        "!reference scalar",
         "!reference mapping",
+        "!reference-all scalar",
         "!reference-all mapping",
         "!flatten sequence",
-        "!merge sequence"
+        "!merge sequence",
+        "!ignore scalar",
+        "!ignore sequence",
+        "!ignore mapping"
     ]
 ```
 
 ## CLI interface
 
-There is a CLI interface for this package which can be used to read a YAML file which contains `!reference` tags and dump its contents as pretty-printed JSON with references expanded. This is useful for generating a single file for deployment or other purposes. Note that the keys of mappings will be sorted alphabetically. This CLI interface is used to test the contract of this package against the `yaml-reference-specs` project.
+There is a CLI interface for this package which can be used to read a YAML file which contains composition tags such as `!reference`, `!reference-all`, `!flatten`, `!merge`, and `!ignore`, and dump its contents as pretty-printed JSON with references expanded and ignored content removed. This is useful for generating a single file for deployment or other purposes. Note that the keys of mappings will be sorted alphabetically. This CLI interface is used to test the contract of this package against the `yaml-reference-specs` project.
 
 ```bash
 $ yaml-reference-cli -h

tests/unit/test_flatten.py

@@ -197,6 +197,21 @@ def test_flatten_with_scalars(stage_files):
     assert data["data"] == [1, 2, 3, 4, 5, 6, 7]
 
 
+def test_flatten_ignores_ignored_sequence_items(stage_files):
+    """Test that !ignore items inside a !flatten sequence are omitted from the flattened result."""
+    files = {
+        "test.yml": """
+data: !flatten
+  - [1, 2]
+  - !ignore [99, 100]
+  - [3, 4]
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data["data"] == [1, 2, 3, 4]
+
+
 def test_flatten_mixed_objects_references(stage_files):
     """Test flattening a sequence of objects, references, and reference-all tags."""
     files = {

tests/unit/test_ignore.py

@@ -0,0 +1,184 @@
+from yaml_reference import (
+    Ignore,
+    load_yaml_with_references,
+    parse_yaml_with_references,
+    prune_ignores,
+)
+
+
+def test_ignore_parse_produces_ignore_object(stage_files):
+    """Test that !ignore tags are parsed into Ignore objects by parse_yaml_with_references."""
+    files = {
+        "test.yml": "key: !ignore some_value",
+    }
+    stg = stage_files(files)
+    data = parse_yaml_with_references(stg / "test.yml")
+    assert isinstance(data["key"], Ignore)
+
+
+def test_ignore_dict_value_removed(stage_files):
+    """Test that a dict value tagged with !ignore is removed from the output."""
+    files = {
+        "test.yml": """\
+keep: hello
+drop: !ignore world
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert "keep" in data
+    assert data["keep"] == "hello"
+    assert "drop" not in data
+
+
+def test_ignore_list_item_removed(stage_files):
+    """Test that a list item tagged with !ignore is removed from the output."""
+    files = {
+        "test.yml": """\
+items:
+  - one
+  - !ignore two
+  - three
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data["items"] == ["one", "three"]
+
+
+def test_ignore_standalone_value_becomes_none(stage_files):
+    """Test that a standalone (non-list, non-dict) !ignore value is replaced with None."""
+    files = {
+        "test.yml": "!ignore standalone",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data is None
+
+
+def test_ignore_multiple_items_in_list(stage_files):
+    """Test that multiple !ignore items in a list are all removed."""
+    files = {
+        "test.yml": """\
+items:
+  - !ignore a
+  - b
+  - !ignore c
+  - d
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data["items"] == ["b", "d"]
+
+
+def test_ignore_multiple_keys_in_dict(stage_files):
+    """Test that multiple !ignore values in a dict are all removed."""
+    files = {
+        "test.yml": """\
+a: !ignore 1
+b: 2
+c: !ignore 3
+d: 4
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data == {"b": 2, "d": 4}
+
+
+def test_ignore_mapping_value(stage_files):
+    """Test that a mapping tagged with !ignore is removed."""
+    files = {
+        "test.yml": """\
+keep:
+  x: 1
+drop: !ignore
+  y: 2
+  z: 3
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert "keep" in data
+    assert "drop" not in data
+
+
+def test_ignore_sequence_value(stage_files):
+    """Test that a sequence tagged with !ignore is removed when used as a dict value."""
+    files = {
+        "test.yml": """\
+keep: [1, 2]
+drop: !ignore [3, 4]
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data == {"keep": [1, 2]}
+
+
+def test_ignore_does_not_affect_non_tagged_content(stage_files):
+    """Test that !ignore only affects tagged content and leaves everything else intact."""
+    files = {
+        "test.yml": """\
+name: yaml-reference
+version: 1.0
+description: !ignore internal note
+tags:
+  - alpha
+  - !ignore beta
+  - gamma
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data["name"] == "yaml-reference"
+    assert data["version"] == 1.0
+    assert "description" not in data
+    assert data["tags"] == ["alpha", "gamma"]
+
+
+def test_ignore_in_referenced_file(stage_files):
+    """Test that !ignore tags in a referenced file are pruned during resolution."""
+    files = {
+        "root.yml": "contents: !reference { path: ./inner.yml }",
+        "inner.yml": """\
+public: visible
+private: !ignore hidden
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "root.yml")
+    assert data["contents"] == {"public": "visible"}
+    assert "private" not in data["contents"]
+
+
+def test_prune_ignores_standalone(stage_files):
+    """Test prune_ignores() directly on a structure containing Ignore objects."""
+    data = {
+        "a": Ignore("should be removed"),
+        "b": 42,
+        "c": [1, Ignore("also removed"), 3],
+    }
+    result = prune_ignores(data)
+    assert "a" not in result
+    assert result["b"] == 42
+    assert result["c"] == [1, 3]
+
+
+def test_ignore_preserves_none_values(stage_files):
+    """Test that existing null/None values in YAML are preserved (not confused with ignored values)."""
+    files = {
+        "test.yml": """\
+present: ~
+also_present: null
+dropped: !ignore something
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert "present" in data
+    assert data["present"] is None
+    assert "also_present" in data
+    assert data["also_present"] is None
+    assert "dropped" not in data

tests/unit/test_merge.py

@@ -124,6 +124,21 @@ def test_merge_nested(stage_files):
     assert data["result"] == {"a": 1, "b": 2, "inner": {"x": 2, "y": 1}}
 
 
+def test_merge_ignores_ignored_sequence_items(stage_files):
+    """Test that !ignore items inside a !merge sequence are omitted before merging."""
+    files = {
+        "test.yml": """
+result: !merge
+  - {a: 1}
+  - !ignore {ignored: true}
+  - {b: 2}
+""",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data["result"] == {"a": 1, "b": 2}
+
+
 def test_flatten_and_merge(stage_files):
     """Test flattening and merging together."""