Introduce a unique ID/KEY to all OSCAL objects

[juano2310]

User Story

As a developer or integrator working with OSCAL, I want every object in the OSCAL models to include a unique ID (if one is not already present), so that objects across catalogs, profiles, and assessments can be uniquely and consistently identified.

Goals

🔐 Uniqueness Across Instances

• A unique ID ensures that each object has a globally unique identifier.
• This prevents collisions when merging data from multiple OSCAL sources.

🔗 Traceability & Referencing

• Enables other OSCAL artifacts (e.g., profiles, assessments) to reliably reference specific elements.

🔄 Interoperability

• Many tools integrate OSCAL models. Unique IDs help ensure consistent object identification across systems and vendors.

⚙️ Automation & Linking

• Essential for automated tooling, where object linking cannot rely on human-readable names (which may change).
• Supports the development of machine-readable policies and automated compliance assessments.

Dependencies

N/A

Acceptance Criteria

• All objects that do not currently define a unique ID are reviewed.
• A unique ID is introduced where appropriate, ensuring backward compatibility.
• Documentation is updated to reflect the addition and usage of unique IDs.
• All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)

Revisions

No response

[brian-ruf]

INTRODUCTION

First, I want to expand on the rationale behind this request:
While XML has xPath that can augment ID/UUID values to unambiguously identify data objects, there is no single, mature standard for a JSON equivalence, and JSON is by far the most prevalent OSCAL format in use among industry tools.

Further, many tools are created using internet technologies, and tool developers are aided when when can use a simple object identifier to unambiguously identify a target data object. Unfortunately, simple ordinal values are not reliable in JSON. They could shift from one moment to the next. Especially due to concurrency where one user is adding objects while another user is querying those objects.

---

CHANGE BREAKDOWN

The ID/UUID discussion breaks down as follows:

1. Objects that already have a mandatory ID or UUID, where no change is needed.
2. Objects that already have an optional ID or UUID, where it is helpful for the identifier to always be present.
   a. Make all optional IDs and UUIDs mandatory; or
   b. Agree as a community to always assign as a best practice.
3. Objects that do not currently have an ID or UUID, where the presence of aUUID would be helpful.
   a. Adding UUID as mandatory; or
   b. Adding UUID as optional
4. Including UUID where there is already ID
   a. Adding as mandatory; or
   b. Adding as optional

Items 2a, 3a and 4a are Breaking Changes

Changing an optional ID/UUID attribute to mandatory or adding a mandatory UUID attribute where it did not previously exist are breaking changes. These changes can only be made in OSCAL 2.x

2b, 3b, and 4b are Non-Breaking Changes

While these are all non-breaking changes, they are not equal. As they progress, each represents a great impact along with greater benefits to tool developers.

Minimum Impact: 2b

A minimalist change in response to this issue is for the community to agree - as a best practice - to always assign an ID or UUID, even where this information is optional.

Moderate Impact: 3b

It is possible to add optional UUID attributes to additional objects without breaking current tools. This makes UUID available in more places. Similar to 2b, once UUID is available, industry could agree to always populate it as a best practice.

Significant Impact: 4b

The use of ID is helpful for canonical references, yet is limiting when the data object has no canonical significance. Further, it requires OSCAL tools to handle two variations of identifiers (ID and UUID) with different rules that prevent tools from simply using a UUID v4 or v5 value in an ID field. This adds complexity for adoption.

It is possible for ID and UUID to co-exist on the same data object. A non-breaking change would be to add an optional UUID attribute to all objects that currently have an ID. Consistent with 2b and 3b above, the community could then agree to always populate the UUID values as a best practice.

---

OSCAL 1.x Recommendation

• Add optional uuid attributes broadly
  • This includes to groups, controls, params, parts, roles
  • Add to other data objects based on a more granular community discussion
• Seek agreement among OSCAL stakeholders to always populate uuid attributes when present, regardless of their cardinality (mandatory or optional)

---

OSCAL 2.0 Recommendation

• Add mandatory uuid attributes broadly

  • including every place that has an id (groups, controls, params, parts, roles)
  • make all uuid attributes mandatory

• Keep id attributes where there is a likelihood of canonical references

  • All id attributes are optional
  • Every field/assembly with an id attribute also has a mandatory uuid
  • More specifically:
    • groups, controls and roles
      • keep id
      • convert id to optional
      • include a mandatory uuid
  • params:
    • Either drop id or convert id to optional (for backward compatibility)
  • parts:
    • Either drop id or keep as optional.

[xee5ch]

First, I want to expand on the rationale behind this request: While XML has xPath that can augment ID/UUID values to unambiguously identify data objects, there is no single, mature standard for a JSON equivalence, and JSON is by far the most prevalent OSCAL format in use among industry tools.

@brian-ruf, Have you reviewed JSONPath (RFC9535) and JSONPointer (RFC6901) in combination with one another? I assume your introduction is discussing "identity by location in document" use cases for the XPath spec, but that does exist in JSON with JSON Path. It was only recently standardized (in the IETF sense) about a year ago, so you are right for much of OSCAL's existence, but that change has been relatively recent.

[xee5ch]

User Story

As a developer or integrator working with OSCAL, I want every object in the OSCAL models to include a unique ID (if one is not already present), so that objects across catalogs, profiles, and assessments can be uniquely and consistently identified.

Goals

🔐 Uniqueness Across Instances

* A unique ID ensures that each object has a globally unique identifier.

* This prevents collisions when merging data from multiple OSCAL sources.

I look forward to seeing how this group receives this request, @juano2310. I had in my mind a similar, but perhaps slightly different, proposal that had been percolating in my brain so I'll ask a question: is any ID acceptable format acceptable (UUID4 versus UUID5 versus W3C DIDs versus ULID versus Snowflake IDs versus custom human readable IDs and many more)? Does it have to verifiably unique per document? In the theoretical address space of all documents?

[brian-ruf]

I'd suggest that it has to be verifiably different within any given "stack" of OSCAL documents, given the referential requirements from AR -> AP -> SSP -> Profile(s) -> Catalog(s).

While it doesn't have to be UUID v4 or v5, those have proven ideal as they ensure uniqueness without any form of ID registration or management system, which also leads to OSCAL's interoperability goal. Further they are very friendly in terms of use in a URL, API call, or database field.

As for a JSON equivalent of xPath, I'll assume it also works for YAML. Even with a JSON/YAML xPath equivalence, that is still a new syntax/skill and a layer of complication that represents a barrier to adoption. Being able to directly reference IDs is simple, an existing convention among software developers and significantly more performant than xPath processing.

The current usage of xPath typically requires 3rd party libraries or complex coding. Referencing an object by ID is much easier both for native OSCAL and especially for tool developers translating OSCAL content to/from a database that has not xPath equivalence.

XPath and similar have their place, and I personally enjoy using xPath in my software development. But the learning curve was steep and the complexity of code to support was a dramatic barrier.

[rsherwood-nist]

It is possible to use UUID in any ID field today by prepending "urn:uuid:" which will turn it into a URI and a valid token. This wouldn't require any change to the current specification, but obviously only applies where an ID field is defined.

[xee5ch]

It is possible to use UUID in any ID field today by prepending "urn:uuid:" which will turn it into a URI and a valid token. This wouldn't require any change to the current specification, but obviously only applies where an ID field is defined.

I get where you are going with this and I quite like it. From my perspective, I would like a change to the following.

1. Remove all occurrence of @uuid (and fields with the name uuid as the whole name or as a suffix like thing-uuid) to change to @id and thing-id respectively.
2. Explicitly convert all @id and thing-id occurrences to the URI type and leave the ambiguity out.
3. Opt for using the upstream urn:uuid, urn:example and other types once registered following RFC 8141 and using the official IANA registry. Thoughts?

And to be clear on my position: I am really only interested in future state OSCAL reenvisioned as 2.x or 3.x, something very different, I personally have little to no interest in near-term use of 1.x OSCAL models when I make these recommendations.

[xee5ch]

I'd suggest that it has to be verifiably different within any given "stack" of OSCAL documents, given the referential requirements from AR -> AP -> SSP -> Profile(s) -> Catalog(s).

So if I understand you correctly, you're saying it has to be at least unique for that document collection, but not at least across the world of all possible documents? Just trying to tease out your perspective here, @brian-ruf.

As for a JSON equivalent of xPath, I'll assume it also works for YAML. Even with a JSON/YAML xPath equivalence, that is still a new syntax/skill and a layer of complication that represents a barrier to adoption. Being able to directly reference IDs is simple, an existing convention among software developers and significantly more performant than xPath processing.

Sorry I wasn't suggesting that as the only or preferred way. I am just teasing out from your previous comment there is "unique ID for an object as given" and also "a way of identifying an object uniquely by path in one document or document collection" if we call that identity (re XPath). All I am saying is there an equivalent concept and implementation in JSON/YAML (you're right it can support both, but I believe that is library implementation specific). I do agree universally unique IDs for lookups, cached or not, are preferred. This is how the Metaschema software, the NIST and Metaschema Framework community fork, both process repeat data access efficiently.

[rsherwood-nist]

Referencing an object by ID is much easier both for native OSCAL and especially for tool developers translating OSCAL content to/from a database that has not xPath equivalence.

It's true that dereferencing and identifier is more efficient and easier than something like XPath, but I want to second @xee5ch's point that unique ID references are not a substitute for something like MetaPath. A unique ID is an efficient way to find an object if you already know where it is. More often, however, you need to express something like "Every object of type Foo with property Bar". In this case, you don't know where the object is already and you need to express the set of attributes that describe the object(s) you want.

MetaPath/Xpath/JsonPath exist for this reason, and are as complicated as they are because of what they need to do. In the same way, SQL is more complicated and difficult than just keeping a list of the ID's for the database rows you already know you're interested in. They're both necessary however. Even if this proposal is accepted and implemented, anyone processing an OSCAL document will either need MetaPath, or something else just as complicated.

[rsherwood-nist]

While it doesn't have to be UUID v4 or v5, those have proven ideal as they ensure uniqueness without any form of ID registration or management system, which also leads to OSCAL's interoperability goal.

If we move forward with this proposal, and if I understand the intent, I think UUIDv4 and/or UUIDv5 should be a mandatory feature to avoid accidental name collision - currently the specification requires ID uniqueness, but this must be enforced are runtime by the validator.

Application developers must not rely on the uniqueness properties of UUIDs however, since they only address accidental collision, not malicious or stupid collision :)

[rsherwood-nist]

I want to add a point to this discussion that I learned from the OSCAL devs early in my journey - I was trying to migrate from a bespoke database schema to implement a system that would use OSCAL as its native backing storage format. The OSCAL team counseled against this - and I learned that they were correct.

OSCAL is well designed for its purpose - a neutral interchange language for data exchange between systems. It is not designed as a data schema for storage. The lack of universal ID's to reference every sub-element reflects this early design decision. If we want OSCAL to be a database schema, we'll need a lot more changes. In addition, there's a good reason why data interchange languages have been developed, rather than everyone just exchanging database backups - the requirements for exchange vs storage are quite different.

[rsherwood-nist]

One more.

Any discussion of an identifier format also has to address the question of identifier lifecycle. In a DB schema, the db Key points to a row in a table regardless of its content or semantics, and is guaranteed to be unique in perpetuity. The row reference does not change if the underlying data is changed. This is a good decision for the problem domain that the DB key was designed for.

The UUID in the current OSCAL format is not designed that way. To take the example of a Catalog UUID:

Every time the content of an OSCAL file changes, the following must also change:
A new UUID value must be generated and assigned to the root element's uuid.

This reflects the idea that the UUID is not a reference to a "row in a database" but a reference to a structured representation of a specific set of content.

If you want to use UUID as a database key, it shouldn't change, but on the other hand, from the perspective of the content, the UUID may be referencing something completely different. This may be fine, but should be explicitly included in the proposal.

The distinction between "pointer to a piece of content" and "pointer to a document location" is lost when we propose adding a UUID as a reference without discussing the relationship between the lifecycle of the identifier and content or location it identifies.

[rsherwood-nist]

1. Remove all occurrence of @uuid (and fields with the name uuid as the whole name or as a suffix like thing-uuid) to change to @id and thing-id respectively.

Would you lose the distinction between human readable and systemic identifiers in this case? @id is usually intended for humans, right? If we're talking about the distant future, can we propose something like a @reference tag which is only for systemic reference to a specific piece of content as a replacement for @uuid, it must be a URN, and may be a UUID?

We'll still need a clear statement about the lifecycle if the @reference and the thing it references!