We should explore five top-level entries in schema

[sthagen]

The contributed source, provenance, and use entries as the only three top level entries led to inclusion of properties in one of the three, that in fact describe the whole set.

Thus, we should explore adding a set entry as fourth top level to move such properties in there and also to add more workflow and distribution related data on the whole metadata set in there too.

Last but not least in the JSON Schema we will want to enforce documenting the claimed schema the instances will validate against (or not) by adding the $schema property as top-level number five.

I would expect instances with stable insert order to present the five entries in the following order:

1. $schema
2. set
3. source
4. provenance
5. use

[davaya]

The $schema property is part of JSON Schema Core, it is a metaschema ID that specifies the JSON Schema version that our schema is written in. Valid values are listed on the JSON Schema website, presumably we'd use the latest: https://json-schema.org/draft/2020-12/schema. Note that although a JSON Schema file is a JSON file, the version is an identifier (schema) not a filename (schema.json).

$schema is not a property of the root type so it can't be included in the set of root properties.

A provenance metadata value (may/should/must?) contain a property identifying the DPS schema version; that would be a copy of the $id value in the DPS schema. Like the JSON Schema version, the $id is an identifier, not a filename:

"$id": "https://docs.oasis-open.org/dps/meta/v1.0/schema"

is a unique version identifier; the filename is superfluous:

"$id": "https://docs.oasis-open.org/dps/meta/v1.0/schema/data-provenance.json"

---

As for stable order, a map is unordered so if the messages are maps (object literals) an array will be needed somewhere:

{
  "data-provenance": [
    "https://docs.oasis-open.org/dps/meta/v1.0/schema",
    {
      "set": {},
      "source": {},
      "provenance": {},
      "use": {}
    }
  ]
}

or if all the properties must be both named and ordered:

{
  "data-provenance": [
    {"dps-version": "https://docs.oasis-open.org/dps/meta/v1.0/schema"},
    {"set": {}},
    {"source": {}},
    {"provenance": {}},
    {"use": {}}
  ]
}

[sthagen]

Leaving the $schema discussion for another day, my expectation was on instances with stable insert order (from host languages like Python, JS, or ES, where dicts/objects have stable insert order, always in PYthon and in JS/ES when it comes to string keys) - and string keys we exclusively use - to only name some order of description not to suggest every implementation shall provide that in the field. I think runtimes should, but we may give every language a few more decades for that 🙈 ...

David, your suggestions on partial or total ordering for the top-level are valid, of course, although I think no one would in practice want to wrap the top-level into another layer with a single key. I remember we had to do that when browsers confused JSON with JS, I mean these {"d": ... } wrappers ...

I know that languages like C++ and Java still have not found a way to provide competitive map implementations with insert order and we were all told the tale for decades, that for a map we have to give up ordering (not talking about dynamically sorted containers).

So to me descriptions of instances would look like this (top-level) in YAML:

$schema: ...
set: ...
source: ...
provenance: ...
use: ...

or in JSON:

{
  "$schema": "value",
  "set": "values",
  "source": "values",
  "provenance": "values",
  "use": "values"
}

or in XML (with a local schema for simplicity and kebab-style as a dash is welcome in XML names, as long as it is not the first character):

<data-provenance xsi:noNamespaceSchemaLocation="data-provenance.xsd" <!-- the $schema, kind of -->
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <set>...</set>
  <source>...</source>
  <provenance>...</provenance>
  <use>...</use>
</data-provenance>

when we describe them, or in the definition of the schema, but not necessarily when produced or consumed. There the order may be garbled depending on the host languages and the (de-)serialization mechanisms chosen.

[davaya]

I view Python's decision to preserve insertion order as a betrayal of defensive programming in the name of convenience :-) Back in the old days we could rely on the random hash seed to change ordering, as sets still do today, thereby exercising code paths and uncovering bugs that might not get coverage today without explicit randomizing test suites. But even so, JSON is a data interchange language that makes no guarantees about property order, leaving anything in the communication path, including cloud document databases, free to rearrange without violating the JSON spec.

Ehh, but Python is what it is, humans aren't going to be looking directly at DPS data, and machines will find the values wherever they are in the file. Stability only matters for streaming.

For YAML the array isn't onerous to look at:

data-provenance:
 - $schema: ...
 - set: ...
 - source: ...
 - provenance: ...
 - use: ...

and has the side benefit of having a root name - though that is another discussion for another day.