[WIP] Schema v2: proposal

[lmolkova]

Part of open-telemetry/opentelemetry-specification#4427

Harmonize identity property names

Principles:

• same as OTLP property when possible/makes sense
• default to name

signal/object	current id semconv	proposal	in proto
attributes	id (unresolved schema) and name (resolved)	key	key
metrics	metric_name	name	name
events	name	name	event_name, no name for log records
spans	-	type	will propose type, spans already have name, kind and spanId
entity	name	type	type
profiles	-	-	-
scopes	-	-	name

Have dedicated types for different objects/signals

attributes: # instead of `registry.*` attribute groups
- key: foo.name
  ...
- key: foo.target
metrics:  # instead of group.type: metric
- name: foo.duration
entities:
- type: foo
events:
- name: foo.ended
spans:
- type: foo.client.request # new
  kind: client
  name: {foo.name} {foo.target}  # new

References have different types as well

(referencing is only available in unresolved schema)

metric_refs:
- id: my_flavor.foo.duration
  extends: foo.duration # or fully-qualified otel.foo.duration, namespace delimiter tbd
  attributes:
  - ref: azure.resource.id 
    brief: ...

each _ref instance has to be identifiable. There could be multiple favors of the same metric that have the same name, but different ids.

Imports

no changes, works naturally and looks consistent

imports:
- attributes: foo.*
- spans: foo.*

Stages

1. weaver supports new unresolved schema format
   • unresolved schema is still pretty much contained within semconv, easy to change
   • weaver should probably support old schema for the time being and convert it to the same internal representation
     --- we're here on 8/18/25 --
2. new resolution format is produced behind feature flag - resolved schema is extensively used by codegen
   • need to socialize among maintainers
   • jq helpers used by codegen produce something close to the proposal (group by attributes/metrics/etc), so we can probably keep back-compat
3. default to the new format

[jsuereth]

@lmolkova (and others) When you get a chance can you look at: jsuereth@192d539

This defines signals (not attributes) with backwards conversions for Weaver, with tests.

• I assume all signals reference (or refine) attributes and do not allow defining an attribute, e.g. in the metrics section.
• I have added some aspects of the data model as required in V2, e.g. stability, or instrument on metric.
• I have not wired the reading of this new format all the way through

PTAL - I think this is inline with the direction, but there's some details I'd like to iron out quickly.

[lquerel]

@lmolkova @jsuereth Should we also take this opportunity to introduce (or at least define) the concept of namespace as an object in the model?

Another question: what can we reference in attributes? Attribute groups, attributes, or a mix of both? In the current version of the resolved schema, the attribute catalog is a collection of attributes (not groups), each with an internal identifier computed by Weaver (a number but could be something else if needed). These identifiers are used to reference the attributes from the signals. This is done because the initial attribute identifiers/names cannot be used due to the multiple overloaded versions that can exist after resolution.

[lquerel]

@lmolkova @jsuereth Okay, after reading carefully, I think that only the attributes indexed by a key are referenced, which matches the current organization of the catalog. However, as I mentioned, this key cannot be the id or name of the attribute because of the reference mechanism with overloading.

[lmolkova]

thanks @jsuereth ! left some comments, but it looks awesome! It pretty much covers the v2 of the current resolved schema along with back-compat.

The substantial tl;dr of my comments is whether we can avoid adding legacy (tags, span events, prefix) into v2.

There are some things we'll need to work through for the unresolved schema. E.g.

1. we currently use attribute groups to avoid copy-pasting (e.g. multiple metrics share the same set of attributes with slight variations/additions) - I feel we have to come up with some way.
2. To @lquerel's point - it would be useful to assign (generate if not present?) unique id to all refs to make lineage straightforward. Not sure if we want to reference an attribute ref, it depends on how we decide to do p1.

Another big thing from @lquerel's point that's applicable to resolved schema - what would we do for multiple references of the same metric/event/span - e.g. there is a generic database span and a postgres flavor of it.

I'd imagine postgres flavor is defined as a SpanRef, which has a unique identifier so I can render it in the docs and generate postgres specific code. I.e. I should be able to see both in the resolved schema. We support it with unique group ids + extends today.

[lmolkova]

After #829 (thanks to @jsuereth) , we can define everything except attribute groups using v2. Attribute groups are useful in two cases:

1. avoiding copy-paste: e.g. we define attributes.http.client which are then used on spans and metrics
2. declaring a group of attributes that should be documented/code-generated together. Examples: thread, code, enduser, session. E.g. generated function signature might look like
   annotate_with_code_attributes(T signal). These groups may be a subset of attributes in a namespaces or contain attributes from different namespaces (e.g. error and exception).

How can we support this in v2:

Option A. Re-introduce attribute-groups:

attributes:                                     # attributes definition in server.yaml - nothing new 
- key: server.address
  # ...

attribute_groups:                               # attribute group in http_common.yaml - this is new
- id: http.client.attributes
  brief: common http attributes
  visibility: private                           # default. Very few attribute groups need to be visible in resolved schema. For public groups we'll need to establish stability guarantess.
  attributes:
  - ref: server.address 
    brief: the name of the HTTP server as captured in the Host header

spans:
- type: http.client
  ...
  include_groups: [ http.client.attributes ]     # this is new - just a list of group ids to include
  attributes:
    - ref: server.address                        # it's possible to refine attributes included via groups
      sampling_relevant: true 

metrics:
- name: http.client.operation.duration
  include_groups: [ http.client.attributes ]     # groups are reused on multiple signal definitions
  ...

This option solves scenarios 1 & 2.

Option B. Named attribute references

attribute_refs:                                 # this is new
- ref: server.address
  ref_id: http.server.address
  brief: the name of the HTTP server as captured in the Host header
  # ...

spans:
- type: http.client
  ...
  attributes:
    - ref: server.address
       ref_id: http.server.address 
       sampling_relevant: true 

This option solves scenario 1 only. It'd also mean including ref_id on all HTTP signals which would be easy to miss causing inconsistencies.
We can partially solve scenario 2 assuming group boundary is the same as the namespace boundary.

Option C: Const definitions

constants:
- http_server_address_brief: the name of the HTTP server as captured in the Host header

spans:
- type: http.client
  ...
  attributes:
    - ref: server.address
      brief: ${http_server_address_brief}
      sampling_relevant: true 

This option partially scenario 1 only. It'd also mean having ${http_server_address_note}, ${http_server_address_examples} and repeating them on each signal. This is repetitive and error prone. This feature might be interesting orthogonally to attribute groups though.

---

So my preference is on some form of the the option A.

[lmolkova]

Alternative syntax for Option 1

attribute_groups:
- id: http.client.attributes
  brief: common http attributes
  visibility: private
  attributes:
  - ref: server.address 
    brief: the name of the HTTP server as captured in the Host header

spans:
- type: http.client
  ...
  attributes:
    - group:  http.client.attributes
    - group:  ...
    - ref: server.address                        # it's possible to refine attributes included via groups
      sampling_relevant: true 

[thompson-tomo]

I also Like option A which is conceptually the same as what i proposed in #871. Note attribute groups currently enable inheritance as well so it would be good to know what the plan there is.

[lmolkova]

The proposal is to leverage composition instead of extension. A signal can mix and match multiple attribute groups which are not aware of each other.

I.e. instead of writing

  - id: attributes.http.common
    attributes:
      - ref: http.request.method
      - ref: http.response.status_code
      - ref: error.type
      - ref: network.protocol.name
      - ref: network.protocol.version

  - id: attributes.http.client
    extends: attributes.http.common
    attributes:
      - ref: server.address
      - ref: server.port
      - ref: url.scheme
      - ref: url.template

  - id: span.http.client
    extends: attributes.http.client
    attributes:
    ...

I'd write

attribute_groups:
  - id: attributes.http.common
    attributes:
      - ref: http.request.method
      - ref: http.response.status_code
      ...

  - id: attributes.http.client.authority
    attributes:
      - ref: server.address
      - ref: server.port

   - id: attributes.http.client.url
     attributes:
      - ref: url.scheme
      - ref: url.template

spans:
- type: http.client
  attributes:
  - group: attributes.http.common
  - group: attributes.http.client.authority
  - group: attributes.http.client.url
  - key: ....
  ...

It's an interesting question if an attribute group should be able to include another attribute group. I would start without it and only add it if it proven to be useful. It should be possible to rewrite HTTP or DB conventions without it in a reasonable way.

Another interesting question is conflict resolution. With extension there are at most two sources of attribute - the parent and the group itself and the latter always wins. With composition, I believe we should disallow conflicts.

I.e. we should fail with error on

attribute_groups:
- id: group.a
   attributes:
   - ref: foo.bar
     brief: brief1

- id: group.b
   attributes:
   - ref: foo.bar
     brief: brief2

span:
- type: foo.client
  ...
  attributes:
  - group: group.a
  - group: group.b

since they include the same attribute and it's not clear which one should be resolved.

This would enforce us to define modular and independent groups and make lineage clear.
We'd need a weaver prototype to validate it's sufficient for HTTP and DB conventions (DB are the most hairy ones when it comes to groups).

[thompson-tomo]

I agree with you that groups should not be able include other groups especially as by allowing signals to have multiple groups significantly reduces the need to support for it.

What I do however think is that we should still retain that an attribute can be explicitly overwritten at the signal level. A scenario would be defining a server attribute group and then wanting to update the brief on the database span to mention database.

[lmolkova]

Have a couple of hacky prototypes for groups v2: weaver, semconv.

A few of points:

Sampling-relevant works better as a general property. We can validate and fail if someone uses it on metrics/events/entities.

Separating sampling-relevant attributes is not helping. E.g. I have ~10 DB span definitions that share attributes, I can't declare a common group and define attrs as sampling-relevant in the common place, I have to copy-paste sampling-relevant portion only, which makes is error-prone. So I removed differentiation in my weaver hack.
I imaging there could be other solutions to this problem.

Internal, impl-detail attribute groups

I ended up hacking the following:

attribute_groups:
  - id: attributes.db.client.authority
    attributes:
      - ref: server.address
        brief: >
          Name of the database host.
      - ref: server.port
        requirement_level:
          conditionally_required: If using a port other than the default port for this DBMS and if `server.address` is set.
...
spans:
  - type: db.client
    ...
    include_groups: [ attributes.db.client.authority, ...]

Any telemetry item or group can include other groups. No conflicts within included groups are allowed.

I tried it on HTTP and DB semconv - updated them to v2 and 1) got the same MD 2) found things that are probably bugs coming from confusing extends and deep hierarchies 3) shaved a bit of yaml off, but not much. So it seems to be a positive change.

External, documentable/importable attribute groups

No prototype.
This is the key point I think we need to discuss/decide.

Are public-facing groups and internal groups the same thing?

The answer is NO.

On the schema/weaver side, they are different: internal ones don't have stability guarantees, brief, notes, annotations, deprecation. They are an id with a list of attribute-refs. External do have it all (CommonFields).

Is this difference big enough to justify two different concepts?

The problem might be tiny. We could have a visibility property and checks that internal don't have any public-facing properties and public ones do.

My slight and subjective preference is to have them as separate types. We've barely started scratching the surface of public-facing-attr-groups, and the problem space might be big.

If they are different, how do we call them?

Ideas:

• private: mixin | internal_attribute_group
• public attribute_group | semantic_attribute_group, public_attribute_group

I'd go with mixin (internal) and attribute_group (public) unless there are alternative suggestions.