Update Entity Fields RFC for 9.2 Stack Release

[MikePaquette]

1. What does this PR do?

Adds newly proposed entity.* fields to the Entity Fields RFC
Adds currently used entity.* fields to this RFC
Proposes a nested location in the naming hierarchy for entity.* fields populated by ECS producers
Proposes a root namespace location for entity.* fields when used by ECS consumers and entity-related data stores.

2. Which ECS fields are affected/introduced?

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[eyalkraft]

Hi Mike, Thanks for the proposal!

Is there a reason for having the new entity.schema_version instead of using the standard ecs.version?

[MikePaquette]

Is there a reason for having the new entity.schema_version instead of using the standard ecs.version?

Good question. I don't know when this was added, or how it is used, but I found it in the existing mappings in 8.19.1.

"entity": {
          "properties": {
            "definition_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "definition_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "display_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "identity_fields": {
              "type": "keyword"
            },
            "last_seen_timestamp": {
              "type": "date"
            },
            "name": {
              "type": "keyword"
            },
            "schema_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "source": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },

[mjwolf]

entity.definition_id is repeated here

[mjwolf]

I'm not too sure about this field being reserved. There are no other fields in ECS that are reserved for use by Elastic, and I don't really know if it makes sense to have them. Since the intention of ECS is a common schema that's shared with others, I don't if this would make sense to include.

Are there alternatives to adding this, such as using a custom field that's not defined in ECS?

[MikePaquette]

Yes, I agree, it's an internal-only field - no need to have it defined in the ECS spec. We already intend to publish the entity store schema in our docs. Any problem with using entity.Definition_id as a custom field? (nesting a custom leaf field under an ECS-defined root object entity.*

[romulets]

@MikePaquette as we speak we are storing this information under entity.Metadata.EngineType. What do you think about keeping it under metadata?

[mjwolf]

Suggested change

| entity.attributes.* | object | A set of static or semi-static attributes of the entity. Usually boolean or keyword field data types. Examples include: `entity.attributes.Storage_class`, `entity.attributes.Mfa_enabled` , `entity.attributes.Privileged` , `entity.attributes.Granted_permissions` , `entity.attributes.Known_redirect` , `entity.attributes.Asset` , `entity.attributes.Managed` ,`entity.attrbitues.Os_current` , `entity.attibutes.Os_patch_current` , `entity.attributes.Oauth_consent_restriction`). Use this field set when you need to track static or semi-static characterstics of an entity for advanced searching and correlation of normalized values across different providers/sources and entity types. Note the initial capitalization pattern for Examples indicates that any such fields are custom entity-specific fields that won't be enumerated in the ECS schema, and won't collide with any fields that may be defined by ECS in the future. |

| entity.attributes | object | A set of static or semi-static attributes of the entity. Usually boolean or keyword field data types. Examples include: `entity.attributes.Storage_class`, `entity.attributes.Mfa_enabled` , `entity.attributes.Privileged` , `entity.attributes.Granted_permissions` , `entity.attributes.Known_redirect` , `entity.attributes.Asset` , `entity.attributes.Managed` ,`entity.attrbitues.Os_current` , `entity.attibutes.Os_patch_current` , `entity.attributes.Oauth_consent_restriction`). Use this field set when you need to track static or semi-static characterstics of an entity for advanced searching and correlation of normalized values across different providers/sources and entity types. Note the initial capitalization pattern for Examples indicates that any such fields are custom entity-specific fields that won't be enumerated in the ECS schema, and won't collide with any fields that may be defined by ECS in the future. |

The existing flattened fields in ECS don't use .* in the name. I think it makes sense to remove here too, to be consistent

[MikePaquette]

Thanks @mjwolf what do you mean by "flattened" here? As proposed, the object will contain keyword or boolean leaf fields, and not the flattened field datatype. No problem with removing the .* though, thanks.

[mjwolf]

Do you think generic will have any other fields apart from generic.entity.*

[MikePaquette]

Not foreseen. Can anyone else think of a reason why we'd use the root field set generic.* for another purpose?

[romulets]

I can't think of any other reason

[mjwolf]

Could you expand on what this means, I don't think I understand it. What do you mean by "used". Why can a data store only use entity.* if the producers can write to other top level fieldsets?

I think this is also an implementation specific detail that doesn't need to be part of ECS, so maybe it can be removed.

[MikePaquette]

I agree we can remove this from the ECS spec, but the idea is that the entity store schema will include entity.* at the root level in each document. this will allow us and users to query the entity store for entity ID's (and other attributes) w/o burdening them with knowing under which entity class {host, user, service, generic, etc.} it might be stored.

[romulets]

@MikePaquette as we speak we are storing this information under entity.Metadata.EngineType. What do you think about keeping it under metadata?

[romulets]

Is this a new guideline? Or is it something that already happens? I'm not aware of an entity schema published in Elastic Security Docs that usually conforms to elastic stack version

[MikePaquette]

No, I found this in the existing index mappings 8.19.1, and was not aware of it. Included here in case it was already in use, but it does not appear to be used, so we can remove it from this RFC.

[romulets]

The current pattern of snake case with capital first letter isn't friendly for most coding tools and linters, since it's not a standard pattern.

Could we adopt a pattern such as PascalCase. Reading the documentation over capitizaion for non ecs fields I see precedent for it, since it's mentioned both in HAProxy and NGINX examples

[MikePaquette]

@mjwolf do you have a position on this topic?

[mjwolf]

ECS uses snake case for multiword fields, so I think it makes the most sense to keep using that. OTel semantic conventions also uses snake case.

I think for these examples in documentation it should keep using snake case. For the actual implementations, we don't require it, so other cases are allowed (as long as it starts with a capital).

[romulets]

I understand what you mean "static or semi-static attributes" as opposed to lifecycle and behaviour fields, but maybe we have a better way of describing what do we mean by that? By the static or semi-static definition, first_seen and issued_at would also fit under it, wouldn't it?

As I see attributes should be used to describe non temporal entity properties that could not be expressed in other parts of the entity field set.

[MikePaquette]

I see your point, perhaps we can add "non-temporal" to the definition of the entity.attributes.* fields? Would that help?

[romulets]

I think so!

[romulets]

Should be of type flattened instead of object, no?

[MikePaquette]

I think so for entity.raw . We've agreed to remove the .* from the objects, and this would be the same.

[romulets]

I can't think of any other reason

[romulets]

I believe Field Usage it's still a good section name since we are also talking about ECS consumers using entity.* as root