Proposed trait extension of data model

[sydnerecord]

We propose a trait extension of the Ecological Community Data Design Pattern (ecocomDP), which was developed for harmonizing community ecology biodiversity data (O’Brien et al., 2021). The original ecocomDP model is extended with two features: 1) reconfiguring the table for mapping variables to external dictionaries to allow any variable attribute (e.g., a trait) to be recorded and linked to an external dictionary of concepts, such as ontologies, and 2) adding additional descriptive fields to the dataset summary table. Because ecocomDP already accommodates community-level analyses, this extension would enable researchers to ask questions across levels of biological organization (e.g., from individuals to populations to communities). Another advantage of incorporating trait data into ecocomDP is that existing NEON and Long Term Ecological Research (LTER) Network data from various taxa are already harmonized with ecocomDP, making it a good candidate for the future incorporation of traits. We are working on testing this new trait extension with Metagenome-derived Reference Genomes and Traits for microbes in collaboration with the EMERGENT LTER working group, plant phenological traits with the Environmental Science Innovation and Inclusion Laboratory Macrophenology group, and NEON beetle traits in collaboration with the Imageomics Institute.

[clnsmth]

Hi @sydnerecord,

Thanks for the heads up on these proposed data model changes. What's the leading candidate for these changes at the moment?

It would be great to eventually see how these would look in the model documentation (the webpage rendered from this R Markdown file), but it might be too early to mock that up just yet?

Thanks!

[clnsmth]

Heads up @sydnerecord, @sokole - I've created the branch 158-traits-extension for us to iteratively work on this issue. This branch is based off of development and will be merged in once our work is stable enough.

[sydnerecord]

Thanks!!

…

On Wed, Jul 9, 2025 at 5:10 PM Colin Smith ***@***.***> wrote: *clnsmth* left a comment (EDIorg/ecocomDP#158) <#158 (comment)> Heads up @sydnerecord <https://github.com/sydnerecord>, @sokole <https://github.com/sokole> - I've created the branch 158-traits-extension for us to iteratively work on this issue. This branch is based off of development and will be merged in once our work is stable enough. — Reply to this email directly, view it on GitHub <#158 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACK6RVG43Y2HASMU6FKQHGD3HWAN5AVCNFSM6AAAAAB4PPHN32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANJUGA2DGNJWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[clnsmth]

Hey @sydnerecord and @sokole ,

I'm sitting down to plan out the code changes for the new model and have a few thoughts and questions:

variable_mapping Table

Can we leave the name of this table as it is rather than changing it to variable_attribute? Changing the name introduces a backwards-incompatible breaking change to the model, and I'm not sure there's a significant semantic difference between the two names. But maybe there is?

variable_type Column

Is the variable_type column in the proposed variable_attribute table supposed to be free text? I'm wondering if we could make it a URI value from a vocabulary/ontology to make the data more machine-actionable. I also wonder if we could make this new column useful for variables in general, not just trait-related variables?

dataset_summary_table

• dataset_level_bio_organization: Do we have a recommended vocabulary or ontology to control the terms in this column for machine actionability?
• observation_finest_level: Similarly, do we have a recommended vocabulary or ontology for the terms in this column?
• Number of variables: What's the utility of this column? Should it only apply to trait variables, or can it apply to other variables as well?

I just wanted to confirm a few things before I launch into the code changes. Thanks!

[mobb]

Per @clnsmth suggestions:
We should talk about possible backward incompatibility. I think that the name "variable_attribute" is not quite right, for a couple of reasons: ecocomDP is EML based. And in EML, a table column is called an "attribute" - as in the database sense, where a a data 'object' (e.g., table) has an 'attribute' (e.g. column). So it would not be a good idea to attach additional meanings to the term "attribute".

After looking over notes, leaving the table name variable_mapping doesn't work any more, because the table now has more than just a mapping (it has other information). Maybe "variable_info" is a better choice. But we should decide on how to deal with version incompatibility first (starting a new issue for that).

the column variable_mapping.variable_type - I thought we made that clear that this was not free text, but a controlled vocabulary. The vocabulary currently has only 2 values: "trait" or "other", so it can hold traits right away, and any other descriptions of type are "other". Of course, the vocab can be extended to include other types if these arise in the future.

dataset_summary_table: I thought we defined the new content here, too. I'll check the docs.

[clnsmth]

The implementation of the traits extension in the code base is complete, with some notable caveats:

• The NEON mapping functions have not yet been modified; this still needs to be done.
• The variable_mapping table has not been renamed to the variable_attribute table as proposed in the traits documentation. We are waiting for an agreement on this before changing it, as it will introduce a backwards-breaking change.

With that in mind, I think we're in a pretty good position to try reformatting an L0 dataset with traits, into ecocomDP.

[clnsmth]

@sydnerecord, @sokole, @mobb, and I met to discuss the variable_mapping table name change and opted against it, as it would create a backwards-breaking change with the ecocomDP datasets in the EDI data repository.

I'll make an editing pass to remove any use of the proposed variable_attribute table.

This topic is also mentioned in #167.

[clnsmth]

One thing that came up during the ESIIL Macrophenology working group last week was the need to define more precise semantics for the values in the variable_name columns of the L1 dataset. The goal is to be able to group a set of columns under a single relation about a morphological trait or a phenological phenomenon.

Under the current variable_mapping table, we cannot do this for two reasons:

1. There is an assumed predicate (i.e. is a measurement of) to the mapping.
2. The table is only meant to accommodate one semantic annotation per variable name. In our use case, we want to more precisely define the semantic predicate as well as allow a number of annotations to be put on a single variable name.

Therefore, there is a proposed change to this table:

1. Add predicate_label and predicate_uri columns.
2. Rename the mapped_label and mapped_id columns to object_label and object_uri, respectively.

This second item would introduce a backwards incompatibility that we have been able to avoid up until this point. One way around this is to simply keep the mapped_label and mapped_id column names as they are.

I see two possible ways forward, but there may be others:

1. Continue forward with the next version of the ecocomDP data model as we have discussed up until this point.
2. Improve the semantics of the variable_mapping table and include these proposed changes in the model update.

Thoughts on how to proceed @mobb @sydnerecord @sokole?

[mobb]

I agree that we need to improve the semantics, and that the variable_mapping table as is is limited. BTW, it can handle more than one mapping per variable - each would have a different ID. This also should apply in the future (ie, to a changed table, or the proposed semantic_metadata table).

I think for now, the conversation is whether/how to do something backward-incompatible, and what the ramifications are.

But here's a thought: the flat-file representation we are talking about only uses 6 of the tables - it doesn't use the variable mapping table at all. We suspect that the variable mapping table may be under-used.

So, what if we think of this not as an "ecocomDP v2", but instead it's a Level 2? That is, it's a transformation of the 6-table ecocomDP dataset, with an additional table of semantics? there would be no changes to the L1 model at all. Anything in the variables_mapping table would be added to the semantic_metadata table during the transformation. Of course, other Level 2s are still possible.

There are two things going on:

1. the general form of the 2 tables (flat and semantic-metadata), which it looks like is coming together
2. the traits use case. I think it would be important (at least very helpful) if there was some control over the vocabularies chosen for predicates and objects in the semantics table. for example, if we are going to try to infer that a variable is a trait, one way to do that is if it's from a Trait Ontology. Further, we already want to have our own semantic vocab, that describes the L1-columns. And we have planned (or maybe already done?) annotations to darwin-core fields. so we're already exercising some control.

anyway, what do you think? is the idea of an ecocomDP-sanctioned-Level-2 confusing?