feat: Adapting flex-config to SensorsToShowSchema

[joshuaunity]

Description

This PR adopts a new shape for our SensorsToShowSchema, and also includes the adaptation of the flex-config into the schema as well.

TODO

• Add changelog entry
• Allow backwards compatibility for data with the old schema structure
• Implement new schema structure to allow sensor or sensors, as well as assets' flex-config to be added to plots
• Update all existing sensorToShowSchema test cases with the new expectations

Look & Feel

None

How to test

This new feature can only be interacted with through the API for now.

1. Pick any asset(make sure it has sensors you can reference)
2. Go to the API http://localhost:5000/api/v3_0/assets/<asset_id>
3. Update the asset's sensor_to_show with the below data

NOTE: for the below, use sensors that are under the current asset being updated

[
  {
    "title": "No Title",
    "sensors": [
      <sensor_id>
    ]
  },
  {
    "title": "No Title 2",
    "plots": [
   
      {
        "asset": <asset_id>,
        "flex-model": "soc-min"
      },
      {
        "sensors": [
          <sensor_id>,
          <sensor_id>
        ]
      }
    ]

  }
]

Further Improvements

Support for the addition of fixed values as plot entries.

Related Items

This PR closes #1880

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #31853255 | 📁 Comparing cf1d6bb against latest (f1707fc)

🔍 Preview build

Show files changed (11 files in total): 📝 11 modified | ➕ 0 added | ➖ 0 deleted

File	Status
changelog.html	📝 modified
genindex.html	📝 modified
_autosummary/flexmeasures.data.models.charts.belief_charts.html	📝 modified
_autosummary/flexmeasures.data.models.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.schemas.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.schemas.sensors.html	📝 modified
_autosummary/flexmeasures.utils.coding_utils.html	📝 modified
_autosummary/flexmeasures.utils.html	📝 modified
_autosummary/flexmeasures.utils.unit_utils.html	📝 modified
api/v3_0.html	📝 modified
views/asset-data.html	📝 modified

[nhoening]

Alright - I could get something to work here, which is nice! There are some bigger TODOs, but the approach is valid!

I set this JSON:

[{"title": "P3", "sensors": [44208]}, {"plots": [{"asset": 3749, "flex-model": "soc-min"}], "title": "flex-model-tests"}]

I believe it is very close to the example you give in the PR description.

Note: my method of testing was to set the attribute via SwaggerUI.
That meant I had to escape quotes in my JSON - do you have a nicer workaround of setting these values?

I have some documentation remarks and others inline.

Here are my larger remarks. Maybe you deliberately do not want to work on anything in the graph modal dialogue - then take point 2 and 4 as potential inspiration for the PR where you actually work on that :)

Test

I noted that inline, too. Tests for the schema should include at least one of the newer input values.

Sensor description

In the UI, I then see for this graph:

ID: [undefined](http://localhost:5000/sensors/undefined), Unit: undefined, Name: undefined, Asset: undefined, Account: PUBLIC

Even when the sensor's data is loaded okay.

• We need to display here which flex-config field was used (e.g. "flex_model.soc_min of asset 3749")
• And then the sensor information as we do it for others.

Fixed value on flex config

For the fixed soc_mon ("30kWh"), I see no data
After I took a better look at the code, it seems there is no support yet for fixed values from the flex config - maybe a misunderstanding from an earlier question.

When fixed values are in the flex config, I do want them supported here.

Not (yet) defined flex config fields

It didn't matter if the soc-min field had been defined on asset 3749's flex model or not. I saw the same info about my plot in the left part of the modal dialogue.

If the field does not exist on the asset's flex-model, we should probably say so in the UI. The user should see the plot, as the intention to show data gotten via the flex config, but that the field still needs to be set.

Validate incoming data

I noticed that the incoming sensors_to_show JSON (via the API) is being checked on well-formed JSON, but not against the SensorsToShowSchema. See here, where that could actually happen.

That might make a good addition. I know your modal dialogue probably only sends valid content, but for me testing via the API and also for debugging and seeing bugs earlier, it is a good idea to only let good data in. I hope it would be rather easy to do?

[nhoening]

So we have a list of fields we expect ("plots", "sensors", "sensor", "title", ... - any others?) and can raise a validation error if we get any other.

If an invalid key was added, it doesn't break the UI in any way, so this validation would just be a nice-to-have. Are there scenarios you think foreign keys would cause a problem?

My motivation was that it's quicker for a bug to be spotted, e.g. is the frontend somehow sends an unknown field. But okay let's move on.

[nhoening]

Thanks!

Please confirm testing the first tutorial is still okay.

[nhoening]

Thanks!

Please confirm testing the first tutorial is still okay.

@joshuaunity have you confirmed this?

[joshuaunity]

Thanks!
Please confirm testing the first tutorial is still okay.

@joshuaunity have you confirmed this?

I'll circle back on this after the completion of the UI work

[Flix6x]

I did some refactoring in #1984, please review.

[joshuaunity]

i have confirmed that this error also occurred on the main branch

[nhoening]

i have confirmed that this error also occurred on the main branch

I re-ran locally and everything is fine on main.

The collation error hints at your db container or data being in the way somehow.

Try removing the db image (docker rmi flexmeasures-dev-db-1) and also deleting the local data (sudo rm -rf docker-compose-data).
Or follow the hint in your screenshot to fix the data collation. But as it's recreatable dev data, probably removing and restarting is easier.

[joshuaunity]

Try removing the db image (docker rmi flexmeasures-dev-db-1) and also deleting the local data (sudo rm -rf docker-compose-data).

This works now for me; my data was the issue, not the code

[nhoening]

Just two small text suggestions that can be accepted right now.

I made two requests to use the new capabilities in the tutorial, but that is better addressed in PR #1926 (I will link to them in my review there).

Merging strategy once both are approved: Let's merge #1926 into this one, then merge both as a whole.