Configuration and API Simplification

[gupichon]

Overview

The goal of this discussion is to consolidate the ongoing improvements around configuration and API design in order to maintain overall consistency. It also aims to validate key API choices early, in particular to avoid redundancy or overlap between existing concepts such as YellowPages and ElementHolder, and to ensure a coherent direction across all related developments.

This discussion focuses on improving both the configuration model and the API usability in PyAML. The goal is to simplify workflows, increase flexibility, and enable more dynamic behavior while keeping the system consistent and maintainable.

---

Configuration Model

Backend Linking

The configuration should rely on a key/value manager to link objects to the backend:

• A static YAML catalog provides the base configuration.
• Dynamic catalogs, specific to each control system (CS), allow contextual adaptation.

This approach is aligned with the catalog work introduced in PR #192 .

---

Workflow

The expected workflow is:

1. Load a static configuration

2. Optionally modify it dynamically:

   • Fix errors
   • Adapt to a specific execution context (e.g., different tools or environments)

This implies that configurations should not be considered immutable after loading.

An open question is whether dedicated tools should be introduced to help build configurations. If so, several aspects need to be clarified:

• What form should these tools take (Python API, CLI, external scripts, UI, etc.)
• What should be their source of truth (existing control system, databases, simulation models, catalogs, lattice files, etc.)
• How they should integrate with the current configuration workflow

---

Configuration Composition and Modification

Current State

Configuration composition is already supported through multiple YAML files. However:

• It is not fully user-friendly
• It requires building a main aggregated YAML file manually

---

Proposed Improvements

1. Direct Multi-file Loading

Current state:

Configuration composition is already possible by splitting the configuration into multiple YAML files and referencing them from a main file. However, this approach has several limitations:

• It requires creating and maintaining an explicit main configuration file
• The composition logic is externalized and not handled by the builder itself
• It prevents directly passing multiple independent configuration fragments to the loader
• It makes dynamic or context-dependent composition more difficult

In particular, it is currently not possible to write something like:

Accelerator.load([
    "base.yaml",
    "magnets.yaml",
    "diagnostics.yaml",
])

or:

Accelerator.load(
    "base.yaml",
    additional_configs=["magnets.yaml", "diagnostics.yaml"]
)

It is also not possible to express composition directly within YAML itself. For example, the following is not supported:

devices:
  - bpms.yaml
  - diag_tools.yaml
  - tuning_tools.yaml
  - magnets.yaml

Proposed improvement:

Allow the PyAML builder to accept multiple configuration fragments directly, without requiring a pre-built main file.

This would:

• Simplify user workflows
• Avoid redundant configuration generation steps
• Enable more flexible and dynamic configuration assembly
• Better support programmatic use cases (e.g., tests, simulations, environment-specific overrides)

---

2. Dynamic Post-load Modification

After loading a configuration, it should be possible to:

• Update object references
• Rename objects and tools
• Fix inconsistencies dynamically

---

3. Validation and Mutability Support

To support safe dynamic updates:

• Introduce a validation decorator
• Introduce a "changeable" decorator for mutable elements

This approach has already been explored in PR #214.

Benefits:

• Cleaner and more maintainable code
• Explicit control over what can be modified at runtime
• Safer dynamic reconfiguration

---

API: Access to Tools and Objects

Improving the API for accessing high-level objects is already being discussed in:

• Issue #199

The objective is to provide:

• A more intuitive access pattern
• Reduced boilerplate
• Better discoverability of available objects

---

High-Level Object Management

Introducing a manager for high-level objects could significantly improve consistency across:

• Multiple control systems
• Simulators
• Digital twins

Expected Benefits

• Centralized update propagation
• Consistent object identity across contexts
• Simplified synchronization between environments

---

Relation with Yellow Pages

The Yellow Pages concept could be extended to support this:

• Act as a central registry of high-level objects
• Provide a unified view across control modes
• Facilitate dynamic lookup and updates

[JeanLucPons]

devices:
  - bpms.yaml
  - diag_tools.yaml
  - tuning_tools.yaml
  - magnets.yaml

The above has to be supported and it should be easy to fixed during list expansion:

    # Recursively expand a list
    def expand_list(self, l: list):
        idx = 0
        while idx < len(l):
            value = l[idx]
            if hasToLoad(value):
                obj = load(value, self.files_stack)
                if isinstance(obj,list):
                    l[idx:idx+1] = obj
                    idx += len(obj)
                else:
                    l[idx] = obj
                    idx += 1
            else:
                self.expand(value)
                idx += 1

[JeanLucPons]

I did a PR #226 fix it

[TeresiaOlsson]

A key aspect to include is that the configuration must be able to handle third-party applications or devices or all the labs who are interested in integrating their existing tools and development with pyAML won't be able to do it and we will lose them as contributors and users.

To achieve that I think some things have to be refactored and be made more flexible. These are the things I have found so far which cause problems:

• The assumption that every class will have a ConfigModel. Third-partly classes are not likely to have that and we can't enforce it without forcing labs to make significant redevelopment of their code.
• The assumption that the ConfigModel and the class that will be initialized from it is in the same module. This is also not likely to be the case for all third-partly classes and we can't enforce it.
• The use of `PYAMLCLASS``. This we also can't enforce.

My suggestion is that we take inspiration from HAPPI and generalise the configuration by changing it to specify which class to initialise and which attributes to use to initialise an object of that class. Since that is Python generic it makes it possible to write a generic factory to build any class and optionally validate the input data in case a ConfigModel has been provided, for example by using a validator decorator to link a class with it's validation model. In that way, labs can use the same configuration to initialise both pyAML standard classes and lab specific development in any mix they prefer.

[TeresiaOlsson]

I also suggest that we give up on the idea of directly using the yaml file and start to only consider it as one option as a backend for a config database (other options can be json, mongodb etc depending on what labs want to add support for in the future). This is meant to be hidden from the users and never be directly modified, but instead only be interacted with through a config client with some user-friendly API.

Then there is no need of being able to do something like:

Accelerator.load([
    "base.yaml",
    "magnets.yaml",
    "diagnostics.yaml",
])

because that is something that would be handled internally by the config client. In Accelerator.load you would then instead specify which database to load and if it turns out to be a yaml backend database the database will internally keep track of that it needs to load several yaml files. Or even better, maybe several yaml files are not even needed if no human is ever intended to read the file directly?

The config client can then be responsible for handling dynamic changes, the yellow pages and generating a configuration from scratch and in that way provide a single, clear entry point for the configuration for both users and different parts of the the code that need access to the configuration data. It can also provide the factory to generate objects based on the current configuration data if that turns out to be a good idea as is done in HAPPI.

I envision two ways to use this type of config client.

1. Independently when you want to generate/modify the configuration without initialising any business logic objects. Then you can directly create a client object by loading an existing database or initialising a new one and use the API of the client to interact with the configuration. A GUI application for the configuration could also use the client in this way.

2. As part of an Accelerator object. When doing Accelerator.load this can internally create a client object inside the Accelerator which you can access through the API of the Accelerator to browse the current configuration, do dynamic changes etc.

I also envision that perhaps option 1 could allow for having several accelerators defined inside the same database? If I have understood correctly, that's how the TANGO database works? At least at MAX IV it was the same database for all of their accelerators. Option 2 could then be the way to use scopes to only get the configuration for a specific accelerator in the database. For example, you can use the config client directly if you want to see the config for transfer line, booster, storage ring etc all together but if you use the client through the Accelerator that would give you only the configuration for the one specific machine you have loaded.

I think that would be nice because it would allow you to only have to maintain one configuration database for all your machines if you want. And maybe this could also make it easier to integrate with the TANGO database and use the introspection capabilities of TANGO to automatically fill in some fields in the configuration database for the labs which like that idea.