Override config keys with environment variables

[frankenjoe]

I can see that you are currently improving how configuration files are handled in audb (see #474). I lately found it useful to have - in addition to a user config file - also the option to override values via environment variable. Maybe this could be a useful extension for audb, too?

[hagenw]

Yes, thanks for the suggestions. We support it partially at the moment with AUDB_CACHE_ROOT and AUDB_SHARED_CACHE_ROOT, see https://audeering.github.io/audb/caching.html.

The problem why we have not use it for all entries at the moment is that audb.config.REPOSITORIES does not contain text, but Python objects. In the YAML file this is solved by a lengthy entry, e.g.

repositories:
    - name: audb-internal
      backend: s3
      host: s3.dualstack.eu-north-1.amazonaws.com
    - name: audb-public
      backend: s3
      host: s3.dualstack.eu-north-1.amazonaws.com

So far I wasn't sure how to best translate this in an environment variable.

[frankenjoe]

In my case I support string, int, float, bool, list and dictionary. In case of list and dictionary, the user must provide it is a JSON string.

[frankenjoe]

The function to convert the value to the expected type looks like this:

def _convert_type(
    value: typing.Any,
    default: typing.Any,
) -> typing.Any:

    if default is None:
        return value

    if isinstance(default, bool):
        return value.lower() in ("true", "1", "yes")
    elif isinstance(default, int):
        return int(value)
    elif isinstance(default, float):
        return float(value)
    elif isinstance(default, (list, dict)):
        try:
            return json.loads(value)
        except json.JSONDecodeError:
            msg = (
                f"Environment variable value "
                f"'{value}' "
                f"for key does not match expected type "
                f"{type(default).__name__}"
            )
            raise ValueError(msg)
    else:
        return value

And this is the function that looks for matching environment variables:

    def from_env(self):
        for key in self.__dict__:
            env_name = f"{define.CONFIG_ENV_PREFIX}{key.upper()}"
            if env_name in os.environ:
                value = os.environ[env_name]
                value = _convert_type(value, self.__dict__[key])
                self.__dict__[key] = value

[frankenjoe]

I.e. the user should be able to set the repositories with something like:

os.environ["AUDB_CONFIG_REPOSITORIES"] = '[{"name": "audb-internal", "backend": "s3", "host": "s3.dualstack.eu-north-1.amazonaws.com"}, {"name": "audb-public", "backend": "s3", "host": "s3.dualstack.eu-north-1.amazonaws.com"}]'

[hagenw]

Cool, thanks for the example. I guess, I will target to add the helper functions to audeer and then use it in all packages that provide configuration files to have consistent behavior.