Configuration-to-configuration and configuration comparison

[thorwhalen]

The context of this discussion was to (see private opus discussion) make some tools that helped CI template migration.

But really, as we go up the abstract stack, we realize that, once templates have been "embodied" -- that is, conventions have been expanded to their explicit configuration, optionally merged with any specific configuration edits that may have been specified -- our problem boils down to comparing mappings, and then doing something based on that comparison.

Original context

(...)

Perhaps something like this might help, to both isolate and study the differences, and therefore know what to do about them -- if they're not too many, perhaps diagnosis + manual edits is the way to go:

from meshed import code_to_dag

@code_to_dag
def diagnose_differences():
    repo_yml = get_repo_yml(repo)
    repo_info = get_repo_info(repo)
    conventional_configuration_yml = expand_convention(convention_yml, repo_info)
    conventional_configuration_dict = yml_to_dict(conventional_configuration_yml)
    actual_configuration_dict = yml_to_dict(repo_yml)
    differences_dict = diff_dict(actual_configuration_dict, conventional_configuration_dict, equivalence_func)
    _ = look_at_differences_and_decide_what_to_do(differences_dict)

diagnose_differences.dot_digraph()

Note: equivalence_func is there because the default comparison operator (operator.eq, that is, ==) may not be what's appropriate, so we might want to control that bit.

Bits of code

Two of the functions here are candidates for mini-frameworks: expand_convention and diff_dict.
The expand_convention could be a config2py mini-framework, and is related to embody (as well as third party tools like cookiecutter, flit, etc.).

Here's some starter code for the rest.

import json
from pathlib import Path
import os

def get_repo_yml(
        repo: str, 
        *,
        src_template: str = 'https://raw.githubusercontent.com/{repo}/master/.github/workflows/ci.yml'):
    r"""
    Get the contents of the .qo.yml file in the repository.

    >>> get_repo_yml('i2mint/config2py')  # doctest: +SKIP
    'name: Continuous Integration\non: [push, pull_request]\nenv:...
    >>> get_repo_yml(
    ...     'config2py', 
    ...     src_template='~/Dropbox/py/proj/i/{repo}/.github/workflows/ci.yml'
    ... )  # doctest: +SKIP
    'name: Continuous Integration\non: [push, pull_request]\nenv:...

    """
    src = src_template.format(repo=repo)
    if src.startswith('http'):
        import requests
        r = requests.get(src)
        if r.status_code == 200:
            return r.text
    elif any(map(src.startswith, [os.path.sep, '.', '..', '~'])):
        src = os.path.expanduser(src)
        return Path(src).read_text()
    else:
        raise ValueError(f'Unknown source type: {src}')

def yml_to_dict(yml_content):
    """
    Convert YAML content to a Python dictionary.
    """
    import yaml
    return yaml.safe_load(yml_content)

def diff_dict(dict1, dict2, equivalence_func):
    """
    Recursively compare two dictionaries and return the differences based on the provided equivalence function.
    
    Parameters:
    - dict1, dict2 (dict): Dictionaries to be compared.
    - equivalence_func (func): Function to determine if two items are equivalent.
    
    Returns:
    - dict: Dictionary containing differences between dict1 and dict2.

    >>> diff_dict(
    ...     dict(a=1, b={'bb': 1}, c=3),
    ...     dict(a=1, b={'bb': 2}, c=4),
    ...     lambda x, y: x == y
    ... )
    {'c': (3, 4), 'b': {'bb': (1, 2)}}
    """
    differences = {}

    # Check for keys in dict1 not in dict2 and vice versa.
    for key in dict1.keys() - dict2.keys():
        differences[key] = ("in_dict1_only", dict1[key])
    for key in dict2.keys() - dict1.keys():
        differences[key] = ("in_dict2_only", dict2[key])

    # Check for differences in values for common keys.
    for key in dict1.keys() & dict2.keys():
        if isinstance(dict1[key], dict) and isinstance(dict2[key], dict):
            nested_diff = diff_dict(dict1[key], dict2[key], equivalence_func)
            if nested_diff:
                differences[key] = nested_diff
        else:
            if not equivalence_func(dict1[key], dict2[key]):
                differences[key] = (dict1[key], dict2[key])

    return differences

Note: This diff_dict is just for starters: We'd like to have a more general mini-framework for mapping comparison -- either in dol or here in config2py. One that, for example, doesn't use a boolean equivalence_func but rather a comparison_func that outputs a comparison object that can then be used down-stream for filtering, further analysis, etc.