Implement VOCS interface

[shuds13]

This implements Step 3 in #269

For first pass, keep internals the same and implement interface:

Implementation:

• Implement interface for simple generators (base/line/grid/random sampling) and tests
• Check correct inheritence of generators from standard.
• Add basic validate_vocs functions
• Handle setting type of observables
• Handle / replace extra parameter attributes in Optimas
• Check exploration_diagnostics changes
• Determine if need integer variables
• Basic update Ax generators and tests
• Update Ax generators constraints specification to use VOCS
• Update multitask gen to use _id and internal mapping - undone
• Update multitask to use discrete variable for trial_type.
• Discrete variables should support strings at base class.
• New: Update conversion functions for new vocs where observables are objects.
• EXPLORE objective not used - if not minimize/treated as maximize. Perhaps include in gens validate_vocs that no EXPLORE. Or always raise error for now.
• Update scripts in examples dir
• Update notebooks for vocs
• Update docs for vocs (merge in Update documentation for vocs #285)

Points that could be addressed in follow up PRs (tick means discussed and added to new issue).

• Update validate_vocs functions and incorporate previous check functions.
• Add tests for validate_vocs exceptions
• Decide whether to remove extra fields given to gens (_ignored, _ignored_reason, _status).
• Constraints are given in tests, but are currently ignored in Optimas. How to deal with this.
• Constants are not used currently in Optimas generators, we could add tests or leave until needed.
• Verify that the fields passed between ingest/suggest methods are correctly derived from VOCS specifications (due to keeping the old internals, the generators work with Optimas legacy parameter handling). Indirectly this happens.
• Add xopt generator example.

Checks:

• Ensure examples work (not just tests) - run on Perlmutter.
• Ensure works with external VOCS generators from libensemble. See Add APOSMM nlopt example #283.
• Check/update docstrings and other references to old data structures
• Check for any TODO comments remaining.
• Ensure using gen standard main rather than a branch

Reviewer checks:

• Check results
• Check and update validate_vocs functions
• Check issues outlined below and solutions
• Check generator/test docstrings and comments
• Check notebooks in docs
• Check output from examples run on Perlmutter (<groupspace>/shudson/vocs_std_gens/optimas/examples).
• Check docs

[shuds13]

In updating Optimas generators for the standard, how do we deal with these extra fields in VaryingParameter.

    lower_bound: float
    upper_bound: float
    is_fidelity: Optional[bool] = False
    fidelity_target_value: Optional[float] = None
    default_value: Optional[float] = None
    _is_fixed: bool = PrivateAttr(False)

re: https://github.com/optimas-org/optimas/blob/main/optimas/core/parameter.py#L79

In the standard, variables have default_value & domain, but no equivalent to the others.

https://github.com/campa-consortium/generator_standard/blob/main/generator_standard/vocs.py#L16

Do we need is_fidelity / fidelity_target_value for each parameter? Or could it be a generator option?
(fidelity_target_value is given to Ax as the fidelity we want to optimize for).

is_fixed allows fixing a parameter value between calls to run(). Generator still has to support it.

e.g.

    var1 = VaryingParameter("x0", -50.0, 5.0)
    var2 = VaryingParameter("x1", -5.0, 15.0)

    # Start with a fixed value of x0.
    var1.fix_value(-10.0)
    ...
    exploration.run(n_evals=5)
    var1.free_value()
    gen.update_parameter(var1)
    exploration.run(n_evals=5)

could be instructed directly to generator instead - for supporting gens.

e.g.

gen.fix_var('x0', -10.0)

Alternatively, we could add fields to the standard, or create subclasses to ContinuousVariable for Optimas. This may be more elegant.

Currently, I have used the following generator functions (shown with example calls) to set the Optimas internals for these, without adding to or subclassing ContinuousVariable in VOCS.

gen.fix_value("x0", -10.0)
gen.update_range("x0", -20.0, 0.0)
gen.set_fidelity_param("res", is_fidelity=True, fidelity_target_value=8.0)

These are used in test_ax_generators.py.

[shuds13]

Observables (analysed parameters) in Optimas can specify a type, but this isn’t supported in the standard interface. A type is necessary for constructing a NumPy structured array (for libEnsemble).

For example, in this test, the type must be a string.

https://github.com/optimas-org/optimas/blob/main/tests/test_env_script.py#L27

In test_template_evaluator.py and test_function_evaluator.py it stores an array and a plot object.

I have suggested we allow specifying a type for observables and constants in the standard.
campa-consortium/gest-api#39

The standard does not support arrays generally, but I have made it possible to specify a string or a tuple for observable type, which means numpy arrays can be used.
campa-consortium/gest-api#40

e.g.

def test_template_evaluator():
    # Define variables and objectives.
    vocs = VOCS(
        variables={"x0": [-50.0, 5.0], "x1": [-5.0, 15.0]},
        objectives={"f": "MAXIMIZE"},
        observables={
            "p0": (float, (2, 4)),
            "p1": "O",
            "fig": "O",
        },
    )

    # Define variables and objectives.
    gen = RandomSamplingGenerator(vocs=vocs)

[shuds13]

exploration_diagnostics:

exploration_diagnostics.py:
_create_exploration Still reads from a file which will contain data in the old Optimus format.

This file is created and read using Optimas internal parameters (varying_parameters etc), so it is consistent. It will be updated in a future cycle if the internals are updated to use vocs directly.

test_exploration_diagnostics.py:
Removes a check that now seems redundant

        # Check that all possible objective inputs give the same result.
        _, trace1 = diags.get_objective_trace()
        _, trace2 = diags.get_objective_trace("f1")
        # _, trace3 = diags.get_objective_trace(obj)  # Can be removed
        np.testing.assert_array_equal(trace1, trace2)
        # np.testing.assert_array_equal(trace1, trace3)

[shuds13]

Note that validate_vocs gets called in the generator standard base class. This makes it a bit difficult at the moment to incorporate the other functions like _check_parameters or _check_inputs as they work on the converted data types.

[shuds13]

Small point. In test_ax_generators.py constraints use <= in Optimas, where as standard uses LESS_THAN.

outcome_constraints=["p1 <= 30"],

becomes:

constraints={"p1": ["LESS_THAN", 30.0]},

[shuds13]

What do we do for test test_ax_single_fidelity_int. Do we need to support this?

    var1 = VaryingParameter("x0", -50.0, 5.0, dtype=int)
    var2 = VaryingParameter("x1", -5.0, 15.0)

In vocs this becomes:

    vocs = VOCS(
        variables={"x0": set(range(-50, 6)), "x1": [-5.0, 15.0]},
        objectives={"f": "MAXIMIZE"},
    )

I don't see anything to deal with discrete vars in Optimas, so I have converted to integer type when its a range but that seems hacky.

To support this elegantly, we could bring back IntegerVariable type in the vocs standard.

[RemiLehe]

Do these variables fit the standard of VOCS?
My impression is that it would be better to pass them to AxMultiFidelityGenerator instead.

[shuds13]

Fixed, I will be running, testing the full examples on Perlmutter soon.

[RemiLehe]

I am a bit confused ; does this if condition ever become true?
Given that we have:

        custom_trial_parameters = [
            TrialParameter("arm_name", "ax_arm_name", dtype="U32"),
            TrialParameter("ax_trial_id", "ax_trial_index", dtype=int),
        ]

[RemiLehe]

Actually: looking at the rest of the code in this PR, it seems that this would be a case where we use set_fidelity_param on the generator, no?

[shuds13]

It's created in the generator. When a discrete variable type is given in vocs. Optimas internally uses trial_parameters. So trial_type is a discrete variable.

optimas/optimas/generators/base.py

Line 204 in fa74f3f

def _convert_vocs_discrete_variables_to_trial_parameters(

In the multifidelity generator (resolution), its using a Continuous variable, which is basically the same as it was before in Optimas (using a VaryingParameter), but it may be fidelity is better expressed as discrete always?

The arm_name and ax_trial_id are created in the generator as these are not vocs parameters. They are fixed generator fields.

[RemiLehe]

@shuds13 Thanks again for this PR!
It looks like this notebook also needs to be updated for the tests to pass:
https://github.com/optimas-org/optimas/blob/vocs/doc/source/user_guide/advanced_usage/build_gp_surrogates.ipynb

[shuds13]

Updated notebooks, docs is now building

[shuds13]

Now we have an install error in pytorch on CI.

undefined symbol: iJIT_NotifyEvent

Same on main. This is seems to be an environment resolution issue and may resolve itself.

That said, suggestions include adding:

conda install -y -c conda-forge ittapi

or getting pytorch from conda-forge channel instead of pytorch channel (uses OpenBLAS instead of MKL)

Trying to fix at #284