Set default values for <obtain_from_environment> with [DEFAULT] section

[ddietz89]

If the reconcile fails here

olcf-test-harness/harness/machine_types/rgt_test.py

Line 521 in 816d6a7

def _reconcile_with_shell_environment_variables(self):

Could we grab a default value from a [DEFAULT] section in the config?

For example:

[DEFAULT]
nodes = 123

[Replacements]
job_name = GFDL_AVEC_c768L127nh
batch_queue = batch
nodes = <obtain_from_environment>
cluster = c6

The DEFAULT section currently works, however, only if the line is completely omitted. If you leave the variable unset, the harness code complains during reconciliation. Some logic could pull the value from the defaults if variable is unset in the environment (possibly continue throwing an error if there are other issues).

[hagertnl]

Also, improve the messaging when an obtain_from_environment is not found. For example ERROR: This test requires the environment variable RGT_NODES to be set., vs something more like ConfigParser error: nodes = <obtain_from_environment> is not found

[hagertnl]

Should this actually be accomplished in the machine.ini file, rather than the test's .ini file? Because of #183 , we can now define arbitrary environment variables globally, including default values for things like RGT_NODES or other variables. These are overridden by whatever is set by the user.

[hagertnl]

I was thinking about this more. I might actually like some sort of syntax like this to allow defaults. Maybe something like one of these:

nodes = <obtain_from_environment> DEFAULT 123
nodes = <obtain_from_environment>(123)
nodes = <obtain_from_environment(123)>

The first option makes it the most clear that you're defining a default value.