`elvis.config`: implement `warnings_as_errors`

[kivra-pauoli]

Is your feature request related to a problem? Please describe

Now that we have Elvis -level checks (like the config. checks implemented by @bormilan) I think a good addition to the elvis.config rules would be (false by default) warnings_as_errors where the goal would be to stop execution prior to rule handling if there's warnings.

Another warning we already have in place is "... yielded none" which could benefit from this.

Describe the solution you'd like

An elvis.config -level option warnings_as_errors (:: boolean()) that'd stop execution prior to analysis if warnings were emitted prior to it.

Describe alternatives you've considered

None yet. But we can always do a grep and exit.

Additional Context

n/a

[paulo-ferraz-oliveira]

As proposed in #504 (comment) we could also make old_configuration_format a warning and use this mechanism.

[paulo-ferraz-oliveira]

@elbrujohalcon, though I've never used it we also support application-level config. (application:get/...). Apart from that we support rebar.config and elvis.config. I wouldn't mind deprecating one or two, but you lemme know... (personally I think elvis.config should be enough - and I've never quite understood why the elvis key at the top; was there an idea to support something else at any moment in time?).

To continue with my current implementation I'm gonna support all modes but hopefully in a way it's easy to change.

[paulo-ferraz-oliveira]

Nah, it seems it's advertised, so we'll have to live with it. As long as the format is consistent it should always be the same input/output.

[paulo-ferraz-oliveira]

It seems rebar.config must be used by elvis since it's not referenced in elvis_core and the application-level config. is similar to the elvis.config. (well, maybe they all are/should be)

[paulo-ferraz-oliveira]

Note: elvis_ruleset:set_rulesets is called at least once per time we consult an option.

[elbrujohalcon]

Note: elvis_ruleset:set_rulesets is called at least once per time we consult an option.

Hummm… there is room for optimization there.

[elbrujohalcon]

I'm not sure what to do about the multiple configuration options.
On the one hand, I agree that most if not all of our users use elvis.config, but having Elvis configuration within rebar.config seems useful to me. I'm not sure about the other options, tho…
@bormilan what do you think?

[bormilan]

I don't know either.. But I like to simplify things, so cutting the unnecessary configuration options to make the development better feels good for me.

[paulo-ferraz-oliveira]

My current approach:

1. prefers elvis.config above all else
2. falls back to rebar.config after that
3. falls back to "application env." after that

It's potentially maintainable as is (results only locally, for the time being). And this way we can still accommodate for what we have in tests, Elvis and rebar3_lint (and potentially other external consumers, for which we don't need to break the interface).

It'll hopefully not over-complicate development, since I always read consistently while falling back, so these mechanisms are conceptually isolated. I'll post some results soon, when I move forward some more.

[paulo-ferraz-oliveira]

If a rule is not part of a ruleset, should it be possible to have it in rules alongside that ruleset? (make sure we also validate this...)

[elbrujohalcon]

If a rule is not part of a ruleset, should it be possible to have it in rules alongside that ruleset? (make sure we also validate this...)

Yeah! For instance, custom rules are, in principle, not part of any ruleset. But it should also be possible to specify "I want to use the erl_files ruleset, plus these two strict ones".

[paulo-ferraz-oliveira]

Also mention that rule execution output isn't a warning, but a notice, so warnings_as_errors doesn't affect the exit status. (at this moment?)