Fix coverage for recent lcov (>= 2)

[ddeclerck]

This PR attemps to fix coverage with recent versions of lcov.

This is achieved through different options, each solving a different problem.

CODE_COVERAGE_LCOV_RMOPTS_DEFAULT="--ignore-errors unused"

This is required because this appears in the logs:

lcov: ERROR: 'exclude' pattern '/tmp/*' is unused.
	(use "lcov --ignore-errors unused ..." to bypass this error)

Note that the /tmp/* pattern is added by m4/ax_code_coverage.m4, so maybe an updated version of this M4 script would work with newer versions of lcov ?

CODE_COVERAGE_LCOV_OPTIONS_DEFAULT="--ignore-errors source"

This is required because this appears in the logs:

geninfo: ERROR: unable to open /home/runner/work/gnucobol/gnucobol/_build/cobc/ppparse.c: No such file or directory
	(use "geninfo --ignore-errors source ..." to bypass this error)

This error occurs for all generated files (cobc/pplex.c, cobc/scanner.c, cobc/ppparse.c, cobc/parser.c), probably because they are in the source tarball and hence missing from the build directory (copying them here would also workaround the error). Since all these files are in CODE_COVERAGE_IGNORE_PATTERN, it seems safe to ignore this error anyways, as we're doing here.

Then, we also apply a "fixup" for a warning:

CODE_COVERAGE_LCOV_OPTIONS_DEFAULT="--rc geninfo_unexecuted_blocks=1"

Needed because we have this:

geninfo: WARNING: /home/runner/work/gnucobol/gnucobol/libcob/numeric.c:469: unexecuted block on non-branch line with non-zero hit count.  Use "geninfo --rc geninfo_unexecuted_blocks=1 to set count to zero.
	(use "geninfo --ignore-errors gcov,gcov ..." to suppress this warning)

Finally, to be more future-proof, we define those:

CODE_COVERAGE_LCOV_SHOPTS_DEFAULT="--rc branch_coverage=1"
CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT="--rc genhtml_branch_coverage=1"

Because we have these warnings:

geninfo: WARNING: RC option 'lcov_branch_coverage' is deprecated.  Consider using 'branch_coverage. instead.  (Backward-compatible support will be removed in the future

genhtml: WARNING: RC option 'genhtml_branch_coverage' is deprecated.  Consider using 'branch_coverage. instead.  (Backward-compatible support will be removed in the future

But maybe an updated version of ax_code_coverage.m4 could work ?

[GitMensch]

The macro was updated in 2024 and explicit does not have the tmp part in any more https://gitweb.git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=history;f=m4/ax_code_coverage.m4, let me update that (and possible other m4 macros - that's planned in any case) and see where it leads.

[GitMensch]

The big question not answered here yet: does your configuration work with lcov 1 (at least 1.16, which is in Debian stable) as well?

[GitMensch]

Answer to my question: just using branch_coverage (without the lcov_/genhtml_ prefixes) raises no warning or error in lcov-1.16... but it does not do the branch coverage any more. The macro should therefore be conditional on the version...

[GitMensch]

--rc geninfo_unexecuted_blocks=1 could be added to Makefile.am (it doesn't seem to raise anything with lcov1), if upstream autoconf-archive does not add it; maybe we get the branch coverage conditional on lcov version (like it is done on meson) - feel free to answer-all to my mail with a patch of this autoconf macro, if you feel that way :-)

For the errors - we could use --exclude with the ignore pattern automatically in the macro, while this can be done with lcov 1 as well (which does not error in the case that source files from gcdata are not found) - but sadly that doesn't skip the file open - see linux-test-project/lcov#411
Just moving the files may work - but would be possibly problematic because they still contain the #line directives and should match the path they are in.
... again "--ignore-errors source" works also in lcov1, so we could add that to Makefile.am as well.

This effectively leaves only the warning about deprecated options in, which we can live with (until the option is removed, so hopefully the autoconf-archive m4 is adjusted and updated until then).

[ddeclerck]

Answer to my question: just using branch_coverage (without the lcov_/genhtml_ prefixes) raises no warning or error in lcov-1.16... but it does not do the branch coverage any more. The macro should therefore be conditional on the version...

Yeah, I noticed that. Note that lcov_branch_coverage and genhtml_branch_coverage are still accepted, although deprecated. I made the change just for things to be cleaner.

Also note that all the proposed fixes are only in the workflow definition, which is unlikely to rollback to an older version of lcov.

[GitMensch]

Also note that all the proposed fixes are only in the workflow definition, which is unlikely to rollback to an older version of lcov.

Yes, but the issue is that users run it as well, and we don't want coverage to error then :-)

[GitMensch]

Part of this is handled upstream; running code-coverage in a subtree (where the srcdir has the generated bison/flex files directly in cobc) raises warnings about the generated files not being in the subtree (ignored) and same for scanner.l (abort) ?!?

[ddeclerck]

Part of this is handled upstream; running code-coverage in a subtree (where the srcdir has the generated bison/flex files directly in cobc) raises warnings about the generated files not being in the subtree (ignored) and same for scanner.l (abort) ?!?

Yes, I mentionned that in my initial comment, that's why we need --ignore-errors source (or we could even use --ignore-errors source,source to completely silence the warning).
Note that this already occured with lcov 1.15, except it was "only" a warning.

Anyways, I rebased and checked the results, both with lcov 1.15 and lcov 2.
Works fine - just two remaining warnings about deprecated options that can be silenced with:

CODE_COVERAGE_LCOV_SHOPTS_DEFAULT="--rc branch_coverage=1"
CODE_COVERAGE_GENHTML_OPTIONS_DEFAULT="--rc genhtml_branch_coverage=1"

[GitMensch]

That's nice. The remaining question is if we can make those two conditional in the code-coverage m4 macro depending on the lcov version - because otherwise this will be broken outside of the CI (either for lcov1 or lcov2 environments).

[ddeclerck]

That's nice. The remaining question is if we can make those two conditional in the code-coverage m4 macro depending on the lcov version - because otherwise this will be broken outside of the CI (either for lcov1 or lcov2 environments).

Are you suggesting we modify m4/ax_code_coverage.m4 ? I'm not really M4-proficient.

To me it would be okay to not use those new options for now and let the deprecation warning occur...

[GitMensch]

Are you suggesting we modify m4/ax_code_coverage.m4 ?

Yes, then send it upstream for review and maintenance.

I'm not really M4-proficient.

Me neither, but that m4 file is mostly shell-script written in macro-style.
And you use m4 all the time for the testsuite sources :-)

To me it would be okay to not use those new options for now and let the deprecation warning occur...

If everything works without those options and we just get warnings, then that would be fine for now - does it work with ubuntu-latest and without those options?

[GitMensch]

Friendly ping @ddeclerck