embunit / unittests: several improvements by miri64 · Pull Request #16145 · RIOT-OS/RIOT

miri64 · 2021-03-04T10:43:44Z

Contribution description

While implementing some embUnit tests, I noticed several issues:

OUTPUT=XML is only supported by tests/unittests,
run_check_unittests() is not used by all tests that use embUnit,
run_check_unittests() is not able to recognize output formats other than the default one an OUTPUT=TEXT.
The color outputters do not reset the color properly.

This PR aims to improve things on all of these fronts. The only output format I did not address was OUTPUT=COMPILER (which only prints when a test fails and only the failed assertions). For that, I think it would make most sense to reverse the logic of run_check_unittests(): expect on errors, and use TIMEOUT as the success condition. But that might not desirable for other tests. So I ignored that output format for now.

Testing procedure

The tests changed should still run and succeed, using OUTPUT=XML provides an XML output.

Issues/PRs references

None

miri64 · 2021-03-05T15:15:54Z

While implementing some embUnit tests, […]

For the curious ones: it were the tests in #16156 ;-)

benpicco · 2021-09-02T17:03:27Z

Murdock does not like it.

miri64 · 2021-09-03T09:16:49Z

Murdock does not like it.

Let's give it another run. Looks more like a murdock error than a build error.

miri64 · 2021-09-03T09:17:32Z

The tests changed should still run and succeed, using OUTPUT=XML provides an XML output.

This still needs to be run manually though ;-)

miri64 · 2021-09-03T09:29:01Z

See #16803 for why I removed the CI: ready for build label.

This allows for use of OUTPUT=XML in all tests using embunit, not just `tests/unittests`. The other output formats work without the outputter being explicitly set.

benpicco · 2021-09-09T14:43:58Z

Murdock still does not like it.

miri64 · 2021-09-09T15:02:12Z

Might be the same problem. Will look into it later!

miri64 · 2021-09-09T15:37:22Z

Might be the same problem. Will look into it later!

Done. Was indeed a wrong OUTPUT definition again :-). Not sure about the suit_manifest test. Seems unrelated.

miri64 · 2021-09-09T15:38:45Z

(but can reproduce locally)

miri64 · 2021-09-09T15:42:40Z

Arghs, used the wrong function in that test. Is now fixed.

miri64 · 2021-09-13T10:04:12Z

Of course, it was only a matter of time that the somewhat generally named OUTPUT macro will collide at some point, somewhere. Will look into this later. If any one wants to pick up this work in the meantime, I would not mind :-).

stale · 2022-04-16T03:02:36Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

benpicco · 2022-04-16T11:03:10Z

Still WIP?

miri64 · 2022-04-16T12:35:52Z

Still WIP?

Nothing happened (except for Stale-Bot actions) since I set that label, so yes ;-).

stale · 2022-11-02T02:28:41Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

MrKevinWeiss · 2022-11-02T10:43:38Z

Maybe @Teufelchen1 would like to take a look at this?

miri64 · 2026-01-27T17:11:42Z

Maybe @Teufelchen1 would like to take a look at this?

Ping @Teufelchen1 (or maybe @mguetschow)?

miri64 added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: tests Area: tests and testing framework Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR labels Mar 4, 2021

miri64 requested review from aabadie and leandrolanzieri March 4, 2021 10:43

miri64 changed the title ~~Embunit/enh/improvements~~ embunit / unittests: several improvements Mar 4, 2021

miri64 requested review from MichelRottleuthner, fjmolinas and smlng as code owners March 4, 2021 10:44

miri64 added this to the Release 2021.04 milestone Mar 4, 2021

MrKevinWeiss self-requested a review March 5, 2021 16:29

kaspar030 modified the milestones: Release 2021.04, Release 2021.07 Apr 23, 2021

MrKevinWeiss removed this from the Release 2021.07 milestone Jul 15, 2021

benpicco added CI: run tests If set, CI server will run tests on hardware for the labeled PR CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Aug 21, 2021

miri64 added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Sep 3, 2021

miri64 removed the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Sep 3, 2021

benpicco added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Sep 3, 2021

miri64 added 5 commits September 3, 2021 15:18

embunit: Use Outputter for all embunit tests

3585aba

This allows for use of OUTPUT=XML in all tests using embunit, not just `tests/unittests`. The other output formats work without the outputter being explicitly set.

embunit/ColorOutputter: properly reset colors and styles after tests

45ca1df

testrunner: try to guess OUTPUT format of unittests and act accordingly

fa5798c

tests: use run_check_unittests() where appropriate

f47b125

tests/gnrc_lorawan: fix output definition for test

d4c624e

benpicco added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Sep 9, 2021

miri64 added 2 commits September 9, 2021 17:36

tests/gnrc_ipv6_ext_frag: fix output definition for test

3621508

tests/gnrc_rpl_srh: fix output definition for test

aa3638a

fixup! tests: use run_check_unittests() where appropriate

2e2b91f

fixup! tests: use run_check_unittests() where appropriate

e682a64

miri64 force-pushed the embunit/enh/improvements branch from 99713f2 to e682a64 Compare September 9, 2021 16:01

miri64 added the State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet label Sep 13, 2021

stale bot added the State: stale State: The issue / PR has no activity for >185 days label Apr 16, 2022

miri64 removed the State: stale State: The issue / PR has no activity for >185 days label Apr 16, 2022

stale bot added the State: stale State: The issue / PR has no activity for >185 days label Nov 2, 2022

maribu added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 2, 2022

stale bot removed the State: stale State: The issue / PR has no activity for >185 days label Nov 2, 2022

miri64 added the State: waiting for author State: Action by the author of the PR is required label Nov 2, 2022

crasbe added the State: needs rebase State: The codebase was changed since the creation of the PR, making a rebase necessary label Jan 28, 2026

Teufelchen1 mentioned this pull request Feb 24, 2026

tests: Use common run_check_unittest #22102

Merged

crasbe added the State: Pick me up plz 🥺 This issue/PR is still relevant but won't be moved forward by the original author anymore, label Feb 24, 2026

Conversation

miri64 commented Mar 4, 2021

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

miri64 commented Mar 5, 2021

Uh oh!

benpicco commented Sep 2, 2021

Uh oh!

miri64 commented Sep 3, 2021

Uh oh!

miri64 commented Sep 3, 2021

Uh oh!

miri64 commented Sep 3, 2021

Uh oh!

benpicco commented Sep 9, 2021

Uh oh!

miri64 commented Sep 9, 2021

Uh oh!

miri64 commented Sep 9, 2021

Uh oh!

miri64 commented Sep 9, 2021

Uh oh!

miri64 commented Sep 9, 2021

Uh oh!

miri64 commented Sep 13, 2021

Uh oh!

stale bot commented Apr 16, 2022

Uh oh!

benpicco commented Apr 16, 2022

Uh oh!

miri64 commented Apr 16, 2022

Uh oh!

stale bot commented Nov 2, 2022

Uh oh!

MrKevinWeiss commented Nov 2, 2022

Uh oh!

miri64 commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants