testing: introduce RobotFramework tests by smlng · Pull Request #10095 · RIOT-OS/RIOT

smlng · 2018-10-01T20:16:10Z

Contribution description

This introduces the RobotFramework for writing and running RIOT test, it ships with some adapted tests. Integration into the build system is provided by a new make target: robot-test. Also adapted
the static tests, just call make robot-static-test.

Testing procedure

Simply build and execute a robot test, e.g. by calling:

make -C tests/bench_sizeof_coretypes all robot-test

tested on various boards such as arduino-mega2560, pba-d-01-kw2x, samr21-xpro and native.

Issues/PRs references

currently depends on #10038 (maybe adapted, see discussion there)

MrKevinWeiss · 2018-10-11T14:05:44Z

Can you somehow add a tag into the output xml that contains the board, test, and commit hash?

smlng · 2018-10-11T14:51:14Z

in theory you "know" all this because you invoke it like BOARD=something make -C tests/test-app flash robot-test, but I guess you would like to see it in the HTML output, right?

smlng · 2018-10-11T15:06:47Z

@MrKevinWeiss I removed the robot-open bc it doesn't work on Linux as expected. I also added some logging output including all the stuff you wanted.

MrKevinWeiss · 2018-10-12T07:40:14Z

in theory you "know" all this because you invoke it like BOARD=something make -C tests/test-app flash robot-test, but I guess you would like to see it in the HTML output, right?

More useful in the xml output. If it is possible that it just makes everything easier and more modular. Lets say we feed the output.xml to a data agitator, then we don't need anything special to pass or parse, just the xml.

smlng · 2018-10-12T12:39:54Z

@MrKevinWeiss please try again I added the output you wanted

smlng · 2018-10-12T12:40:05Z

its now part of the suite setup

MrKevinWeiss · 2018-10-12T13:28:40Z

its now part of the suite setup

Perfect, should be very easy to integrate!

cladmi

As said several times before, I do not like the format of the .robot tests descriptions. I do not find them really clear and prefer the python tests. They could also be improved and refactored without changing the whole format.

However I like the direction of producing test results files (xunit) instead of only 0 and 1 output but it is not specific to robotframework.

It is personal opinion and I put a request changes so other maintainers/devs are asked for feedback on the format. If it is only me, it is my problem and will kick me in the butt to use it.

I also have other issues with the current integration and behavior but I do not want to nitpick on it before talking about the format.

tcschmidt · 2018-10-15T16:04:13Z

@cladmi are you just saying that you don't like the robot way of describing tests, i.e., the framework? Or do you propose to express tests differently/more clearly in the robot keywords?

tcschmidt · 2018-10-15T17:38:25Z

@cladmi I understand your concern. On the other hand, we are comparing apples with oranges: In the specific example, the python test is a very static, fixed example, while the robot test works dynamically for changing configurations.
All in all, python is a generic scripting language that most people are familiar with. So easy to approach, but things must be handcrafted individually. The robot framework is specifically designed for testing and solves many generic problems ... but is specific. However, robot keywords may be chosen according to the domain context and 'intuition'.
It should also be noted that robot is well established with a large community of users and developers ... so nothing out of a weird niche.

kaspar030 · 2018-10-15T18:40:26Z

it ships with some adapted tests.

Did you try modeling our release specs with the robot framework?

I'm curious how compilation/flashing/handling of multiple nodes would look like. I remember finding it not trivial to implement a generic way to e.g., ping the link-local address of the 802154 interface of node A from node B without knowing it beforehand, or testing node A with examples/gnrc_networking compiled with "GNRC_SETTING_FOO=whiskybar", node B with "GNRC_SETTING_FOO=milkybar".

This introduces a new environment variable for a common directory that holds all output of the build process, such as application or package binaries. This would also allow to easily redirect output to any other location, e.g. for out-of-source builds.

tcschmidt · 2018-10-25T15:15:36Z

Now that the issue #10241 is out, the discussion about the general beauty and usefulness of Robot FW as well as its customization should go there.
This PR is currently only contributing the hook for the (urgently needed) HIL testing suite, which is done with Robot for simplicity. If the discussion in #10241 will lead to a different testing tool, we can switch with the HIL testing later.
@cladmi please review the technical appropriateness here, only, and take further aspects to #10241.

kaspar030 · 2018-10-25T15:42:58Z

The current build system just executes every executable in the tests/ folder.
Can the robot scripts be executed like that? Why is a seperate make target necessary?

smlng · 2018-10-29T13:36:28Z

The current build system just executes every executable in the tests/ folder.
Can the robot scripts be executed like that? Why is a separate make target necessary?

TL;DR

no it is not possible to make the robot scripts executable.

Longer answer

1.) making the robot test scripts executable will not work, as it is necessary to run them either with robot binary or python -m robot.run. Further, several parameters are necessary, e.g. where to find libraries and resource files, but also to set output directory and so on.

2.) the extra make target is not required and the tests could be integrated into the current make test target ... that would even be possible without making them executable. However, the idea was to make this first introduction of RF tests non-intrusive which also means not to interfere with the testing currently done by Murdock. So it is more a precaution for now and make robot tests available to users without breaking existing stuff. If this works out and we are happy with RF, we can integrate with make test and remove robot-test target.

miri64 · 2018-10-29T13:48:28Z

1.) making the robot test scripts executable will not work, as it is necessary to run them either with robot binary or python -m robot.run. Further, several parameters are necessary, e.g. where to find libraries and resource files, but also to set output directory and so on.

Contrary to what you probably wanted to say this sounds to me that you can make them executable, as they are just scripts to the robot binary / robot module. Have you tried setting a she-bang in line one and setting the x flag?

MrKevinWeiss · 2018-11-29T10:41:24Z

@cladmi Any update on the blocking comment? It seems some progress has been made on the RFC.

This introduces a new way to write RIOT tests using the RobotFramework, starting by adding new make targets for easy usage.

This adds a couple of tests already ported to RobotFramework.

Add simple wrapper to run static-tests using RobotFramework, this adds some nice HTML-output for free.

MrKevinWeiss

I couldn't find too much. A few recommendations on the keyword names but I think generally they are fine.

MrKevinWeiss · 2018-12-12T09:53:32Z

dist/robotframework/res/riot_util.keywords.txt

+    ${count} =          Get Number of Lines Matching Pattern  ${pattern}
+    Should Be Equal As Integers  ${count}  ${number}
+
+Verify Lines Matching Pattern Greater


Keyword recommendations (though it is pretty long):
Number of Lines Matching Pattern Should Equal
Number of Lines Matching Pattern Should Be Greater
Number of Lines Matching Pattern Should Be Less

As stated elsewhere (can't find it right now): I would prefer if don't re-implementing pexpect in RobotFramework keywords. As far as I understand it should be possible to have a thin wrapper around pexpect with RobotFramework. This would give us a far more stable and future proof implementation.

I explained my point there instead: #10241 (comment)

smlng · 2019-01-22T12:57:11Z

superseded by #10843

smlng requested a review from cladmi October 1, 2018 20:16

smlng mentioned this pull request Oct 5, 2018

make: introduce common BUILD_DIR #10038

Merged

miri64 added the State: waiting for other PR label Oct 5, 2018

smlng force-pushed the pr/robot/init branch 4 times, most recently from e48c1fb to 9129271 Compare October 11, 2018 07:55

smlng requested a review from MrKevinWeiss October 11, 2018 09:18

smlng force-pushed the pr/robot/init branch 3 times, most recently from f4b4e30 to 3421c9d Compare October 11, 2018 09:51

smlng mentioned this pull request Oct 11, 2018

testing: add periph I2C robot tests for HIL #10147

Closed

smlng force-pushed the pr/robot/init branch from 66adfde to 39884c6 Compare October 12, 2018 12:39

smlng force-pushed the pr/robot/init branch 2 times, most recently from e10c53b to 675ad08 Compare October 12, 2018 13:24

miri64 removed the State: waiting for other PR label Oct 12, 2018

smlng force-pushed the pr/robot/init branch from 675ad08 to b92ab76 Compare October 15, 2018 13:48

cladmi previously requested changes Oct 15, 2018

View reviewed changes

smlng force-pushed the pr/robot/init branch 4 times, most recently from fc395d3 to 45f1a91 Compare October 24, 2018 11:53

make: introduce BUILDOUT_DIR

d625b57

This introduces a new environment variable for a common directory that holds all output of the build process, such as application or package binaries. This would also allow to easily redirect output to any other location, e.g. for out-of-source builds.

smlng force-pushed the pr/robot/init branch from 45f1a91 to 0bcd83c Compare October 24, 2018 12:02

smlng mentioned this pull request Oct 24, 2018

RFC: Use RobotFramework to write RIOT tests #10241

Closed

smlng force-pushed the pr/robot/init branch 2 times, most recently from 284e8fc to 2b31819 Compare October 26, 2018 08:09

kb2ma mentioned this pull request Nov 13, 2018

[RFC/WIP]: Use pytest to run CoAP tasks RIOT-OS/Release-Specs#86

Closed

MrKevinWeiss mentioned this pull request Nov 21, 2018

tests/periph_i2c: Example i2c test using pytest #10448

Closed

smlng added 4 commits December 11, 2018 16:33

tests: robotframework initial integration

3d78ade

This introduces a new way to write RIOT tests using the RobotFramework, starting by adding new make targets for easy usage.

tests: add/adapt tests for RobotFramework

658f92c

This adds a couple of tests already ported to RobotFramework.

tests: adapt static-tests to RobotFramework

d4eee76

Add simple wrapper to run static-tests using RobotFramework, this adds some nice HTML-output for free.

robotframework: add README.md

48a3416

smlng force-pushed the pr/robot/init branch from 2b31819 to 48a3416 Compare December 11, 2018 15:41

MrKevinWeiss reviewed Dec 12, 2018

View reviewed changes

This was referenced Jan 15, 2019

tests: I2C HIL tests with RobotFramework #10774

Closed

tests: init RobotFramework and add basic xtimer test #10843

Closed

smlng closed this Jan 22, 2019

smlng deleted the pr/robot/init branch June 25, 2019 08:41

Conversation

smlng commented Oct 1, 2018

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

MrKevinWeiss commented Oct 11, 2018

Uh oh!

smlng commented Oct 11, 2018

Uh oh!

smlng commented Oct 11, 2018

Uh oh!

MrKevinWeiss commented Oct 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smlng commented Oct 12, 2018

Uh oh!

smlng commented Oct 12, 2018

Uh oh!

MrKevinWeiss commented Oct 12, 2018

Uh oh!

cladmi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tcschmidt commented Oct 15, 2018

Uh oh!

tcschmidt commented Oct 15, 2018

Uh oh!

kaspar030 commented Oct 15, 2018

Uh oh!

tcschmidt commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaspar030 commented Oct 25, 2018

Uh oh!

smlng commented Oct 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Longer answer

Uh oh!

miri64 commented Oct 29, 2018

Uh oh!

MrKevinWeiss commented Nov 29, 2018

Uh oh!

MrKevinWeiss left a comment

Choose a reason for hiding this comment

Uh oh!

MrKevinWeiss Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

miri64 Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

smlng commented Jan 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MrKevinWeiss commented Oct 12, 2018 •

edited

Loading

cladmi left a comment •

edited

Loading

tcschmidt commented Oct 25, 2018 •

edited

Loading

smlng commented Oct 29, 2018 •

edited

Loading