testing: add periph I2C robot tests for HIL by smlng · Pull Request #10147 · RIOT-OS/RIOT

smlng · 2018-10-11T10:03:24Z

Contribution description

This adds a new test suite for the I2C peripheral drivers based on the RobotFramework and PHiLIP to allow for hardware-in-the-loop (HIL) testing.

Testing procedure

To run this you need a board (device under test) hooked up to a PHILIP test board, see https://github.com/riot-appstore/PHiLIP . Then run BOARD=<fill in> make -C tests/periph_i2c flash robot-test.

Issues/PRs references

depends on #10095

MrKevinWeiss · 2018-10-11T13:36:34Z

I ran the test on the nrf52dk (that supports reset on p21) with PHiLIP-N and all tests passed but make robot-open seems to have some problems with it.
When I tried everything together I received:
open: Unable to open /dev/tty2: Permission denied

When I tried with sudo and just robot-open I got:
objcopy not found. Hex file will not be created.

smlng · 2018-10-11T13:50:05Z

I think I remove the robot-open target, it was intended as a short-hand to open the generated html output and on macOS it works nicely, but Linux seems to have a different meaning of open.

This introduces a new environment variable for a common directory that holds all output of the build process, such as application or package binaries. This would also allow to easily redirect output to any other location, e.g. for out-of-source builds.

MrKevinWeiss · 2018-11-07T16:05:35Z

tests/periph_i2c/tests/periph_i2c.keywords.txt

+
+*** Keywords ***
+DUT Must Have Periph I2C Application
+    API Call Should Succeed     I2C Get ID


To prevent initial error from connecting and resetting I think it would be good to send the I2C Get ID command once without any checks first, then send and check the id. Either include it here or do it before calling this keyword.

okay, will add a simple I2C Get ID without checking the result before this line

MrKevinWeiss · 2018-11-07T16:07:22Z

tests/periph_i2c/tests/02__periph_i2c_write_register.robot

+
+Suite Setup         Run Keywords    Reset DUT and PHILIP
+...                                 DUT Must Have Periph I2C Application
+Test Setup          Reset DUT and PHILIP


I think each test should include a DUT Must Have Periph I2C Application or at least send the I2C Get ID without checks. Maybe make a clear buffers command that does that.

so basically Test Setup should be the same as Suite Setup

MrKevinWeiss · 2018-11-07T16:24:58Z

Also I think it would be good for all data driven tests to have a acquire and release around it (maybe this can be done with the setup and teardown)

MrKevinWeiss · 2018-11-07T16:30:27Z

also add the riot_pal version to the metadata

MrKevinWeiss · 2018-11-07T16:44:12Z

... Also grep for verfiy, I think there are a few places that this occurs.

MrKevinWeiss · 2018-11-07T16:51:01Z

hmmm, my suggestions still don't solve the remote-revb problems. Maybe a retry command n times would be better. Sometimes it takes 2 timeouts after the reset to recover.

MrKevinWeiss · 2018-11-07T17:05:32Z

Here is my brute force solution for the remote-revb problems (but this should also help many other initial connection problems

--- a/tests/periph_i2c/tests/01__periph_i2c_base.robot
+++ b/tests/periph_i2c/tests/01__periph_i2c_base.robot
@@ -2,8 +2,13 @@
 Documentation       Basic tests to verify functionality of the periph I2C API.

 Suite Setup         Run Keywords    Reset DUT and PHILIP
+...                                 I2C Get ID
+...                                 I2C Get ID
 ...                                 DUT Must Have Periph I2C Application
-Test Setup          Reset DUT and PHILIP
+Test Setup          Run Keywords    Reset DUT and PHILIP
+...                                 I2C Get ID
+...                                 I2C Get ID
+...                                 I2C Get ID

 Resource            periph_i2c.keywords.txt
 Resource            api_shell.keywords.txt
diff --git a/tests/periph_i2c/tests/02__periph_i2c_write_register.robot b/tests/periph_i2c/tests/02__periph_i2c_write_register.robot
index 206013009..c73658cb1 100644
--- a/tests/periph_i2c/tests/02__periph_i2c_write_register.robot
+++ b/tests/periph_i2c/tests/02__periph_i2c_write_register.robot
@@ -2,8 +2,14 @@
 Documentation       Data driven tests to verify writing to a valid register.

 Suite Setup         Run Keywords    Reset DUT and PHILIP
+...                                 I2C Get ID
+...                                 I2C Get ID
 ...                                 DUT Must Have Periph I2C Application
-Test Setup          Reset DUT and PHILIP
+Test Setup          Run Keywords    Reset DUT and PHILIP
+...                                 I2C Get ID
+...                                 I2C Get ID
+...                                 I2C Get ID
+
 Test Template       I2C Write Bytes To Register Should Succeed

 Resource            periph_i2c.keywords.txt

==============================================================================
tests periph i2c ON remote-revb
==============================================================================
tests periph i2c ON remote-revb.Periph I2C Base :: Basic tests to verify fu...
==============================================================================
Acquire and Release Should Succeed :: Verify I2C acquire and relea... | PASS |
------------------------------------------------------------------------------
Acquire after Release Should Succeed :: Verify acquiring an I2C bu... | PASS |
------------------------------------------------------------------------------
Double Acquire Should Timeout :: Verify that acquiring a locked I2... | PASS |
------------------------------------------------------------------------------
Unacquired Read Register Should Succeed :: Verfiy reading a regist... | PASS |
------------------------------------------------------------------------------
Read Register After Release Should Error :: Verfiy reading a regis... | FAIL |
'Success' does not contain 'Error'
------------------------------------------------------------------------------
Read Register After NACK Should Succeed :: Verify recovery of I2C ... | PASS |
------------------------------------------------------------------------------
tests periph i2c ON remote-revb.Periph I2C Base :: Basic tests to ... | FAIL |
6 critical tests, 5 passed, 1 failed
6 tests total, 5 passed, 1 failed
==============================================================================
tests periph i2c ON remote-revb.Periph I2C Write Register :: Data driven te...
==============================================================================
One Byte Should Succeed                                               | PASS |
------------------------------------------------------------------------------
Two Bytes Should Succeed                                              | PASS |
------------------------------------------------------------------------------
Ten Bytes Should Succeed                                              | PASS |
------------------------------------------------------------------------------
tests periph i2c ON remote-revb.Periph I2C Write Register :: Data ... | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
tests periph i2c ON remote-revb                                       | FAIL |
9 critical tests, 8 passed, 1 failed
9 tests total, 8 passed, 1 failed
==============================================================================

smlng · 2018-11-08T08:47:39Z

Also I think it would be good for all data driven tests to have a acquire and release around it (maybe this can be done with the setup and teardown)

yeah Test Setup and Test Teardown would be the best way, so we do not clutter the actual test

smlng · 2018-11-08T08:48:24Z

... Also grep for verfiy, I think there are a few places that this occurs.

what do you mean?

smlng · 2018-11-08T08:50:00Z

hmmm, my suggestions still don't solve the remote-revb problems. Maybe a retry command n times would be better. Sometimes it takes 2 timeouts after the reset to recover.

you mean I2C Get ID should be called multiple times, before running the test? I guess then it would be better to have something like: retry as long as it times out, but max n times

MrKevinWeiss · 2018-11-12T10:53:21Z

Very nice. I like the changes and it seems to improve the non i2c related issues.

This introduces a new way to write RIOT tests using the RobotFramework, starting by adding new make targets for easy usage.

This adds a couple of tests already ported to RobotFramework.

Add simple wrapper to run static-tests using RobotFramework, this adds some nice HTML-output for free.

Add initial support for the periph_i2c test for the RobotFramework.

kaspar030 · 2018-12-12T10:52:19Z

Would you mind creating a version that does not depend on robot? No need to stall HIL testing for a framework discussion.

tcschmidt · 2018-12-12T11:01:38Z

Would you mind creating a version that does not depend on robot? No need to stall HIL testing for a framework discussion.

No issue here: HIL testing is deployed and working. No need to intermingle with the asynchronous framework discussion.

kaspar030 · 2018-12-12T11:06:59Z

HIL testing is deployed and working. No need to intermingle with the asynchronous framework discussion.

Then we can close this?

MrKevinWeiss · 2018-12-12T11:11:14Z

@kaspar030 please don't, at the moment we are using this for the HiL results in the nighties (and I have been using it for i2c improvements that can use some review 😉)

MrKevinWeiss · 2018-12-12T11:13:56Z

Would you mind creating a version that does not depend on robot?

Checkout the tests/periph_i2c/tests/test.py (it doesn't use test runner and was meant as an example of what should be tested but it is an independent test).

kaspar030 · 2018-12-12T11:14:21Z

please don't, at the moment we are using this for the HiL results in the nighties

I know, and I didn't intent to. I tried to clarify what @tcschmidt means by "this is deployed and working" vs. there's an open PR.

edit deleted accidental duplicate post

tcschmidt · 2018-12-12T11:22:38Z

@kaspar030 the point is simple: Framework discussion is asynchronous, nothing more.

smlng · 2019-01-17T20:10:20Z

closing in favour of #10774

smlng added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: tests Area: tests and testing framework Type: new feature The issue requests / The PR implemements a new feature for RIOT State: waiting for other PR labels Oct 11, 2018

smlng requested review from MrKevinWeiss and cladmi October 11, 2018 10:03

smlng force-pushed the pr/robot/i2c branch 3 times, most recently from 2f87601 to 02c4730 Compare October 12, 2018 13:26

miri64 removed the State: waiting for other PR label Oct 12, 2018

smlng force-pushed the pr/robot/i2c branch 4 times, most recently from 1d10d3a to 5717d91 Compare October 22, 2018 09:30

smlng force-pushed the pr/robot/i2c branch from 5717d91 to ad24ae4 Compare October 24, 2018 11:54

make: introduce BUILDOUT_DIR

d625b57

This introduces a new environment variable for a common directory that holds all output of the build process, such as application or package binaries. This would also allow to easily redirect output to any other location, e.g. for out-of-source builds.

smlng force-pushed the pr/robot/i2c branch from ad24ae4 to 01e6fa6 Compare October 24, 2018 12:02

smlng mentioned this pull request Oct 24, 2018

RFC: Use RobotFramework to write RIOT tests #10241

Closed

smlng force-pushed the pr/robot/i2c branch from a7815c3 to dae3ef3 Compare October 26, 2018 08:16

MrKevinWeiss reviewed Nov 7, 2018

View reviewed changes

smlng force-pushed the pr/robot/i2c branch from 4bcf765 to 722e771 Compare November 13, 2018 09:37

smlng force-pushed the pr/robot/i2c branch from 8a01839 to 2b66b9f Compare November 21, 2018 10:49

This was referenced Nov 21, 2018

tests/periph_i2c: Example i2c test using pytest #10448

Closed

tests/periph_uart: Add uart robot tests #10497

Closed

smlng added 5 commits December 11, 2018 16:33

tests: robotframework initial integration

3d78ade

This introduces a new way to write RIOT tests using the RobotFramework, starting by adding new make targets for easy usage.

tests: add/adapt tests for RobotFramework

658f92c

This adds a couple of tests already ported to RobotFramework.

tests: adapt static-tests to RobotFramework

d4eee76

Add simple wrapper to run static-tests using RobotFramework, this adds some nice HTML-output for free.

robotframework: add README.md

48a3416

tests: add periph_i2c robotframework test

2775815

Add initial support for the periph_i2c test for the RobotFramework.

smlng force-pushed the pr/robot/i2c branch from 50b47ca to 2775815 Compare December 11, 2018 15:44

smlng mentioned this pull request Jan 8, 2019

make: introduce common BUILD_DIR #10038

Merged

smlng mentioned this pull request Jan 15, 2019

tests: I2C HIL tests with RobotFramework #10774

Closed

smlng closed this Jan 17, 2019

smlng deleted the pr/robot/i2c branch June 25, 2019 08:41

Conversation

smlng commented Oct 11, 2018

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

MrKevinWeiss commented Oct 11, 2018

Uh oh!

smlng commented Oct 11, 2018

Uh oh!

MrKevinWeiss Nov 7, 2018

Choose a reason for hiding this comment

Uh oh!

smlng Nov 8, 2018

Choose a reason for hiding this comment

Uh oh!

MrKevinWeiss Nov 7, 2018

Choose a reason for hiding this comment

Uh oh!

smlng Nov 8, 2018

Choose a reason for hiding this comment

Uh oh!

MrKevinWeiss commented Nov 7, 2018

Uh oh!

MrKevinWeiss commented Nov 7, 2018

Uh oh!

MrKevinWeiss commented Nov 7, 2018

Uh oh!

MrKevinWeiss commented Nov 7, 2018

Uh oh!

MrKevinWeiss commented Nov 7, 2018

Uh oh!

smlng commented Nov 8, 2018

Uh oh!

smlng commented Nov 8, 2018

Uh oh!

smlng commented Nov 8, 2018

Uh oh!

MrKevinWeiss commented Nov 12, 2018

Uh oh!

kaspar030 commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tcschmidt commented Dec 12, 2018

Uh oh!

kaspar030 commented Dec 12, 2018

Uh oh!

MrKevinWeiss commented Dec 12, 2018

Uh oh!

MrKevinWeiss commented Dec 12, 2018

Uh oh!

kaspar030 commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tcschmidt commented Dec 12, 2018

Uh oh!

smlng commented Jan 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kaspar030 commented Dec 12, 2018 •

edited

Loading

kaspar030 commented Dec 12, 2018 •

edited

Loading