Skip to content

tests/test_tools: add a test for the testing tools environment#12072

Merged
MrKevinWeiss merged 1 commit intoRIOT-OS:masterfrom
cladmi:pr/test_tools/shell_no_echo
Aug 26, 2019
Merged

tests/test_tools: add a test for the testing tools environment#12072
MrKevinWeiss merged 1 commit intoRIOT-OS:masterfrom
cladmi:pr/test_tools/shell_no_echo

Conversation

@cladmi
Copy link
Contributor

@cladmi cladmi commented Aug 23, 2019

Contribution description

Check the interaction with a board through make term.
It is using a shell without echo or prompt for reference.

It currently checks that there is no local echo.

Now that the issue in testrunner and the RIOT_TERMINAL is fixed, it is more a sanity check/regression test to verify your environment or a board configuration does not mess up with what is required for testing.

Testing procedure

Run BOARD=your_board make -C tests/test_tools/ flash test with different RIOT_TERMINAL, currently supported (not on all boards) pyterm, socat, picocom.

The test should succeed in all cases.

RIOT_TERMINAL=pyterm samr21-xpro
RIOT_TERMINAL=pyterm  BOARD=samr21-xpro RIOT_CI_BUILD=1 make --no-print-directory -C tests/test_tools/ flash test
Building application "tests_test_tools" for "samr21-xpro" with MCU "samd21".

   text    data     bss     dec     hex filename
   9144     140    2612   11896    2e78 /home/harter/work/git/RIOT/tests/test_tools/bin/samr21-xpro/tests_test_tools.elf
/home/harter/work/git/RIOT/dist/tools/edbg/edbg  -t atmel_cm0p -b -v -p -f /home/harter/work/git/RIOT/tests/test_tools/bin/samr21-xpro/tests_test_tools.bin
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800004678 01.1A.00FB (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Programming........................................ done.
Verification........................................ done.
/home/harter/work/git/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200"
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2019-08-23 18:08:58,892 - INFO # Connect to serial port /dev/ttyACM0
Welcome to pyterm!
Type '/exit' to exit.
2019-08-23 18:09:02,136 - INFO # main(): This is RIOT! (Version: buildtest)
2019-08-23 18:09:02,139 - INFO # Running 'tests_tools' application
shellping
2019-08-23 18:09:02,192 - INFO # shellpong
true this should not be echoed
shellping
2019-08-23 18:09:03,297 - INFO # shellpong
RIOT_TERMINAL=picocom samr21-xpro
RIOT_TERMINAL=picocom BOARD=samr21-xpro RIOT_CI_BUILD=1 make --no-print-directory -C tests/test_tools/ flash test
Building application "tests_test_tools" for "samr21-xpro" with MCU "samd21".

   text    data     bss     dec     hex filename
   9144     140    2612   11896    2e78 /home/harter/work/git/RIOT/tests/test_tools/bin/samr21-xpro/tests_test_tools.elf
/home/harter/work/git/RIOT/dist/tools/edbg/edbg  -t atmel_cm0p -b -v -p -f /home/harter/work/git/RIOT/tests/test_tools/bin/samr21-xpro/tests_test_tools.bin
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800004678 01.1A.00FB (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Programming........................................ done.
Verification........................................ done.
picocom --nolock --imap lfcrlf --baud "115200" "/dev/ttyACM0"
picocom v2.2

port is        : /dev/ttyACM0
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
nolock is      : yes
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        : lfcrlf,
omap is        : 
emap is        : crcrlf,delbs,

Type [C-a] [C-h] to see available commands

Terminal ready
main(): This is RIOT! (Version: buildtest)
Running 'tests_tools' applicationshellping

shellpong
true this should not be echoed
shellping
shellpong

RIOT_TERMINAL=socat samr21-xpro
RIOT_TERMINAL=socat BOARD=samr21-xpro RIOT_CI_BUILD=1 make --no-print-directory -C tests/test_tools/ flash test
Building application "tests_test_tools" for "samr21-xpro" with MCU "samd21".

   text    data     bss     dec     hex filename
   9144     140    2612   11896    2e78 /home/harter/work/git/RIOT/tests/test_tools/bin/samr21-xpro/tests_test_tools.elf
/home/harter/work/git/RIOT/dist/tools/edbg/edbg  -t atmel_cm0p -b -v -p -f /home/harter/work/git/RIOT/tests/test_tools/bin/samr21-xpro/tests_test_tools.bin
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800004678 01.1A.00FB (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Programming........................................ done.
Verification........................................ done.
socat - open:/dev/ttyACM0,b115200,echo=0,raw
main(): This is RIOT! (Version: buildtest)
Running 'tests_tools' applicationshellping

shellpong
true this should not be echoed
shellping
shellpong

native
RIOT_CI_BUILD=1 BOARD=native make --no-print-directory -C tests/test_tools/ flash test
Building application "tests_test_tools" for "native" with MCU "native".

   text    data     bss     dec     hex filename
  22506     628   47652   70786   11482 /home/harter/work/git/RIOT/tests/test_tools/bin/native/tests_test_tools.elf
true 
/home/harter/work/git/RIOT/tests/test_tools/bin/native/tests_test_tools.elf  
RIOT native interrupts/signals initialized.
LED_RED_OFF
LED_GREEN_ON
RIOT native board initialized.
RIOT native hardware initialization complete.

main(): This is RIOT! (Version: buildtest)
Running 'tests_tools' application
shellping
shellpong
true this should not be echoed
shellping
shellpong

On IoT-LAB it works too
IOTLAB_NODE=auto-ssh RIOT_CI_BUILD=1 BOARD=iotlab-m3 make --no-print-directory -C tests/test_tools/ flash test
Building application "tests_test_tools" for "iotlab-m3" with MCU "stm32f1".

   text    data     bss     dec     hex filename
   8800     140    2620   11560    2d28 /home/harter/work/git/RIOT/tests/test_tools/bin/iotlab-m3/tests_test_tools.elf
iotlab-node --jmespath='keys(@)[0]' --format='int'  --list grenoble,m3,58 --update /home/harter/work/git/RIOT/tests/test_tools/bin/iotlab-m3/tests_test_tools.elf | grep 0
0
ssh -t harter@grenoble.iot-lab.info 'socat - tcp:m3-58.grenoble.iot-lab.info:20000'
�main(): This is RIOT! (Version: buildtest)
Running 'tests_tools' application
�main(): This is RIOT! (Version: buildtest)
Running 'tests_tools' application
shellping
shellpong
true this should not be echoed
shellping
shellpong

Issues/PRs references

Split out of

Check the interaction with a board through make term.
It is using a shell without echo or prompt for reference.

It currently checks that there is no local echo.
@jcarrano jcarrano added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Area: build system Area: Build system Area: tests Area: tests and testing framework CI: run tests If set, CI server will run tests on hardware for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Aug 23, 2019
@tcschmidt tcschmidt requested a review from smlng August 24, 2019 13:20
@MrKevinWeiss
Copy link
Contributor

I did some testing and here are my results.

RIOT_TERMINAL=pyterm nucleo-f103rb
2019-08-26 10:58:49,372 - INFO # main(): This is RIOT! (Version: 2019.10-devel-487-gb3cb-tstxxx/12072)
2019-08-26 10:58:49,375 - INFO # Running 'tests_tools' application
shellping
2019-08-26 10:58:49,430 - INFO # shellpong
true this should not be echoed
shellping
2019-08-26 10:58:50,536 - INFO # shellpong
RIOT_TERMINAL=socat nucleo-f103rb
shellping
main(): This is RIOT! (Version: 2019.10-devel-487-gb3cb-tstxxx/12072)
Running 'tests_tools' application
shellpong
true this should not be echoed
shellping
shellpong
RIOT_TERMINAL=picocom nucleo-f103rb
Terminal ready
�main(): This is RIOT! (Version: 2019.10-devel-487-gb3cb-tstxxx/12072)
Running 'tests_tools' application
shellping
shellpong
true this should not be echoed
shellping
shellpong

@MrKevinWeiss MrKevinWeiss added the Reviewed: 1-fundamentals The fundamentals of the PR were reviewed according to the maintainer guidelines label Aug 26, 2019
@MrKevinWeiss
Copy link
Contributor

Fundamentals are sound. We should be testing our tools.

@MrKevinWeiss
Copy link
Contributor

It seems like all the this should not be echoed are failing but the assert doesn't fail the test?

@cladmi
Copy link
Contributor Author

cladmi commented Aug 26, 2019

@MrKevinWeiss What you see is the text sent by pexpect, not a local echo.
And it is indeed not echoed as we see it only once.
I could reword to something like "this must be printed only once", or "this must not be matched in test"

You can check what it is testing with the following diff:

With `pyterm`, when `pexpect` has local echo capturing we get the double output
BOARD=samr21-xpro RIOT_CI_BUILD=1 make --no-print-directory -C tests/test_tools/ flash test

2019-08-26 11:50:59,575 - INFO # Running 'tests_tools' application
shellping
shellping
2019-08-26 11:50:59,629 - INFO # shellpong
true this should not be echoed
true this should not be echoed

Traceback (most recent call last):
  File "/home/harter/work/git/RIOT/tests/test_tools/tests/01-run.py", line 60, in <module>
    sys.exit(run(testfunc))
  File "/home/harter/work/git/RIOT/dist/pythonlibs/testrunner/__init__.py", line 29, in run
    testfunc(child)
  File "/home/harter/work/git/RIOT/tests/test_tools/tests/01-run.py", line 53, in testfunc
    _test_no_local_echo(child)
  File "/home/harter/work/git/RIOT/tests/test_tools/tests/01-run.py", line 38, in _test_no_local_echo
    assert res == 0, "There should have been a timeout and not match stdin"
AssertionError: There should have been a timeout and not match stdin
/home/harter/work/git/RIOT/tests/test_tools/../../Makefile.include:605: recipe for target 'test' failed
make: *** [test] Error 1
diff --git a/dist/pythonlibs/testrunner/spawn.py b/dist/pythonlibs/testrunner/spawn.py
index 84164d1a1..a7c03e22e 100644
--- a/dist/pythonlibs/testrunner/spawn.py
+++ b/dist/pythonlibs/testrunner/spawn.py
@@ -37,7 +37,7 @@ def find_exc_origin(exc_info):

 def setup_child(timeout=10, spawnclass=pexpect.spawnu, env=None, logfile=None):
     child = spawnclass("make term", env=env, timeout=timeout,
-                       codec_errors='replace', echo=False)
+                       codec_errors='replace', echo=True)

     # on many platforms, the termprog needs a short while to be ready...
     time.sleep(MAKE_TERM_STARTED_DELAY)
When picocom has local `--echo` enabled, we get the match
RIOT_TERMINAL=picocom BOARD=samr21-xpro RIOT_CI_BUILD=1 make --no-print-directory -C tests/test_tools/ flash test

Terminal ready
shell: command not found: not
main(): This is RIOT! (Version: buildtest)
Running 'tests_tools' application
shellping
shellping
shellpong
true this should not be echoed
true this should not be echoed

Traceback (most recent call last):
  File "/home/harter/work/git/RIOT/tests/test_tools/tests/01-run.py", line 60, in <module>
    sys.exit(run(testfunc))
  File "/home/harter/work/git/RIOT/dist/pythonlibs/testrunner/__init__.py", line 29, in run
    testfunc(child)
  File "/home/harter/work/git/RIOT/tests/test_tools/tests/01-run.py", line 53, in testfunc
    _test_no_local_echo(child)
  File "/home/harter/work/git/RIOT/tests/test_tools/tests/01-run.py", line 38, in _test_no_local_echo
    assert res == 0, "There should have been a timeout and not match stdin"
AssertionError: There should have been a timeout and not match stdin
/home/harter/work/git/RIOT/tests/test_tools/../../Makefile.include:605: recipe for target 'test' failed
make: *** [test] Error 1
diff --git a/makefiles/tools/serial.inc.mk b/makefiles/tools/serial.inc.mk
index 8abcf7fe1..45f747343 100644
--- a/makefiles/tools/serial.inc.mk
+++ b/makefiles/tools/serial.inc.mk
@@ -22,5 +22,5 @@ else ifeq ($(RIOT_TERMINAL),socat)
   TERMFLAGS ?= $(SOCAT_OUTPUT) open:$(PORT),b$(BAUD),echo=0,raw
 else ifeq ($(RIOT_TERMINAL),picocom)
   TERMPROG  ?= picocom
-  TERMFLAGS ?= --nolock --imap lfcrlf --baud "$(BAUD)" "$(PORT)"
+  TERMFLAGS ?= --echo --nolock --imap lfcrlf --baud "$(BAUD)" "$(PORT)"
 endif

I am not sure why I do not get 3 match when using both patch with picocom though.

Copy link
Contributor

@MrKevinWeiss MrKevinWeiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! ACK!

@MrKevinWeiss MrKevinWeiss added Reviewed: 2-code-design The code design of the PR was reviewed according to the maintainer guidelines Reviewed: 3-testing The PR was tested according to the maintainer guidelines Reviewed: 4-code-style The adherence to coding conventions by the PR were reviewed according to the maintainer guidelines Reviewed: 5-documentation The documentation details of the PR were reviewed according to the maintainer guidelines labels Aug 26, 2019
@MrKevinWeiss MrKevinWeiss merged commit b9a0143 into RIOT-OS:master Aug 26, 2019
@cladmi
Copy link
Contributor Author

cladmi commented Aug 26, 2019

Thank you for the review.

@cladmi cladmi deleted the pr/test_tools/shell_no_echo branch August 26, 2019 15:20
@kb2ma kb2ma added this to the Release 2019.10 milestone Sep 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: build system Area: Build system Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: run tests If set, CI server will run tests on hardware for the labeled PR Reviewed: 1-fundamentals The fundamentals of the PR were reviewed according to the maintainer guidelines Reviewed: 2-code-design The code design of the PR was reviewed according to the maintainer guidelines Reviewed: 3-testing The PR was tested according to the maintainer guidelines Reviewed: 4-code-style The adherence to coding conventions by the PR were reviewed according to the maintainer guidelines Reviewed: 5-documentation The documentation details of the PR were reviewed according to the maintainer guidelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants