tools: add add_insufficient_memory_board.sh

[benpicco]

Contribution description

This adds a script to automatically add a new board to all Makefile.ci files.
To achieve this it builds all examples and tests and checks if a build failure due to insufficient memory occurred.

So to add a new board, you only have to do

dist/tools/insufficient_memory/add_insufficient_memory_board.sh my_tiny_board

Testing procedure

• Remove e.g. arduino-nano from Makefile.ci.
• run add_insufficient_memory_board.sh arduino-nano
• observe that the script added back the board to the Makefile.ci files.

Issues/PRs references

related to #12484 and #12406

[smlng]

sorry I don't see the overall benefit of this, also to me this is misuse of the build system.
But that's my subjective opinion, @cladmi: what do you think?

[benpicco]

I don't think keeping those lists sorted is a good allocation of maintainer time.
This should make adding new boards easier and prevent quick hacks to be added to Makefile.ci in the future. Probably a warning should be added that this is a generated file then.

[smlng]

How does a make target achieve this. If then add the test for alphabetic order into dist/tools/buildsystem_sanity_check/check.sh so travis and murdock complain about if someone adds a board in the wrong place or order, then the authors needs to fix that and maintainers can rest assured that boards are ordered correctly. The make target only eases adding a board but does not prevent manual editing of Makefile.ci breaking the order.

[benpicco]

if someone adds a board in the wrong place or order, then the authors needs to fix that

I'd like to avoid that step entirely.

The make target only eases adding a board but does not prevent manual editing of Makefile.ci breaking the order.

Maybe a warning not to edit that file by hand would help?
I can also add a make target for Makefile.ci that would just re-generate the file without any additions, but properly sorted.

[miri64]

Aren't there some scripts proposed already that could take care of reordering the list by being called in the Make function @benpicco introduces here?

[smlng]

see #6818

[benpicco]

As to why this is useful: It allows me to do

for i in */Makefile.ci; do
    test=$(dirname $i);
    if make BOARD=stm32f030f4-demo -j -C $test &> /dev/null ; then
        echo $test succeeds
    else
        echo $test fails
        make BOARD=stm32f030f4-demo -C $test add_insufficient_memory
    fi
done

(Ok, I can make that a script outside of the build system too)

[maribu]

see #6818

That sorting script is aiming to achieve a bit more. We intentionally made Makefile.ci machine readable by having one line per board, while #6818 tries to also sort human readable lists (with many items per line). I think we should not adopt this script for Makefile.ci in order to keep it one board per line, as there were to reasons for it:

1. To allows tools adding and sorting the list (which dist/tools: add sort-variables script #6818 would also achieve)
2. To reduce the number of merge conflicts (which a space efficient list that dist/tools: add sort-variables script #6818 generates would no longer achieve)

Of course, if the script in #6818 would be generalized to work with both flavors of type setting, my arguments become invalid.

[smlng]

@maribu I (strongly) agree with "one board per line" for reasons you mentioned.

Reference to #6818 was just to point out, that there is similar work already and also that it makes more sense to me to provide a tool/helper script for authors to automatically adapt such lists to match the desired output.

However, I'm still not in favour of having this as a make target, bc its tooling and should not (IMHO) be in the build system. If someone likes it that much they can make it a pre-commit hook in git config.

[smlng]

plus having a check (as already mentioned before) in static tests that verifies lists are ordered and one-board-per-line.

[maribu]

Running make help currently yields

Output of `make help`

all
archive-check
bindist
binfile
check-toolchain-supported
check_bindist
clean
clean-intermediates
clean-pkg
cleanterm
debug
debug-server
dependency-debug
desvirt-check
desvirt-check-topo-args
desvirt-check-topo-file
desvirt-clean
desvirt-define
desvirt-distclean
desvirt-list
desvirt-start
desvirt-stop
desvirt-topology
desvirt-undefine
distclean
doc
eclipsesym
elffile
emulate
flash
flash-only
flashfile
help
hexfile
info-boards
info-boards-supported
info-build
info-buildsize
info-buildsizes
info-concurrency
info-cpu
info-features-missing
info-features-provided
info-features-required
info-features-used
info-files
info-modules
info-objsize
info-toolchains-supported
link
list-ttys
mcuboot
mosquitto_rsmb
objdump
pkg-build
pkg-prepare
preflash
prepare_check_bindist
print-size
reset
riotboot
scan-build
scan-build-analyze
term
termdeps
test
test-input-hash
test-murdock
test-on-ci-enabled

That is quite a bunch of targets with a lot not relevant to most users. E.g. with the DES project being long concluded, is anyone still using make desvirt-*? Does it still work?

So my bottom line is: If there is something that is not generally useful in the daily work flow of a RIOT user, a RIOT contributor or a RIOT maintainer, I would like to have it separately from the standard build system.

@benpicco: Would you mind if the feature would go to dist/tools instead?

[benpicco]

@maribu sure, I can extend it then too.
A script that builds all tests and automatically adds the boards to BOARD_INSUFFICIENT_MEMORY if there is an overflows error would be handy to have.

[aabadie]

I think this feature can be useful in general and the integration in the build system is quite nice since it provides all useful informations: list of unsufficient memory boards + current board.

Maybe the specific handling for this could be moved in a dedicated make files, in makefiles/utils and included somewhere appropriate.

If there is something that is not generally useful in the daily work flow of a RIOT user, a RIOT contributor or a RIOT maintainer, I would like to have it separately from the standard build system.

It depends, make can do a lot of powerful things, not necessarily directly related to the build system. Doing the same as done here with a script would not help I think.

[benpicco]

I moved the code to a script, so all you have to do if you add a new board is running add add_insufficient_memory_board.sh and wait a few minutes.

Running this on #12433 gives…

arduino_hello-world is OK
asymcute_mqttsn is too big for stm32f030f4-demo
bindist is OK
ccn-lite-relay is too big for stm32f030f4-demo
cord_ep is too big for stm32f030f4-demo
cord_epsim is too big for stm32f030f4-demo
default is too big for stm32f030f4-demo
dtls-echo is too big for stm32f030f4-demo
dtls-wolfssl is too big for stm32f030f4-demo
emcute_mqttsn is too big for stm32f030f4-demo
filesystem is too big for stm32f030f4-demo
gcoap is too big for stm32f030f4-demo
gnrc_border_router is too big for stm32f030f4-demo
gnrc_minimal is too big for stm32f030f4-demo
gnrc_networking_mac is OK
gnrc_networking is too big for stm32f030f4-demo
gnrc_tftp is too big for stm32f030f4-demo
hello-world is OK
ipc_pingpong is too big for stm32f030f4-demo
javascript is too big for stm32f030f4-demo
lorawan is too big for stm32f030f4-demo
lua_basic is too big for stm32f030f4-demo
lua_REPL is too big for stm32f030f4-demo
nanocoap_server is too big for stm32f030f4-demo
ndn-ping is too big for stm32f030f4-demo
nimble_gatt is OK
nimble_heart_rate_sensor is OK
nimble_scanner is OK
openthread is OK
posix_sockets is too big for stm32f030f4-demo
riot_and_cpp is too big for stm32f030f4-demo
saul is too big for stm32f030f4-demo
skald_eddystone is OK
skald_ibeacon is OK
suit_update is OK
timer_periodic_wakeup is OK
usbus_minimal is OK
bench_msg_pingpong is too big for stm32f030f4-demo
bench_mutex_pingpong is too big for stm32f030f4-demo
bench_runtime_coreapis is OK
bench_sched_nop is OK
bench_sizeof_coretypes is OK
bench_thread_flags_pingpong is too big for stm32f030f4-demo
bench_thread_yield_pingpong is too big for stm32f030f4-demo
bench_timers is too big for stm32f030f4-demo
bitarithm_timings is OK
bloom_bytes is too big for stm32f030f4-demo
board_calliope-mini is OK
board_microbit is OK
build_system_cflags_spaces is OK
build_system_utils is OK
buttons is OK
can_trx is OK
cb_mux_bench is OK
cb_mux is OK
cond_order is too big for stm32f030f4-demo
conn_can is OK
cortexm_common_ldscript is too big for stm32f030f4-demo
cpp11_condition_variable is too big for stm32f030f4-demo
cpp11_mutex is too big for stm32f030f4-demo
cpp11_thread is too big for stm32f030f4-demo
cpp_ctors is too big for stm32f030f4-demo
cpu_cortexm_address_check is OK
cpu_efm32_features is OK
devfs is too big for stm32f030f4-demo
driver_ad7746 is OK
driver_adcxx1c is OK
driver_ads101x is OK
driver_adt7310 is too big for stm32f030f4-demo
driver_adxl345 is OK
driver_apa102 is too big for stm32f030f4-demo
driver_at30tse75x is OK
driver_at86rf2xx is too big for stm32f030f4-demo
driver_ata8520e is too big for stm32f030f4-demo
driver_at is too big for stm32f030f4-demo
driver_bh1750 is OK
driver_bmp180 is OK
driver_bmx055 is OK
driver_bmx280 is OK
driver_cc110x is too big for stm32f030f4-demo
driver_ccs811_full is OK
driver_ccs811 is OK
driver_dht is OK
driver_ds1307 is OK
driver_ds18 is too big for stm32f030f4-demo
driver_ds3234 is OK
driver_ds75lx is OK
driver_dsp0401 is OK
driver_dynamixel is too big for stm32f030f4-demo
driver_enc28j60 is too big for stm32f030f4-demo
driver_encx24j600 is too big for stm32f030f4-demo
driver_feetech is too big for stm32f030f4-demo
driver_fxos8700 is OK
driver_grove_ledbar is OK
driver_hd44780 is OK
driver_hdc1000 is OK
driver_hih6130 is OK
driver_hts221 is OK
driver_ina220 is OK
driver_io1_xplained is OK
driver_isl29020 is OK
driver_isl29125 is OK
driver_jc42 is OK
driver_kw2xrf is too big for stm32f030f4-demo
driver_l3g4200d is OK
driver_lc709203f is OK
driver_lis2dh12 is OK
driver_lis3dh is OK
driver_lis3mdl is OK
driver_lpd8808 is too big for stm32f030f4-demo
driver_lpsxxx is OK
driver_lsm303dlhc is OK
driver_lsm6dsl is OK
driver_ltc4150 is too big for stm32f030f4-demo
driver_mag3110 is OK
driver_mma7660 is OK
driver_mma8x5x is OK
driver_motor_driver is OK
driver_mpl3115a2 is OK
driver_mpu9x50 is OK
driver_mq3 is OK
driver_my9221 is OK
driver_nrf24l01p_lowlevel is too big for stm32f030f4-demo
driver_nrfmin is OK
driver_nvram_spi is OK
driver_opt3001 is OK
driver_pca9685 is OK
driver_pcd8544 is too big for stm32f030f4-demo
driver_ph_oem is OK
driver_pir is too big for stm32f030f4-demo
driver_pn532 is OK
driver_pulse_counter is OK
driver_rn2xx3 is too big for stm32f030f4-demo
driver_sdcard_spi is too big for stm32f030f4-demo
driver_sds011 is OK
driver_servo is OK
driver_sht1x is too big for stm32f030f4-demo
driver_sht2x is OK
driver_sht3x is OK
driver_si114x is OK
driver_si70xx is OK
driver_soft_spi is OK
driver_srf02 is OK
driver_srf04 is OK
driver_srf08 is OK
driver_sx127x is too big for stm32f030f4-demo
driver_tcs37727 is OK
driver_tmp00x is OK
driver_tps6274x is OK
driver_tsl2561 is OK
driver_tsl4531x is OK
driver_vcnl40x0 is OK
driver_veml6070 is OK
driver_xbee is too big for stm32f030f4-demo
eepreg is OK
emb6 is too big for stm32f030f4-demo
embunit is OK
events is OK
event_wait_timeout is OK
evtimer_msg is too big for stm32f030f4-demo
evtimer_underflow is too big for stm32f030f4-demo
external_module_dirs is OK
fault_handler is OK
float is OK
fmt_print is OK
gnrc_gomach is OK
gnrc_ipv6_ext_frag is too big for stm32f030f4-demo
gnrc_ipv6_ext is too big for stm32f030f4-demo
gnrc_ipv6_fwd_w_sub is too big for stm32f030f4-demo
gnrc_ipv6_nib_6ln is too big for stm32f030f4-demo
gnrc_ipv6_nib is too big for stm32f030f4-demo
gnrc_lwmac is OK
gnrc_mac_timeout is too big for stm32f030f4-demo
gnrc_ndp is too big for stm32f030f4-demo
gnrc_netif_ieee802154 is OK
gnrc_netif is too big for stm32f030f4-demo
gnrc_rpl_srh is too big for stm32f030f4-demo
gnrc_sixlowpan_frag is too big for stm32f030f4-demo
gnrc_sixlowpan is too big for stm32f030f4-demo
gnrc_sock_dns is too big for stm32f030f4-demo
gnrc_sock_ip is too big for stm32f030f4-demo
gnrc_sock_udp is too big for stm32f030f4-demo
gnrc_tcp is too big for stm32f030f4-demo
gnrc_udp is too big for stm32f030f4-demo
heap_cmd is OK
irq is too big for stm32f030f4-demo
isr_yield_higher is too big for stm32f030f4-demo
l2util is OK
leds is OK
libc_newlib is OK
libfixmath is OK
libfixmath_unittests is too big for stm32f030f4-demo
log_color is OK
log_printfnoformat is OK
lua_loader is too big for stm32f030f4-demo
lwip is too big for stm32f030f4-demo
lwip_sock_ip is too big for stm32f030f4-demo
lwip_sock_tcp is too big for stm32f030f4-demo
lwip_sock_udp is too big for stm32f030f4-demo
malloc is OK
mcuboot is OK
memarray is OK
minimal is OK
mpu_stack_guard is too big for stm32f030f4-demo
msg_avail is OK
msg_send_receive is too big for stm32f030f4-demo
msg_try_receive is too big for stm32f030f4-demo
mtd_flashpage is OK
mutex_order is too big for stm32f030f4-demo
mutex_unlock_and_sleep is OK
nanocoap_cli is too big for stm32f030f4-demo
netdev_test is too big for stm32f030f4-demo
netstats_l2 is too big for stm32f030f4-demo
nhdp is too big for stm32f030f4-demo
nimble_l2cap is OK
nimble_l2cap_server is OK
nordic_softdevice is OK
od is OK
periph_adc is OK
periph_backup_ram is too big for stm32f030f4-demo
periph_cpuid is OK
periph_dac is OK
periph_dma is OK
periph_eeprom is OK
periph_flashpage is too big for stm32f030f4-demo
periph_gpio_arduino is OK
periph_gpio is too big for stm32f030f4-demo
periph_hwrng is OK
periph_i2c is OK
periph_pm is too big for stm32f030f4-demo
periph_pwm is too big for stm32f030f4-demo
periph_qdec is OK
periph_rtc is OK
periph_rtt is OK
periph_spi is too big for stm32f030f4-demo
periph_timer is OK
periph_uart is too big for stm32f030f4-demo
periph_uart_mode is OK
periph_wdt is OK
pipe is too big for stm32f030f4-demo
pkg_c25519 is too big for stm32f030f4-demo
pkg_cayenne-lpp is OK
pkg_cifra is too big for stm32f030f4-demo
pkg_cmsis-dsp is too big for stm32f030f4-demo
pkg_cn-cbor is too big for stm32f030f4-demo
pkg_fatfs is too big for stm32f030f4-demo
pkg_fatfs_vfs is too big for stm32f030f4-demo
pkg_hacl is too big for stm32f030f4-demo
pkg_heatshrink is too big for stm32f030f4-demo
pkg_jsmn is OK
pkg_libb2 is too big for stm32f030f4-demo
pkg_libcoap is too big for stm32f030f4-demo
pkg_libcose is too big for stm32f030f4-demo
pkg_libhydrogen is too big for stm32f030f4-demo
pkg_littlefs is too big for stm32f030f4-demo
pkg_lora-serialization is OK
pkg_microcoap is too big for stm32f030f4-demo
pkg_micro-ecc is too big for stm32f030f4-demo
pkg_micro-ecc-with-hwrng is too big for stm32f030f4-demo
pkg_minmea is too big for stm32f030f4-demo
pkg_monocypher is too big for stm32f030f4-demo
pkg_nanocbor is OK
pkg_nanopb is too big for stm32f030f4-demo
pkg_oonf_api is too big for stm32f030f4-demo
pkg_qdsa is too big for stm32f030f4-demo
pkg_relic is too big for stm32f030f4-demo
pkg_semtech-loramac is too big for stm32f030f4-demo
pkg_spiffs is too big for stm32f030f4-demo
pkg_tiny-asn1 is OK
pkg_tinycbor is OK
pkg_tinycrypt is OK
pkg_tweetnacl is too big for stm32f030f4-demo
pkg_u8g2 is too big for stm32f030f4-demo
pkg_ubasic is too big for stm32f030f4-demo
pkg_ucglib is too big for stm32f030f4-demo
pkg_umorse is OK
pkg_wolfcrypt-ed25519-verify is too big for stm32f030f4-demo
pkg_wolfssl is too big for stm32f030f4-demo
posix_semaphore is too big for stm32f030f4-demo
posix_time is OK
ps_schedstatistics is too big for stm32f030f4-demo
pthread_barrier is OK
pthread_cleanup is OK
pthread_condition_variable is too big for stm32f030f4-demo
pthread_cooperation is OK
pthread_flood is too big for stm32f030f4-demo
pthread is OK
pthread_rwlock is too big for stm32f030f4-demo
pthread_tls is OK
puf_sram is OK
riotboot_flashwrite is OK
riotboot_hdr is OK
riotboot is OK
rmutex is too big for stm32f030f4-demo
rng is too big for stm32f030f4-demo
saul is OK
sched_testing is too big for stm32f030f4-demo
shell is OK
slip is too big for stm32f030f4-demo
sntp is too big for stm32f030f4-demo
socket_zep is OK
ssp is OK
stdin is OK
struct_tm_utility is OK
sys_arduino is OK
sys_crypto is too big for stm32f030f4-demo
sys_irq_handler is too big for stm32f030f4-demo
test_tools is OK
thread_basic is too big for stm32f030f4-demo
thread_cooperation is too big for stm32f030f4-demo
thread_exit is too big for stm32f030f4-demo
thread_flags is too big for stm32f030f4-demo
thread_flags_xtimer is OK
thread_float is too big for stm32f030f4-demo
thread_flood is OK
thread_msg_block_race is OK
thread_msg_block_wo_queue is too big for stm32f030f4-demo
thread_msg_block_w_queue is too big for stm32f030f4-demo
thread_msg is too big for stm32f030f4-demo
thread_msg_seq is too big for stm32f030f4-demo
thread_priority_inversion is too big for stm32f030f4-demo
thread_race is OK
trace is OK
trickle is OK
unittests is too big for stm32f030f4-demo
usbus_cdc_acm_stdio is too big for stm32f030f4-demo
usbus_cdc_ecm is too big for stm32f030f4-demo
usbus is too big for stm32f030f4-demo
warn_conflict is OK
xtimer_drift is too big for stm32f030f4-demo
xtimer_hang is too big for stm32f030f4-demo
xtimer_longterm is too big for stm32f030f4-demo
xtimer_msg is too big for stm32f030f4-demo
xtimer_msg_receive_timeout is OK
xtimer_mutex_lock_timeout is too big for stm32f030f4-demo
xtimer_now64_continuity is OK
xtimer_periodic_wakeup is OK
xtimer_remove is OK
xtimer_reset is OK
xtimer_usleep is OK
xtimer_usleep_short is OK
dist/tools/insufficient_memory/add_insufficient_memory_board.sh   349,74s user 54,63s system 155% cpu 4:19,85 total

I might throw a git add in there because newly created Makefile.cis are easily missed.

[maribu]

@abadie:

It depends, make can do a lot of powerful things, not necessarily directly related to the build system. Doing the same as done here with a script would not help I think.

Sorry for being imprecise. I ment that the default Makefiles of applications and tests (and what they include by default) should be in my opinion not cluttered by feauteres that are only used under special circumstances. E.g. the desvirt stuff seems out if place to me. I have nothing against using make for other stuff than building apps.

@benpicco: This is extremely nice! I'll check on Monday, if nobody else is faster.

[benpicco]

I added a workaround to make buildsystem_sanity_check/check.shhappy.
Also, avr-gcc generates a different error message, so the script now also checks for that.

[benpicco]

@maribu It's monday again 😉

[maribu]

Sorry, totally forgot about this. Maybe you could add two or three lines in a README.md, so that users know that the script is the one to execute and not the Makefile? (Alternatively the Makefile could be renamed to something like Makefile.via_sh, so that just typing make will not work?)

[benpicco]

Actually I can just rename it to Makefile.ci and then buildsystem_sanity_check will not complain about setting BOARD_INSUFFICIENT_MEMORY anymore 😆

[maribu]

I think that the sanity check only checks for Makefiles containing BOARD_INSUFFICIENT_MEMORY, so every other file name should be fine.

[maribu]

ACK. Please squash