[GSoC2025] Add Silicon Heaven protocol adaptation into NuttX and SHV examples

[zdebanos]

This commit marks the end of my GSoC 2025 project in the NuttX section.

Currently creating as a draft pull request.
I have agreed with Pavel Pisa to separate the library's dependancy named ulut from the shv-libs4c repository (located currently here https://github.com/silicon-heaven/shv-libs4c/tree/master/submodule). This will require more time for me. Also, I have discussed with Karel Koci from Elektroline a.s. about the posibilities of the exports of external libraries - something that I somehow solved in this commit: #3154.
In the end we have agreed that this should be considered as "hacking the whole Makefile system" and I should rather come up with a better solution, and thus I'm postponing it. Since I require the export for the GSoC, I'd be glad to leave it here as draft until I solve all the issues already mentioned.

I am about to add documentation into the nuttx repo, too.

Summary

Please look into the commit's verbose message :-).

Impact

Adds the Silicon Heaven protocol implementation. A framework designed to register your callbacks into your application. The callbacks are organized into nodes and the nodes can form a tree structure.

Testing

The testing was done on my custom samv7 board. In nuttx:
tools/configure.sh myboard/myconfig - configuration OK
make - downloads the shv-libs4c repo from github, and compiles OK

DISCLAIMER: in my GSoC, I was using make export. But with the solution used here (https://github.com/zdebanos/pysimCoder/tree/shv-refactor) I succesfully compiled against automatically generated source code in pysimCoder.

[zdebanos]

I came across an issue, which must be fixed: locally, I was running against pyshv==0.7.3 (implementation of Silicon Heaven API in python) but when I added requirements and changed the imports to match with pyshv==0.10.0, it stopped working, probably by using a bad function of some sort. I'll have to look deeper into that.

All i know is that the script worked with the previous version, the target device functionality has not changed.

[simbit18]

would reformat /home/runner/work/nuttx-apps/nuttx-apps/apps/examples/shv-nxboot-updater/update-script/gui.py

Oh no! 💥 💔 💥
1 file would be reformatted.

black check failed, run following command to update the style:
  $ black <src>

[zdebanos]

@simbit18 I know about that, I'm working on a different fix and I'm gonna fix the linter error too.

[zdebanos]

I also need to address the shv-libs4c correct version. I've got two options I can implement:

1. download a correct .zip from github releases
2. clone a "specific hash" commit and compile against it

I'm in favor with the second option. That means I'll have to specify the commit hash inside the Makefile.

Currently, I'm running against master as it's still draft and I'm fixing last errors. Then I'll specify the correct version.

[michallenc]

I also need to address the shv-libs4c correct version. I've got two options I can implement:

1. download a correct .zip from github releases

2. clone a "specific hash" commit and compile against it

I'm in favor with the second option. That means I'll have to specify the commit hash inside the Makefile.

Currently, I'm running against master as it's still draft and I'm fixing last errors. Then I'll specify the correct version.

I am also for the second option. The commit hash can even be a configurable option in Kconfig to easily allow the change of a version per project.

[zdebanos]

Thanks to Pavel Pisa, the Makefile system adjusts better to the build system used in the shv-libs4c library. As discussed in the heading, we have also successfully separated the ulut dependancy from the shv-libs4c library and now it's an external dependancy.

Currently, ulut is placed in neutilts/libulut. We can't think of a better place to put this library, as it is a collection of data structures (containers) written in the C language, and as it is a direct dependency of SHV, we put it parallel to libshvc. We should probably put this somewhere else but now we are keeping it here.

Also I suppose the export of the library should be done outside of this PR as a separate commit. I have found out we can use an EXTOBJS variable to handle the job better (only for .o files, not a solution to .h files).

I have also changed the apps a bit, now the mandatory arguments are position independent.

Also the pipeline is failing due to a typo. But I do not think it's a typo the pipeline should fail at since it is used in a macro in the Makefile. But I'll consult this with Pavel Pisa.

[zdebanos]

I have unmarked this commit as draft since we have agreed upon a stable solution. The only thing that needs to be done is the documention, I'm working on it right now.

[xiaoxiang781216]

can we use depends on instead select

[ppisa]

I am not sure. The depend would hide SHV support if LIBULUT is not enabled and we do not want to enable it by default. When somebody looks for SHV (which is probable start point/intention) then enable of SHV would enable ULUT when select is used. And ULUT is not used for anything else in the NuttX for now. It can change in future if we want and common sense would consider useful to include some other our project utilizing uLUt

• uLan communication “Open Source implementation of a multi-master RS485 communication protocol” https://ulan.sourceforge.net/ (already tested with NuttX and includes NuttX support in mainline in addition to Linux, Windows and operating-system-less devices - sysless)
• OCERA Real-Time CAN https://ortcan.sourceforge.net/ with dynamic, EDS lead based CAN open device, manager and monitor (little dated unfortunately, but with extension to CAN FD interesting project still) (Linux NuttX and system less support included)
• Open Real-Time Ethernet (ORTE) DDS RTPS C language implementation https://orte.sourceforge.net/ Again without care for long time and only protocol version 1.0 at that time. But it has been exactly the worldwide second implementation compatible with original RTI NDDS and that interoperability has been base for DDS Internet standard acceptance. It can be interesting still if upgrade to DDS RTPS 2.0 version is finished (there is the start of update by Jiri Vlasak - https://en.osm.town/@qeef).
• Small User Interface Toolkit (SuiTk) https://gitlab.com/pikron/sw-base/suitk/-/wikis/home providing higher level layer above Microwindows which allows for hand written C code run on systemless MCUs with 64 kB RAM and still utilizing event connectors (Qt signal slot equivalents) and with little more RAM it can use full declarative interface, busyness logic description dynamically build from files with XML syntax subset.

So there is some potential for uLUt broader use with NuttX, but it is only support dependency for SHV in NuttX for now. When situation changes we can reconsider the choice.

[raiden00pl]

select XXX has its problems and should be avoided when possible. There were discussions about this topic in several places on github lately (I don't remember exactly where) and on the mailing list: https://lists.apache.org/thread/n7qq5341cln5d3d4roy5nj23yc8hgzn6

The conclusion is that we should use select only for options that are not visible to the user. BTW, the user can always show hidden symbols using "show-all mode" in menuconfig.

[zdebanos]

I initially wanted to use depends on because i think it's not that of a big issue to manually turn it on by yourself. But Pavel Píša (mentor) suggested using select. Still, Im in favor of depends on.

[ppisa]

OK, then I suggest to add into uLUt text (required by SHV) or something similar that even if you search through visible options for SHV, you find the uLUt and enable it to see more.

"uLan Utilities Library (ULUT, required by SHV)"

Is that OK?

[raiden00pl]

We can use something like this:

if !NETUTILS_LIBULUT && !NETUTILS_LIBSHVC
comment "uLan Utilities Library requiered to support SHV"
endif

[zdebanos]

I agree with @raiden00pl, you see what it's for but the moment you turn it on, it dissapears and it does not bloat the kconfig menu.

[zdebanos]

@raiden00pl @ppisa fixed. Also I've tested that the whole system compiles with ulut only and ulut + shv.

[xiaoxiang781216]

should we skip the clone if user already git clone manually like other 3rd party project?

[ppisa]

I am not sure what is the best option. If the repository is already cloned then there should be mechanism how to keep it at the correct version matching given NuttX version/commit.

I have implemented next script for integration into pysimCoder

https://github.com/zdebanos/pysimCoder/blob/gsoc-2025-shv-refactor/ExtLibs/extlib-fetch.sh

It keeps the local branch, if the repository is clean it switches it to the specified library version by reset. If there is only small local change which does not clash with switch to newly required commit, it informs user and switches. If there are conflicting changes it fails. If there is another branch and state is clean it switches and it is user's task to move changes.

I am not sure what is preferred and typical for NuttX

[zdebanos]

I think the current solution is better - you get the correct version that Is supported by the library (i.e. what we think works the best). Your approach would in my opinion overcomplicate things.

[raiden00pl]

I also think we should use a similar approach as in other 3rd party projects to be consistent in the project: download the archive and unpack it.
Example here:

nuttx-apps/wireless/bluetooth/nimble/Makefile

Lines 43 to 58 in 10cab6b

	$(NIMBLE_TAR):
	$(Q) curl -L $(NIMBLE_URL) -o $(NIMBLE_TAR)
	$(NIMBLE_UNPACKDIR): $(NIMBLE_TAR)
	$(Q) tar zxf $(NIMBLE_TAR)
	$(Q) mv mynewt-nimble-$(CONFIG_NIMBLE_REF) $(NIMBLE_UNPACKDIR)
	$(Q) touch $(NIMBLE_UNPACKDIR)
	# Download and unpack tarball if no git repo found
	ifeq ($(wildcard $(NIMBLE_UNPACKDIR)/.git),)
	context:: $(NIMBLE_UNPACKDIR)
	distclean::
	$(call DELFILE,$(NIMBLE_TAR))
	$(call DELDIR,$(NIMBLE_UNPACKDIR))
	endif

By default, if the user only wants to use a 3rd party project, the code will always be downloaded from a verified hash.
But skipping the package download when a git repository exists makes the work of developers working on a given 3rd party project much easier.
Maintaining the correct version of code when a user uses their own git repo is not a problem. The user has to clone the repository to the correct location, and if they do so, we can assume they're doing it consciously and intentionally.

Another thing is that the user may want to use custom 3rd party package with custom patches. With the current solution this is impossible.

[cederom]

In a perfect world we would like to have all external dependencies listed somewhere, fetch before build, and then build. This is important for testing automation in order to provide all dependencies prior build is executed where malicious scripts can be executed. But this is another story of the future :-)

[xiaoxiang781216]

should we skip the clone if user already git clone manually like other 3rd party project

[zdebanos]

Ditto