Skip to content

sys/saul_reg: support for saul context parameter#9099

Closed
ZetaR60 wants to merge 7 commits intoRIOT-OS:masterfrom
ZetaR60:RIOT_saul_context
Closed

sys/saul_reg: support for saul context parameter#9099
ZetaR60 wants to merge 7 commits intoRIOT-OS:masterfrom
ZetaR60:RIOT_saul_context

Conversation

@ZetaR60
Copy link
Contributor

@ZetaR60 ZetaR60 commented May 8, 2018

Note: some of this initial description is now obsolete. The complex / imaginary analogy was dropped, since it was confusing. The method of passing the context parameter is now a conventional argument (which is simpler, but requires more changes).

Description:

Currently SAUL does not handle large arrays of nearly identical devices well. For example, if you want to write to a large number of GPIO pins, or read from a lot of analog inputs, you must currently have a struct with separate parameters for each of them. It would be nice to be able to tell SAUL that you have an array of devices, and to share all parameters between them except for an index value.

My idea on how to do this:

  • The SAUL registry is set up to contain both real devices and imaginary duplicates of devices. They are "imaginary" since they do not have their own dev and params structs.
  • The number of members in the registry and the ordering in the registry takes into account imaginary members (which all appear together, in order of their index value).
  • Pointers to registry members are replaced with "complex" pointers, which are comprised of a real pointer, and an "imaginary" index value.
  • SAUL initialization functions only take real values (since they are not actually separate devices), other functions should just pretend that they have separate devices.

How the duplicates would be used:

  • If foo driver supports duplicates, then it should have a global const foo_imag_count defined in its foo_params.h file.
  • Foo driver's auto_init_foo.c should take imag_count and put it into saul_entries[i].imag_count in auto_init_foo.
  • When calling saul_reg_read / saul_reg_write and dev.imag != 0 the value of imag is stuffed into the first byte of the device descriptor. The first element of the device descriptor struct MUST be a byte intended for this use. C guarantees that the address of the first element is the same as the address of the struct. This value could also be passed by changing the definitions of saul_read_t and saul_write_t, but that would require a much more extensive rewrite.
  • Some other misc. things would need to be tweaked, such as foo_saul_info could use sprintf to put imag into the name of the device. You could get a nice output from the saul command, e.g. "Arduino Pin %u".

This is intended to be as conservative a change as possible, with almost all new behavior confined to when imag_count > 0.

Currently changes in this PR are limited to an example, because I want to wait for some feedback on the method before doing more extensive edits.

See also:

@bergzand bergzand added Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR Process: API change Integration Process: PR contains or issue proposes an API change. Should be handled with care. Area: SAUL Area: Sensor/Actuator Uber Layer labels May 8, 2018
Copy link
Member

@bergzand bergzand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some pre-coffee initial remarks

*/
typedef struct saul_complex_ptr {
struct saul_reg *real; /**< pointer to real device in SAUL registry */
uint8_t imag; /**< pointer to imaginary device duplicate */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more an index than a pointer right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is an analogy. It is a "pointer" that points to an "imaginary" device by way of an additional parameter. So far as the SAUL functions are concerned: It is as if there is an imaginary part of memory that holds the (actually nonexistent) data about the device.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, IMHO using pointer in the comment is a bit confusing. Thanks for clarifying!

* @return .real = NULL if no device is registered at that position
*/
saul_reg_t *saul_reg_find_nth(int pos);
saul_complex_ptr_t saul_reg_find_nth(int pos);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would change this to
int saul_complex_ptr_t saul_reg_find_nth(saul_complex_ptr_t *node, int pos);
and have it return 0 or 1 based on if there is a device at the position. This way checks if the device at @p pos exists are still easy and you have only an extra pointer in the arguments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

saul_reg_find_nth(pos).real != NULL gives you an easy way to test this as well. If you want to assign and test at the same time you could even do (foo = saul_reg_find_nth(pos)).real != NULL. Both are similar to the previous behavior (with just the addition of .real).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem is that this will require a double pointer to be passed in the argument, because you want saul_reg_find_nth to be able to modify it rather than its own local copy.

* @return .real = NULL if no device of that type could be found
*/
saul_reg_t *saul_reg_find_type(uint8_t type);
saul_complex_ptr_t saul_reg_find_type(uint8_t type);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

* @return .real = NULL if no device with that name could be found
*/
saul_reg_t *saul_reg_find_name(const char *name);
saul_complex_ptr_t saul_reg_find_name(const char *name);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And another :)

@ZetaR60 ZetaR60 mentioned this pull request May 16, 2018
24 tasks
@ZetaR60 ZetaR60 force-pushed the RIOT_saul_context branch from 1a33070 to 617cce3 Compare June 14, 2018 22:04
@ZetaR60 ZetaR60 changed the title sys/saul_reg: support for imaginary dev duplicates sys/saul_reg: support for saul context parameter Jun 15, 2018
@ZetaR60
Copy link
Contributor Author

ZetaR60 commented Jun 23, 2018

Now tested and working on mega-xplained.

user@user ~/Desktop/RIOT_saul_context/examples/saul $ make BOARD=mega-xplained term
/home/user/Desktop/RIOT_saul_context/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "9600"
2018-06-23 19:30:23,899 - INFO # Connect to serial port /dev/ttyACM0
Welcome to pyterm!
Type '/exit' to exit.
2018-06-23 19:30:25,502 - INFO # 

2018-06-23 19:30:25,602 - INFO # main(): This is RIOT! (Version: UNKNOWN (builddir: /home/user/Desktop/RIOT_saul_context))
2018-06-23 19:30:25,669 - INFO # Welcome to RIOT!
2018-06-23 19:30:25,669 - INFO # 
2018-06-23 19:30:25,702 - INFO # Type `help` for help, type `saul` to see all SAUL devices
2018-06-23 19:30:25,702 - INFO # 
> saul
2018-06-23 19:30:28,198 - INFO #  saul
2018-06-23 19:30:28,198 - INFO # ID	Class		Name
2018-06-23 19:30:28,199 - INFO # #0	SENSE_BTN	Button 0
2018-06-23 19:30:28,202 - INFO # #1	SENSE_BTN	Button 1
2018-06-23 19:30:28,269 - INFO # #2	ACT_SWITCH	LED 0
2018-06-23 19:30:28,270 - INFO # #3	ACT_SWITCH	LED 1
2018-06-23 19:30:28,302 - INFO # #4	SENSE_ANALOG	NTC thermistor
2018-06-23 19:30:28,402 - INFO # #5	SENSE_ANALOG	Light sensor
2018-06-23 19:30:28,404 - INFO # #6	SENSE_ANALOG	RC filter
> saul read all
2018-06-23 19:30:31,501 - INFO #  saul read all
2018-06-23 19:30:31,504 - INFO # Reading from #0 (Button 0|SENSE_BTN)
2018-06-23 19:30:31,569 - INFO # Data:	          0
2018-06-23 19:30:31,571 - INFO # Reading from #1 (Button 1|SENSE_BTN)
2018-06-23 19:30:31,573 - INFO # Data:	          0
2018-06-23 19:30:31,668 - INFO # Reading from #2 (LED 0|ACT_SWITCH)
2018-06-23 19:30:31,669 - INFO # Data:	          0
2018-06-23 19:30:31,671 - INFO # Reading from #3 (LED 1|ACT_SWITCH)
2018-06-23 19:30:31,702 - INFO # Data:	          0
2018-06-23 19:30:31,768 - INFO # Reading from #4 (NTC thermistor|SENSE_ANALOG)
2018-06-23 19:30:31,769 - INFO # Data:	         20
2018-06-23 19:30:31,803 - INFO # Reading from #5 (Light sensor|SENSE_ANALOG)
2018-06-23 19:30:31,868 - INFO # Data:	        789
2018-06-23 19:30:31,869 - INFO # Reading from #6 (RC filter|SENSE_ANALOG)
2018-06-23 19:30:31,902 - INFO # Data:	        308
> saul read 0
2018-06-23 19:30:34,802 - INFO #  saul read 0
2018-06-23 19:30:34,903 - INFO # Reading from #0 (Button 0|SENSE_BTN)
2018-06-23 19:30:34,904 - INFO # Data:	          0
> saul read 1
2018-06-23 19:30:37,072 - INFO #  saul read 1
2018-06-23 19:30:37,075 - INFO # Reading from #1 (Button 1|SENSE_BTN)
2018-06-23 19:30:37,102 - INFO # Data:	          0
> saul read 2
2018-06-23 19:30:39,176 - INFO #  saul read 2
2018-06-23 19:30:39,178 - INFO # Reading from #2 (LED 0|ACT_SWITCH)
2018-06-23 19:30:39,202 - INFO # Data:	          0
> saul read 3
2018-06-23 19:30:41,502 - INFO #  saul read 3
2018-06-23 19:30:41,505 - INFO # Reading from #3 (LED 1|ACT_SWITCH)
2018-06-23 19:30:41,506 - INFO # Data:	          0
> saul read 4
2018-06-23 19:30:43,502 - INFO #  saul read 4
2018-06-23 19:30:43,603 - INFO # Reading from #4 (NTC thermistor|SENSE_ANALOG)
2018-06-23 19:30:43,604 - INFO # Data:	         20
> saul read 5
2018-06-23 19:30:45,602 - INFO #  saul read 5
2018-06-23 19:30:45,703 - INFO # Reading from #5 (Light sensor|SENSE_ANALOG)
2018-06-23 19:30:45,703 - INFO # Data:	        755
> saul read 6
2018-06-23 19:30:48,002 - INFO #  saul read 6
2018-06-23 19:30:48,104 - INFO # Reading from #6 (RC filter|SENSE_ANALOG)
2018-06-23 19:30:48,105 - INFO # Data:	        308
> saul read 7
2018-06-23 19:30:51,703 - INFO #  saul read 7
2018-06-23 19:30:51,705 - INFO # error: undefined device id given
> saul write 2 1
2018-06-23 19:30:57,903 - INFO #  saul write 2 1
2018-06-23 19:30:57,969 - INFO # Writing to device #2 - LED 0
2018-06-23 19:30:57,970 - INFO # Data:	          1
2018-06-23 19:30:58,002 - INFO # data successfully written to device #2
> saul read 2
2018-06-23 19:31:00,886 - INFO #  saul read 2
2018-06-23 19:31:00,888 - INFO # Reading from #2 (LED 0|ACT_SWITCH)
2018-06-23 19:31:00,902 - INFO # Data:	          1
> saul write 2 0
2018-06-23 19:31:04,303 - INFO #  saul write 2 0
2018-06-23 19:31:04,369 - INFO # Writing to device #2 - LED 0
2018-06-23 19:31:04,370 - INFO # Data:	          0
2018-06-23 19:31:04,402 - INFO # data successfully written to device #2
> saul read 2
2018-06-23 19:31:07,803 - INFO #  saul read 2
2018-06-23 19:31:07,904 - INFO # Reading from #2 (LED 0|ACT_SWITCH)
2018-06-23 19:31:07,905 - INFO # Data:	          0
> saul write 3 1
2018-06-23 19:31:14,303 - INFO #  saul write 3 1
2018-06-23 19:31:14,305 - INFO # Writing to device #3 - LED 1
2018-06-23 19:31:14,403 - INFO # Data:	          1
2018-06-23 19:31:14,405 - INFO # data successfully written to device #3
> saul read 3
2018-06-23 19:31:16,703 - INFO #  saul read 3
2018-06-23 19:31:16,705 - INFO # Reading from #3 (LED 1|ACT_SWITCH)
2018-06-23 19:31:16,802 - INFO # Data:	          1
> saul write 3 0
2018-06-23 19:31:30,103 - INFO #  saul write 3 0
2018-06-23 19:31:30,169 - INFO # Writing to device #3 - LED 1
2018-06-23 19:31:30,170 - INFO # Data:	          0
2018-06-23 19:31:30,202 - INFO # data successfully written to device #3
> saul read 3
2018-06-23 19:31:33,603 - INFO #  saul read 3
2018-06-23 19:31:33,605 - INFO # Reading from #3 (LED 1|ACT_SWITCH)
2018-06-23 19:31:33,703 - INFO # Data:	          0
> /exit
2018-06-23 19:31:54,184 - INFO # Exiting Pyterm

@ZetaR60 ZetaR60 force-pushed the RIOT_saul_context branch from bfdf5b3 to 2b18600 Compare June 24, 2018 19:58
@ZetaR60
Copy link
Contributor Author

ZetaR60 commented Jun 24, 2018

Rebased to pick up any new drivers, and API change support for everything in drivers/ added.

@ZetaR60 ZetaR60 added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Jun 24, 2018
@ZetaR60 ZetaR60 requested a review from haukepetersen June 24, 2018 21:52
@ZetaR60 ZetaR60 force-pushed the RIOT_saul_context branch from b2d140e to a3cdd71 Compare July 2, 2018 17:42
@ZetaR60 ZetaR60 force-pushed the RIOT_saul_context branch from a3cdd71 to e36b308 Compare July 2, 2018 18:14
@ZetaR60
Copy link
Contributor Author

ZetaR60 commented Jul 2, 2018

Rebased on trunk and resolved conflicts.

@ZetaR60 ZetaR60 added this to the Release 2018.07 milestone Jul 2, 2018
@haukepetersen
Copy link
Contributor

By just (very quickly) looking over this PR I have to say I dont like it. I moves SAUL into a direction it was never designed for, but I need more time to look into this PR better...

@ZetaR60
Copy link
Contributor Author

ZetaR60 commented Jul 2, 2018

@haukepetersen Thanks for taking a look. Unfortunately there doesn't seem to be a great way of using SAUL for many similar devices (i.e. a driver board with many drivers, or an analog input board with many multiplexed inputs). I have tried to move the context parameter into the background as much as possible, by treating each dev+context as a separate virtual device. Once you get a chance to examine it in more detail, we could discuss a redesign to try to improve things.

@haukepetersen
Copy link
Contributor

@ZetaR60 no problem and I have to excuse in advance that I will be mostly offline for about a month starting next week...

My main problem with this PR is, that it is based on a wrong understanding of what SAUL actually is and what SAUL is supposed to do, leading to this PR further introducing code that brakes the concept of SAUL.

The main idea of SAUL is to provide unified access to device drivers, based on their perception as actual physical entities of some sort, so interacting with devices on a high-level based on what they do/represent in a functional scope (hence SAUL is based on physical units tied to actual 'physical' device types).

It was never meant as an abstraction for logical, generic MCU and device peripherals (such as GPIO/ADC). This is the domain of the periph interface. Now in the past there was some effort and approaches to directly map things like GPIO pins and ADC lines to the SAUL interface, but this was typically done in scope of tying these peripherals to actual physical 'devices', such as a light (LED), or a high-level button/switch, tying the actual peripheral instance used with some additional information (i.e. pin polarity). The cases where peripherals are 'blindly' thrown into the SAUL registry, without tying them to any specific, high-level entity are already very borderline... If I am not mistaken I expressed my concerns about this multiple times in the past.

Now let me try to sketch what I think should be the way forward to handle cases introduces e.g. with drivers like #9054 or the ads101x: the main problem is that periph and SAUL are not properly differentiated! We already started to abuse SAUL by adding logical instances of random peripherals to the SAUL registry, for utilizing SAUL's benefits (i.e. iterate instances, unified shell command). And this is where I see the actual problem. So what we IMO actually want is to introduce tooling similar to SAUL for periph interfaces, making it simple to iterate and access them using some kind of generic read/write together with a way of iterating them.

So in short:

  • SAUL should keep taking individual state for all its endpoints
  • logical devices/peripherals should be mapped and made accessible through periph related 'tooling'
  • we (still) need a mid-level 'periph' wrapper, that allows multiplexing of different kind of low-level devices under a unified API (e.g. multiplex the pins provided by a pca95xx device with the CPU pins into a single API) -> this is a known issue, and SAUL is not the solution nor the scope solving this

Sorry for the long text, I hope my thoughts are clear enough to be followed :-)

@ZetaR60
Copy link
Contributor Author

ZetaR60 commented Jul 4, 2018

@haukepetersen Thanks for the comprehensive explanation! I feel that in the general sense of what the context parameter can do it does break with what you have explained as the goal of SAUL. However, my intention here and the specific use I have put it towards I think is a much smaller conceptual extension to SAUL. The goal of this PR is to decouple memory usage in the SAUL registry from devices that appear in the SAUL registry. The context parameter when used creates the appearance of more devices in the SAUL registry without actually having separate entries in memory. Each of these virtual devices is intended to be a physical entity (such as a pin). As far as users of the SAUL interface are concerned, the only difference between a virtual device and a real one is that multiple virtual devices happen to share settings with adjacent registry entries, that's all. I think that it may be possible to better follow what you have laid out as the goals for SAUL by making the context parameter more invisible to users of the SAUL API.

My concern with a separate API for these sorts of devices (arrays of physical entities that share settings), is that it will be replicating work already done for SAUL, and that you will end up with an API that is almost the same as SAUL but with a context parameter (or similar). Unfortunately, without a parameter similar to the context parameter here, work from SAUL cannot be directly reused for this new API. I suggest that the addition of the context parameter to SAUL is the most straightforward and efficient way to accomplish handling of these devices. I hope that we can salvage work from this PR, and simply adapt it better to your vision of the goals of SAUL.

I think I should take a moment to outline my ultimate goal with this work: I am working towards CoAP access to all sensors and actuators. SAUL is in the perfect position to do this, and it supplies much of the detail required for CoAP. The primary limitation I have seen so far is the memory usage inefficiency when handling arrays of devices that share settings.

@cladmi
Copy link
Contributor

cladmi commented Jul 11, 2018

I think the problem you have is more with saul_reg than with saul.
The shell has this saul command but it actually interacts with saul_reg.

I see your problem of apply all leds as more a missing query to get all_leds in saul_reg than a saul issue. And grouping leds together is really a specific need.

I will explain my point of view below:


I will use the word "collection" for a structure with a group of objects, list, dict, tree (meaning in Python)

Regarding the API change, I see this change as integrating "collections" (list/dict) to the saul device and I see it as very specific need.

For me:

  • saul as an object oriented API. I have an object, I read/write values from/to it.
  • saul_reg is a collection of objects.

A python equivalent

saul_acc = saul_reg.get("accelerometer")
values = saul_acc.read()

By adding this context pointer to saul, you are changing this saul object to a collection of objects.

saul_leds = saul_reg.get("leds")
value = saul_leds.read("blue_led")

Which could make sense for an object represented as a group.

However, if I have a sensor with only one entity, I need to do an additional selector:

saul_accs = saul_reg.get("accelerometers")
value = saul_accs.read(None)

Here I should select it with None or 0/NULL in C for no reasons.

As I see it, saul_reg is one type of collection of objects. There could be other ones.
I could want to store sensors by their usage. I have a sensor for temperature and an associated led to show if it is too warm, and a sensor for the pressure and an associated led to show if the pressure is too high:

saul_pressure_sensor = my_collection.get("pressure_monitor").get("sensor")
saul_pressure_status = my_collection.get("pressure_monitor").get("led")

And if saul store leds as a list, I am bit screwed, I need to internally store dev + ctx_pointer instead of just the device that includes it, so basically work around your PR.

Having a simpler access to all_leds maybe means adding this query to saul_reg or using a different collection for this.


For the argument of saving memory:

On the version of master before this PR:

   text    data     bss     dec     hex filename
  11112    1950     781   13843    3613 /home/harter/work/git/RIOT/examples/saul/bin/mega-xplained/saul_example.elf

With this PR:

   text    data     bss     dec     hex filename
  11652    1964     775   14391    3837 /home/harter/work/git/RIOT/examples/saul/bin/mega-xplained/saul_example.elf

Same RAM but more ROM usage.

For saving memory, you can just put some part of your dev in common in the device implementation accessed with a pointer.
From what I remember, some device at least separate the const and the variable part to put more things in ROM. You could do the same where it is useful without changing the API.

@ZetaR60
Copy link
Contributor Author

ZetaR60 commented Jul 11, 2018

@cladmi Thanks for the analysis. Let me clarify the PRs goals a bit in the context of what you said: The LEDs and buttons on mega-xplained are just a toy problem to demonstrate how the context parameter works (I do not expect it to be a valid demonstration of memory efficiency); the real goal is to be able to efficiently handle automation tasks requiring dozens of almost identical devices in SAUL (such as generic drivers, or multiplexed analog inputs).

The way I was attempting to do efficient handling of identical interfaces in this PR is to separate the abstract idea of an object in SAUL from the actual representation in memory. You can see this in how I have modified saul_reg_find_nth, which treats the list of objects as dev0ctxt0, dev0ctxt1, dev0ctxt2, ..., rather than dev0, dev1, dev2. The intention is that a device + its context is one "object", and the code using the API passes around the objects without actually being aware that the context parameter is part of the definition. However, the context parameter is not yet invisible to users of the API due to how it is being used.

I am hoping that by improving the use of the context parameter until it is invisible to users of the API, it satisfies the goal of efficient access of identical interfaces while being only a very limited extension of @haukepetersen 's vision for SAUL. Improvements to the use of the context parameter would only require modifications to this PR, rather than throwing it out completely.

In short: to use your Python example:

saul_acc = saul_reg.get("Accelerometer #1")
values = saul_acc.read()

The goal is for this to work in exactly the same way for users of the API. The change would be that the saul_acc holds the context parameter as well as the pointer, invisibly to the user of get and read. I acknowledge that there is room for improvement in this invisibility, though I would like to first settle the matter of whether this PR would be approved at all if those improvements are made.

@ZetaR60 ZetaR60 removed this from the Release 2018.07 milestone Jul 22, 2018
@stale
Copy link

stale bot commented Aug 10, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

@stale stale bot added the State: stale State: The issue / PR has no activity for >185 days label Aug 10, 2019
@stale stale bot closed this Sep 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: SAUL Area: Sensor/Actuator Uber Layer CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR Process: API change Integration Process: PR contains or issue proposes an API change. Should be handled with care. State: stale State: The issue / PR has no activity for >185 days

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants