build-system: auto-locate modules by name

[maribu]

Contribution description

This PR introduces the MODULE_DIRS variable that works just like EXTERNAL_MODULE_DIRS, but it is intended for internal modules. The major difference is that external modules can have the same name as an internal module, in which case the external module is used instead
of the internal one. This allows to provide drop-in replacements of internal modules, which can be useful during development e.g. to quickly benchmark different implementations.

Testing procedure

All applications for all boards should still build and yield identical binaries.

Issues/PRs references

Follow up of #16104

[fjmolinas]

This only works with modules in a directory matching the module name right?

[maribu]

This only works with modules in a directory matching the module name right?

Exactly. For those modules, one still has to manually add the dirs to the DIRS variable. But e.g. for all modules in drivers, this already works.

[fjmolinas]

This only works with modules in a directory matching the module name right?

Exactly. For those modules, one still has to manually add the dirs to the DIRS variable. But e.g. for all modules in drivers, this already works.

Can this be documented?

[fjmolinas]

This only works with modules in a directory matching the module name right?

Exactly. For those modules, one still has to manually add the dirs to the DIRS variable. But e.g. for all modules in drivers, this already works.

Can this be documented?

BTW, couldn't we do the same for external modules as well?

[maribu]

This only works with modules in a directory matching the module name right?

Exactly. For those modules, one still has to manually add the dirs to the DIRS variable. But e.g. for all modules in drivers, this already works.

Can this be documented?

BTW, couldn't we do the same for external modules as well?

Yes, but at least one external module has to be found without, which could extend the DIRS variable as needed. (Or the applications Makefile could do so.)

I intend btw. to extend the mechanism to allow assigning prefixes to search dirs. E.g sys/gnrc should find module gnrc_%, where % is the folder name. With this, we might be able to automatically locate all modules eventually.

[fjmolinas]

Won't this be affected by the order of inclusion?

[maribu]

The location of modules is done fully again at each recursion step. If xtimer would be auto-located in a transient state, it would be dropped later on. But some care should be taken so that an Makefile.dep doesn't do anything xtimer related when ztimer_xtimer_compat is also used. But since there is no Makefile.dep in sys/xtimer and in sys/Makefile.dep (above code) the correct guards are in place, this should just work.

A bit of a foot gun, but choosing different implementation options for the same API is just not properly modeled in the build system, so some friction is expected.

[MrKevinWeiss]

Do you think this should be included in this coming release or should we push it?

[benpicco]

Needs a rebase (feel free to squash while you're at it)

[fjmolinas]

One thing that is not clear to me is if this PR wants to imply that module to have the same name as the directory?, Module names that mix the directories into it are a common practice in the build system, e.g.: foo/bar and MODULE = foo_bar. It is widely used in sys so I don't see why do it for sys/bus and not other modules. I won't necessarily say its an argument one way or another but I do find it cleaner to have MODULEs grouped under a directory. E.g.

sys
│
└───ssp
│   │   Makefile 
│   └───subfolder1
└───shell
│   │
│   └───commands
│       │  Makefile
│       │   ...
└───test_utils
│   │
│   └───interactive_sync
│   │   │  Makefile (MODULE=test_utils_interactive_sync)
│   │   │   ...
│   └───result_output
│       │  Makefile (MODULE=test_utils_result_output)
│       │   ...
│       └───check
│       │  │  Makefile (MODULE=test_utils_result_output_check)
│       │  │  ....
│       └───json
│             │  Makefile (MODULE=test_utils_result_output_json)
│             │  ....

It seems to me that if the same change as done for sys_bus is done here things would get messy, and so what is the guideline here? If the name is long and messy then split into subdirs? if not mix them all up together? There is a question here on how we want to structure our code, what will be our conventions, etc.. EXTERNAL_MODULES now have a hard requirement on MODULE names that are not enforced for the internal module.

So what is for me not clear from the PR description is:

• what is the goal of the PR: is it being able to provide drop-in replacements or for conveniently include modules?
• this PR only works for modules that can be auto-located this leads to different behaviour depending on the module naming which is not documented
• what kind of tree structure or we envisioning, what shall be our conventions for naming modules, structuring directories, etc...

[fjmolinas]

What is mods2?

[maribu]

modules to locate ;-)

[maribu]

Module names that mix the directories into it are a common practice in the build system, e.g.: foo/bar and MODULE = foo_bar.

That should actually be possible to still auto-detect, as long as the pattern is consistent. I have an idea how this might be possible to implement in make without complexity getting too high.

[OlegHahm]

@maribu, ping!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

[maribu]

Let's close this. It is so outdated that one would need to start from scratch anyway. And maybe after the KConfig migration we can switch to a better build system instead :)