Proposal: New Directory Structure & Public Python APIs

[kdmccormick]

Supporting principles

• The LMS/CMS distinction is not great. Either most code ends up being "shared", or each app end up being split into 2-3 parts (discussions, discussions, discussions 🤯 ). A better model is layered apps, using importlinter to enforce that low-level apps don't depend on high-level ones. For example, Learning Core successfully uses importlinter to layer its libs and apps.
• Plugin developers are going to import openedx-platform code whether we want them to or not. What's within our power is steering them towards allowlisted Python APIs that we think we can best support long-term. This can be established now, and the current+future openedx-platform directory structures can be just implementation details. For example, Learning Core organizes its stable APIs into openedx_learning.api, even though the internal directory structure is more complicated.

Proposal

• Create a new top-level directory: ./openedx_platform/.
  • This is the directory that we publish to PyPI.
  • ./openedx_platform/api/-- All public APIs. Watched with CODEOWNERS. Breaking changes get DEPRs and a PyPI major version bump. Each module is built by importing stable functions from internal ../apps/ and re-exporting them. For example:
    • ./openedx_platform/api/content_libraries.py
    • ./openedx_platform/api/xblock.py
    • ./openedx_platform/api/rbac.py
    • Certain "public DB models" are allowlisted on a case-by-case basis, e.g.:
      • ./openedx_platform/api/content_libraries_models.py
      • ./openedx_platform/api/xblock_models.py
      • ./openedx_platform/api/rbac_models.py
  • ./openedx_platform/apps/ -- Future home for all djangoapp implementations, layered using importlinter.
  • ./openedx_platform/lib/ -- Future home for internal libraries (no models).
  • ./openedx_platform/settings/ -- Future home for settings.
• Deprecate external imports to all other paths (lms, cms, openedx, common, and xmodule).
  • Long-term, actually move the code from these dirs into openedx_platform. But, no big rush to do this.

Open questions

• We will still want to support running the platform in "LMS mode" and "CMS mode". What's the right way to support this? What benefits from the LMS/CMS split do we want to retain (e.g. different scaling needs for Studio vs LMS)?

[kdmccormick]

This relates both to the Learning Core work (@bradenmacdonald @ormsbee ) and the sample-plugin conference workshop (@feanil ).

[bradenmacdonald]

A better model is layered apps, using importlinter to enforce that low-level apps don't depend on high-level ones. For example, Learning Core successfully uses importlinter to layer its libs and apps.

We are using importlinter layering in this repo too:

openedx-platform/setup.cfg

Lines 209 to 216 in 5a411c2

	layers =
	# Layers from high-level to low-level. Imports should only occur from higher to lower.
	cms.lib.xblock.upstream_sync | openedx.core.djangoapps.content.search | openedx.core.djangoapps.olx_rest_api
	openedx.core.djangoapps.content_libraries
	openedx.core.djangoapps.content_staging
	openedx.core.djangoapps.xblock
	openedx.core.lib.xblock_serializer
	openedx.core.djangoapps.content_tagging

I really like this approach, but I wish that there was a way to make it more obvious which "level" each app is at, other than referring to the linter config file.

./openedx_platform/lib/ -- Future home for internal libraries (no models).

Is the criteria of "does it have database models" really a particularly useful criteria for organizing code? For example, in the existing platform layers I referenced above, the xblock_serializer library (no models) depends on content_tagging (which does have models).

I know it may matter as far as Django's INSTALLED_APPS list is concerned, but I would prefer we pick a directory structure that's useful to humans, and which can better reflect the layering.

We will still want to support running the platform in "LMS mode" and "CMS mode".

Will we though? The code would likely get significantly cleaner and simpler over time if we just combine them into one big headless REST API server. It might be a more efficient use of memory and CPU too.

Otherwise, if we're going to keep the split, I would actually prefer to see more separation between CMS and LMS, rather than this "mode" toggle on the shared app.

different scaling needs for Studio vs LMS

We scale that way today because it's the only option, but I suspect it would be much better if we could, for example, scale up a few lightweight servers that handle only the problem_check and "get unit" endpoints (e.g. during exams). Focusing in on a few hot endpoints and separating them out would allow those processes to use much less memory compared to workers that load into memory anything used by the entire LMS.

We could also work to move some of the slower API endpoints to use asynchronous Django for I/O calls (e.g. the database or the discussions service), which I suspect would be a much bigger win for scalability than the LMS+CMS split ever was.

• ./openedx_platform/api/-- All public APIs.

Where does the implementation of REST APIs fall under this scenario? I would love if we can move to a model where the REST APIs generally mirror the python APIs 1:1. That's easiest to do when the REST APIs are only allowed to import the public APIs, rather than living in the apps code and importing internal APIs and models. Ideally we could use django-ninja instead of DRF to make this much nicer and faster.