Versioning scheme

[howetuft]

Problem statement

To build a new version of BlendLuxCore, we need three components:

1. The external dependencies
2. The pyluxcore library and the Python wheels
3. BlendLuxCore

Problem is: how to define the rules for constructing version numbers for these components.

Observations

Top-down propagation

In terms of versions, if component n changes version, then components >n must also change version.
For example, if we fix a bug in the dependencies, then the wheels, which are based on these dependencies, must change version too. And similarly, BlendLuxCore, which is based on these wheels, will have to change version as well.

On the other hand, if we fix a bug in BlendLuxCore, but keep the wheels unchanged, there is no need to version pyluxcore/wheels, and even less need to version the dependencies. To the contrary, we should avoid to stamp new release numbers on a top component (deps, for example), if it remains stable.

So we need a versioning system that is flexible enough to allow us to version a fix in BlendLuxCore without having to change the version of the dependencies.

Python package versioning - standard

Python package versioning is standardized: https://packaging.python.org/en/latest/discussions/versioning/.
Following the standard is not only a recommendation, it is also necessary for the wheel mechanism (pip etc.) to work correctly.

Pypi "no way back"

On Pypi, once a wheel has been published, there is no way to reuse its name, meaning:

• it is not possible to update the wheels under the same version number
• if the wheels are removed, it is even not possible to create new wheels under the already used version number.
  Rules and rationale are exposed here: https://pypi.org/help/#file-name-reuse

Confusing situations

Problem is: we should prevent confusing situations, both for users and developpers, such this one:

• deps: 2.10.1 is latest, but 2.10.0 exists
• pyluxcore: 2.10.3 is latest, based on deps 2.10.1, but 2.10.2, 2.10.1, 2.10.0 exist
• BlendLuxCore: 2.10.4 is latest, based on pyluxcore 2.10.3, but 2.10.3, 2.10.2, 2.10.1, 2.10.0 exist

Proposed rules

Semantic versioning

Historically, LuxCore has followed a major.minor scheme. However, best practices recommend adding a patch (or bugfix) to this scheme.

The result is a 3-part version numbers, major.minor.patch, where the project author increments:

• major when they make incompatible API changes,
• minor when they add functionality in a backwards-compatible manner, and
• patch, when they make backwards-compatible bug fixes.

This is called Semantic Versioning. We propose to adopt this de facto standard.

Major/Minor sharing

As Major.Minor deals with feature evolutions, it seems consistent to share the same Major/Minor between the 3 components.
If needed, some upgrades on deps versions could be searched for each Major/Minor increment, so that version increment does correspond to actual changes in deps (in addition to new features in pyluxcore or BlendLuxCore).

Patch

• For deps: patch = deps_counter, alphabetical
• For pyluxcore and wheels (required to stick to standard): patch = wheel_counter
• For BlendLuxCore : patch = 100*wheels. patch + blc_counter

Example

As an example, we could have:

• deps: 2.10.B
• pyluxcore/wheels: 2.10.3 (internally based on deps 2.10.B)
• BlendLuxCore: 2.10.306 (based on LuxCore Wheels 2.10.3)

Feedback welcome!

[CodeFHD]

I'm gonna pick out a few statements for comment. Everything else, consider accepted.

1) "In terms of versions, if component n changes version, then components >n must also change version."

I think between LuxCoreDeps and LuxCore this is probably right, since LuxCoreDeps is not something that would be used standalone.
When it goes up the ladder to BlendLuxCore, FreeCad etc., this might not necessarily be true, if the changes are not relevant to that plugin. For example, only a specific feature is updated that one plugin does not use.

To go further on this, How often will we even push out releases? In the past, (Blend)LuxCore was patching via nightly builds. Do we continue this at all? Will it be feasible? Probably not if it depends on the changes to pyluxcore being available on PyPi. Alternatively, there must be a simpler method for users to use a local wheel compared to the environment variable.
Perhaps the environment variable could be exchanged for checking if there is a wheel in a specific subfolder of BlendLuxCore. (Spontaneous thought right now while writing this)

2) "it seems consistent to share the same Major/Minor between the 3 components"

This might not work. For BlendLuxCore in particular, we should discuss further how we will be managing the support for Blender versions in the future.
Given that LuxCore is not only intended to be used for Blender (of which you reminded in the other discussion yesterday), it doesn't seem to make sense to align our minor-version to Blender, i.e. we will not be doing Blender 4.2LTS <-> LuxCore v2.10, Blender 4.5LTS <-> LuxCore v2.11, etc. Non-LTS support on top of this, but LTS in particular will have to be maintained for longer in parallel.
Question for our support cycle: Do we keep supporting new features for LTS versions indefinitely? Do we release our own timeline? Do we stop integrating new features into anything but the latest Blender version, LTS or not, and only do bugfixing for the rest?
What will actually be easiest given our limited number of developers: Maintain a bugfix-branch for many different applications (BlendLuxCore version X, BlendLuxCore version Y, LuxFreeCad version Z, ...) for LuxCore and LuxCoreDeps? Or do we stay linear on the C++ code, and any bugfix to an Addon has to use the latest pyluxcore. Right now, I am thinking the latter, because we control our own API changes that have to be adapted for.

Do we even have to reflect the LuxCore version in the BlendLuxCore version at all? Why not base it on Blender directly:
BlendLuxCore major = Blender major:
BlendLuxCore minor = Blender minor
BlendLuxCore patch = blc_counter
The GitHub Release page, our Homepage Download page, etc., would then contain a description which LuxCore feature set is supported.

Or, something rather excessive: Does anything speak against a longer patch counter? As a variation of the above:
BlendLuxCore major = Blender major:
BlendLuxCore minor = Blender minor
BlendLuxCore patch = 1000000LuxCore.major + 10000LuxCore.minor + 100*LuxCore.patch + blc_counter
Example for Blender 4.2LTS: BlendLuxCore 4.2.2100305 (Based on LuxCore Wheels 2.10.3, BlendLuxCore patch 5)

3) "For deps: patch = deps_counter, alphabetical"

I'm not actually opposed to this, I do see the appeal in avoiding confusion with numbers. I just think this is one to consider carefully if we really won't shoot ourselves in the foot in the long-term if our internal semantics are not 100% identical.

[Edited Item 1 because I forgot to finish it]

[howetuft]

Hello,

This a very dense, but most interesting, answer!

Following your manner, I've quoted your points and brought my answers below.

This [sharing Major.Minor] might not work.
Do we even have to reflect the LuxCore version in the BlendLuxCore version at all?
I just think this is one to consider carefully if we really won't shoot ourselves in the foot in the long-term if our internal semantics are not 100% identical.

OK, then we should use pure semver for each component (https://semver.org/), it's well-documented, proven and best practice.

LuxCoreDeps will start from 1.0.0, as required by the standard and, from that starting point, it will follow its way.
LuxCore will go on 2.10, like BlendLuxCore, but living their own lives.

Maintain a bugfix-branch for many different applications (BlendLuxCore version X, BlendLuxCore version Y, LuxFreeCad version Z, ...) for LuxCore and LuxCoreDeps? Or do we stay linear on the C++ code, and any bugfix to an Addon has to use the latest pyluxcore.
Maintain a bugfix-branch for many different applications (BlendLuxCore version X, BlendLuxCore version Y, LuxFreeCad version Z, ...) for LuxCore and LuxCoreDeps?

First of all, we shouldn't mix release and branch subjects.

First thing to set is the release policy.
I think that, at a given moment, we should have, for each component:

• a stable release - suitable for production
• an alpha release - new features, debugged and ready for user-testing, but not guaranted to be perfect.
• some pre-release work

In terms of branches, I would say:

• stable release -> branch master, tag vX.Y.Z
• alpha release -> branch for_vX.Y'.0, tag vX.Y'.0-alpha.A
• pre-release work -> branch for_vX.Y'.0 (+ local branches), at head

If we need to write a hotfix for stable, we should checkout master in a new branch, write the fix in the new branch, merge it to master, make release X.Y.Z' in master, and rebase 'for_vX.Y'.0 branch on master.

Pyluxcore alpha releases should be published on Pypi, under their semver version.

How often will we even push out releases?

Depends on type of release:

• patching: as soon as possible, when we've got a fix for a bug.
• features: I would say 3-6 months, but the right process should be to scope new features, decide what we select for next release, and then plan release.

In the past, (Blend)LuxCore was patching via nightly builds. Do we continue this at all? Will it be feasible?

It's possible, but I think we could/should rather rely on continuous integration.

Do we keep supporting new features for LTS versions indefinitely?
Do we release our own timeline?
Do we stop integrating new features into anything but the latest Blender version, LTS or not, and only do bugfixing for the rest?

Sure, we should follow our own timeline!
And supporting this or that LTS should be a feature decision, for each release. We should not stick to Blender versioning, at least because it would violate semver standard, but also because we don't know what Blender guys can invent (5 LTS at same time?)

As an addition, an important thing I've just recalled:

At the moment, pyluxcore (and subsequently LuxCoreDeps) has Blender as a dependency. It's due to this file:
https://github.com/LuxCoreRender/LuxCore/blob/master/src/luxcore/pyluxcoreforblender.cpp,
Thus a given version of pyluxcore can only target one version of Blender.

If we want pyluxcore to be Blender independent (and we should want it, IMHO), I see 2 solutions:

1. remove pyluxcoreforblender.cpp from pyluxcore.so, and make a separate wheel 'pyluxcoreforblender.so' (with pyluxcore as a dep)
2. carve out the part of pyluxcoreforblender that depends on Blender, rewrite it in Python and include the resulting code in BlendLuxCore

2nd option is my preferred one, for 2 reasons at least:

• the current code is not safe, as it reads and writes directly in Blender internal buffers, makes extensive use of reinterpret_cast etc. and thus is unstable across Blender versions (see LuxCoreRender/LuxCore@bc74024)
• the current code mostly implements array manipulations, which could be much more gracefully coded in Python with numpy.

It's not an urgent matter, but it should be done in next version (I will do it).

[CodeFHD]

OK, then we should use pure semver for each component [...]

OK

First of all, we shouldn't mix release and branch subjects. [...]

Your description of the branch- and tag-workflow sounds good to me.

An implication for maintenance across Blender LTS-versions remains, will we support multiple versions in one branch (lots of if bpy.__version__ < (x, y, z) in the code), or maintain separate versions in parallel.
However, we don't need to discuss this further here in my view, probably out of scope ,of this particular discussion, and I don't see this conflicting with the semver approach. I expect it will boil down to what is most practical depending on how far the Blender API will diverge in the future.

How often will we even push out releases?

Just to clarify my intention, this was rather a philosophical look in the glass ball, along the lines of "what will the ratio between releases (alpha/beta/rc/stable) and nightly builds turn out to be". No need for me to put fixed numbers behind it yet, though your response is undoubtedly reasonable.

It's possible, but I think we could/should rather rely on continuous integration.

See also the next discussion I just opened https://github.com/orgs/LuxCoreRender/discussions/984

At the moment, pyluxcore (and subsequently LuxCoreDeps) has Blender as a dependency. It's due to this file: [...]

Very good point, and I agree that it sounds like a good idea to outsource this.

Regarding doing this all in python: I suspect some of this is performance critical. I do know that it was a topic for custom normals, see e.g. #451
and in the code

BlendLuxCore/export/mesh_converter.py

Line 18 in 7ba0f00

def get_custom_normals_slow(mesh):

It might therefore be better to target a separate C++-based library.

[howetuft]

Thank you for your feedback! I'm gonna have a look at the discussion you point at.
However, I wanted to react to your following statement:

Regarding doing this all in python: I suspect some of this is performance critical. I do know that it was a topic for custom normals, see e.g. #451
and in the code
It might therefore be better to target a separate C++-based library.

I don't agree at all! The current implementation is very naive, but a good, pythonic application of Numpy could make it a lot faster, even faster perhaps than an average implementation in C++. As Numpy usually releases the GIL, there even may be room for multithreading in this computation.

For an example of normals computation with Numpy, please allow me to quote myself:
https://github.com/FreeCAD/FreeCAD-render/blob/master/Render/rendermesh_mixins.py#L648
(just a precision: it's a vertex normal computation, a bit more complex than mesh normal computation).
I love C++, but I wouldn't replace this code by any C++ implementation.

[CodeFHD]

@howetuft Reviewing if this Discussion can be closed or should remain open.

As we have reached v2.10 stable with LuxCoreDeps and LuxCore, and BlendLuxCore in release-candidate state, I would say this proposal has been implemented.

Do we have any open actions here?
Perhaps implementing version numbers for certain subcomponents?
Or adding key points into wiki articles or the main repository README.md documents?

[howetuft]

I don't see any remaining point, so I close.