BlendluxCore: Development and Release Workflow

[CodeFHD]

Introduction

We already discussed the new release mechanism using wheels and a new versioning strategy, see also https://github.com/orgs/LuxCoreRender/discussions/983.

In this discussion, I would like to dive a step further, into the implementation in BlendLuxCore and the implications for an efficient workflow during all development stages (that is not just serving my personal taste 😉).

For context: During the last couple of months, we made use of two environment variables BLC_WHEEL_PATH and BLC_DEV_PATH for this purpose.

Scenarios

I divide this discussion into a few different application scenarios and discuss their individual requirements and proposed solutions.
"Normal Use" and "Development" are the most frequent, daily applications,
while "Feature testing by Users" and "Bug Fixing with User Feedback" are less frequent, but have unique requirements.

Normal Use

In normal use, BlendLuxCore is installed as bundled.

• The user installs the wheel using Blender's "install extension" feature.
• The code is placed and executed in Blenders extension folder.
• The pyluxcore wheel is downloaded from pip.
• The exact version of pyluxcore is fixed in a variable inside __init__.py. I propose to call this variable _PYLUXCORE_VERSION.

This scenario seems clear and straightforward.

Development

To facilitate (time-)efficient development, it is desirable to reduce the number of build- and installation processes.
We can subdivide into several steps:

1. Development of the BlendLuxCore python-code (w/o change to pyluxcore)

   • This step, so far, seems solved well using the BLC_DEV_PATH environment variable.
   • Having a local .bat/.sh file on PATH, it is possible to quickly toggle between the local-repository-code and whatever official release wheel is installed.

2. Development of LuxCore (Feature development or bug fixing alike)

   • The C++ source is compiled with the minimum build target spec for the developers local machine using make pyluxcore. This builds only the library.
   • Alternatively, a full wheel is built using the GitHub action and nektos/act.
   • Local compilation is expected to be much quicker than nektos/act, currently seconds vs. minutes. (Optimization of the action for local compilation was not explored.)
   • My current solution: having local .bat/.sh script that copies and replaces the compiled pyluxcore library into the site-packages/ folder.
   • Initial testing suggests that this works, as the file hash (which is stored in the RECORD file) is not checked.
   • An open item is a robust mechanism that will make sure the correct library is re-installed afterwards. Ideally, this happens automatically and in a fail-safe manner in case Blender does not exit gracefully (common e.g. when dealing with a segfault bug)

3. Testing of a pyluxcore wheel before release.

   • This step should involve the full wheel-building action and is generally solved by the current BLC_WHEEL_PATH mechanism.

4. Testing of a BlendLuxCore wheel, incl. corresponding pyluxcore version, before a release.

   • This step should be executed using the regular installation (cf. Normal Use) and thus presents no issue.
   • The release of pyluxcore on PyPi should be executed first, of course.

Feature Testing by Users

For feature testing, two paths are foreseen:

1. Alpha/beta/rc releases

• These will be released as a BlendLuxCore-wheel on GitHub together with a pyluxcore-wheel on PyPi, so the regular process (cf. Normal Use) applies.

2. Nightly builds:

• For getting early feedback from users and involving our more enthusiastic user base, it shall be possible to obtain and use development versions of the code.

  • Pushing all test versions to PyPi is not a solution, so the mechanism around BLC_WHEEL_PATH must be accessible to, and easy enough for less programming-affine users.

• For testing BlendLuxCore only, without changes to pyluxcore, I would say this is solved by the already established "bundle_latest.yml" workflow.

  • The wheel is given to users and they install it (cf. Normal Use)

• For changes to pyluxcore, I propose to offer a second mechanism as an alternative to the BLC_WHEEL_PATH as an environment variable.

  • We reserve a local folder name under BlendLuxCore/
  • For example, to be discussed: BlendLuxCore/pyluxcore_custom/
  • The user downloads the latest pyluxcore and places it in this folder.
  • If a wheel is present in this folder, this wheel is installed.
  • Pointing users to this folders is easy as a standard installation of Blender will point to well-known paths. (relative to ~/ or %APPDATA%, only the Blender version number is individual)

Bug Fixing with User Feedback

This case is essentially the same as was presented for nightly builds. However, I want to separate it to stress a particular importance to make this efficient:
Issue LuxCoreRender/LuxCore#645 can be considered as a recent example. This bug is hardware- and driver-specific and thus not reproducible for active developers. We must thus rely on support and feedback by the reporting user, other LuxCore-community members or even maintainers of external dependencies. In the example, the user obviously has advanced programming knowledge, knows LuxCore already, and is motivated to investigate C++ code themselves. This can not generally be expected.
If we wish to get as much support as possible, we should strive to make it easy for users.

Proposed Implementation in BlendLuxCore

_PYLUXCORE_VERSION

• This variable will primarily be set to a specific version, which will then be downloaded from pip.
• Alternatively, it can be set to "LATEST". In this case, the latest version is downloaded using pip.
• The use of BLC_WHEEL_PATH will override this variable as described below.

BLC_WHEEL_PATH

For this variable I propose the following order of priority:

1. BLC_WHEEL_PATH == "SKIP":

   • This is used when a compiled library is directly copied into site-packages. Everything else is then skipped as the correct version is already there. This optimizes startup performance.

2. BLC_WHEEL_PATH is otherwise set with a local path to a wheel:

   • That wheel is installed

3. BLC_WHEEL_PATH is not set, but a wheel is found in BlendLuxCore/pyluxcore_custom/

   • That wheel is installed

4. BLC_WHEEL_PATH is not set, and no wheel is found in BlendLuxCore/pyluxcore_custom/

   • The version specified by _PYLUXCORE_VERSION is installed via pip.

An open issue with this approach is the downgrading of wheels. Currently, the exact wheel version is updated each time in blender_manifest.toml, but Blender does not perform a downgrade.
We must find a way to properly enforce this to avoid the mistake of testing the wrong wheel. This applies particularly during bugfixing where the version doesn't change.

A further performance optimization would be to compare the currently installed pyluxcore against the expected version.
Currently, I observe that BlendLuxCore dominates the startup time of Blender because some form of pip download is always executed and the process takes some time even for the local wheel.
This step might involve hashes of the library file.

BLC_DEV_PATH

• The variable BLC_DEV_PATH is used last.
• If defined, all BlendLuxCore code is inported from the specified path.
• The exception to this is the module-level __init__.py itself and all functions defined therein.
• The actions defined by BLC_WHEEL_PATH and _PYLUXCORE_VERSION are executed before this happens, therefore the __init__.py in the Blender extension folder has to be edited directly for testing changes to those mechanisms.

Open Items and discussion points

• Decide if we should change any of the variable names to (better) follow naming conventions and clarify their purpose
  • BLC_WHEEL_PATH --> BLC_PYLUXCORE_WHEEL_PATH ? --> Kept name for now
  • BLC_DEV_PATH --> BLC_GIT_REPOSITORY_PATH ? --> Kept name for now
  • _PYLUXCORE_VERSION --> leading underscore or not ? --> No
• Find a robust mechanism for up-/down-/cross-grading wheels when necessary.
• Include the replacement of only the pyluxcore-library file in the above mechanism.
  • No issues observed on LInux to date
  • Windows/macOS not tested
• Decide the name of the sub-folder where pyluxcore_wheels are searched. E.g. BlendLuxCore/pyluxcore_custom/
• Optimize the startup performance by avoiding unnecessary calls to pip download.

[howetuft]

Hello,

This is a good job! I mostly agree with all above. A few comments though:

1. core LuxCore debugging can also be done with the 'samples' programs, ie luxcoreui and/or luxcoreconsole. That way, you're not dependent from Blender and you're not slowed down by the wheel building, wheel install process and so on. You just write your code, make luxcoreui, run your test model and... iterate (unless you're this kind of guy who kill bugs with only one shot ;-)).

2. Optimisation of the action.

Optimization of the action for local compilation was not explored

Well, that's not totally true. There are some cache techniques employed to speed up the build of the wheels. However, properly building a wheel requires to build completely (and in a container, for Linux version) and to repair it. That is what makes the wheels capable to run in all environments; but the counterpart is a somewhat long build time....

[CodeFHD]

Thanks for the feedback, much shorter than I expected on this long article so I guess that is good ;)

No, I am not that one-shot guy, sadly. True about luxcoreconsole/ui. One of the things I thought of while working on it this week but didn't put in this text in the end. From my experience, probably depends on the kind of feature and its implementation stage what is practical. Early implementation of a new engine in C++, then luxcoreconsole with a static scene to see if it works at all. Late stage of a shader, Blender, because you can vary parameters live and not by text file.

Regarding the optimization of the action, I should probably amend my statement to

Optimization of the action for local compilation was not explored by me
to clarify that it is rather an observation of the current state rather than an unsolvable fact.

P.S.: Done some slight formatting to the bullet lists, to better group items belonging together. Had to finish up earlier and didn't double check if everything looks as intended.

[CodeFHD]

The process has been implemented and tested on my side.

The performance optimization to avoid unnecessary downloads is achieved by writing small text file pyluxcore_installation.txt in the extensions root_folder.
This file contains an information depending on which installation path was chosen
- BLC_WHEEL_PATH: hash of the wheel
- _PYLUXCORE_VERSION != None: this variable
- _PYLUXCORE_VERSION == None: the latest version determined from PyPi
Only one variable will be written each time to implicitly indicate which installation method was followed.

Another important step I had to realize was missing to clear the wheels/ folder when installing a different version. If multiple version were there, all were put in the manifest and because of automatic alphanumeric order, only the latest was picked, preventing downgrades.

As a failsave, I am now moving the files into wheels_backup/ first. Then, if the new download succeeds, this backup is deleted. Else, it is moved back.

What I have not yet done is teh renaming of the environment variables as suggested in the original post. Any comments on that?

Further testing during upcoming development is also needed regarding the swapping of only the pyluxcore binary library, but I believe the current performance optimisation skip will cover this.

[howetuft]

I've read more carefully your proposal, and I add here a few comments:

Variable types

It's not clear what kind of variables/constants they are. I will assume:

_PYLUXCORE_VERSION - Python constant
BLC_WHEEL_PATH - Environment variable
BLC_DEV_PATH - Environment variable

Just tell me if I'm wrong.

_PYLUXCORE_VERSION

There shouldn't be a possible "LATEST" value. A given BlendLuxCore version should be locked to an explicit PyLuxCore version, and that's all for me.

The rationale is:

1. we should not update pyluxcore without user explicit consent (provided, for instance, that some upgrades may sometimes unexpectedly break the app...)
2. this can lead to a big gap between PLC version and BLC version, if the system keeps upgrading PLC while user does not upgrade BLC, and also break the app.

If we need to upgrade the PLC version used in BlendLuxCore, the answer is to make a new (sub)version of BLC.
For instance, let's say BLC version is 2.10.14 and it is locked to PLC 2.10.1, and let's say we are to release PLC 2.10.2 (bug fixing, for instance).
If we want to use PLC 2.10.2 in BLC, we just have to release BLC 2.10.15, witha suitable _PYLUXCORE_VERSION="2.10.2".

I would also remove the leading underscore in _PYLUXCORE_VERSION.

BLC_WHEEL_PATH

I'm not comfortable with your "SKIP" value:

• Shell way to indicate a variable should be skipped is not to set it (or unset it, if needed)
• Pythonic way to do the same is to set it to None.
  I think we should stick to these common practices.

BLC_WHEEL_PATH = os.getenv("BLC_WHEEL_PATH")
...
if BLC_WHEEL_PATH:
    <do the right thing>

Hacked wheels

It may be possible to add a feature to the build system to craft a hacked wheel from a pyluxcore lib, even if that pyluxcore has not been compiled under the conditions of a wheel. This could be invoked, for example, by make wheel-dev.
Then, the installation of the targeted pyluxcore could rely on the BLC_WHEEL_PATH mechanism only.
I'm not 100% sure, but I can explore it if it makes sense for you.

In particular, I'm not fully comfortable with the pyluxcore_custom folder, which seems to duplicate with BLC_WHEEL_PATH in my mind (and may be a security hole?). Relying only on BLC_WHEEL_PATH could be a solution.

Blender extension commands

Note that Blender has some interesting CLI commands to handle extensions:

https://docs.blender.org/manual/en/latest/advanced/command_line/extension_arguments.html#extensions-command-line-arguments

They might be interesting to automate some install / debug tasks?

[CodeFHD]

Thanks for the feedback. My responses back:

It's not clear what kind of variables/constants they are. I will assume:

Yes, you understood correctly.

_PYLUXORE_VERSION: There shouldn't be a possible "LATEST" value

For clarification of my intent: This was one idea for handling the nightly build phase. For any release of BlendLuxCore, be it an alpha/beta/rc or stable release, it should be a fixed number as you said.

However, now that I have worked a little more with my initial implementation, I've come to think that this is unneccessary anyways. Since pyluxcore won't be released to PyPi in a nightly build fashion, there will rarely be something "LATEST" for this mechanism to install.

Hence, no objection to removing this.

(Side note: I proposed "LATEST" here but I actually implemented it with None, cf. your parapgraph about BLC_WHEEL_PATH.)

I would also remove the leading underscore in _PYLUXCORE_VERSION.

OK

BLC_WHEEL_PATH: I'm not comfortable with your "SKIP" value

Again for clarification: The main intention behind this suggestion was to have a mechanism to make sure we never call "pip download ...", because I observed that this significantly affects the startup time of blender. Even when the same version was already downloaded and installed before.
Setting this variable to None alone wouldn't provide sufficient information, since this is already an alternative to the download from PyPi, so the instruction to skip both must be explicit.

Again, my initial implementation already contained an alternative that worked well for me so far: The mechnanism via the (temporarily and badly named) pyluxcore_installation.txt. You didn't explicitly comment on any of that post, so would be interested in any comments as well.

So if you agree to saving this information from last startup, the "BLC_WHEEL_PATH=SKIP" method could be removed.

It may be possible to add a feature to the build system to craft a hacked wheel from a pyluxcore lib.
[...]
I'm not 100% sure, but I can explore it if it makes sense for you.

If it is just for me, we could hold off on that for the moment. The process of directly swapping the lib worked well on linux (as rehearsed during my BiDir-debugging), and on Windows it might be worth to understand the existing issues (replacement of lib not working, act not working) first before introducing yet another way.

In particular, I'm not fully comfortable with the pyluxcore_custom folder, which seems to duplicate with BLC_WHEEL_PATH in my mind (and may be a security hole?)

Security holes are always to be taken serious. What mechanism do you have in mind here that any attacker who has acquired he rights to copy a file there could not achieve with the other existing mechanisms? (E.g. setting the environment variable (temporarily or permanently), replacing the lib directly just like I did, replacing the init.py with one that calls a download command, ...)

Relying only on BLC_WHEEL_PATH could be a solution

For ourselves, no problem. I am still thinking of involving the average user, and especially have any easy way for those who are not used to working with a terminal, like many people on Windows an Mac won't be. Pointing them to the right folder is easy.

Note that Blender has some interesting CLI commands to handle extensions
They might be interesting to automate some install / debug tasks?

I don't see anyting right now. Blender seems to handle the wheel installation well, i.e. when the blender_manifest.toml is updated, then it installs the new wheel, else it doesn't do anything.
I did not yet come across any situation that would require more than that.

[howetuft]

You didn't explicitly comment on any of that post,

Yes, sorry, not enough bandwidth (and wanted to close some topics on my side: dependencies certificate, especially). Saving the state in a file is a simple and robust solution. Even though you might prefer some structured format like json, over plain text. Question however: isn't there already a config file attached to BLC? If so, wouldn't it be preferable to merge our information into this file? If not, isn't it the opportunity to create such a file? Another question: does Blender not provide such a mechanism of property saving, via its API?

The process of directly swapping the lib worked well on linux

OK, let's keep it.

What mechanism do you have in mind here that any attacker who has acquired he rights to copy a file there could not achieve with the other existing mechanisms?

Well, not a precise one. It just creates a general opportunity to inject code, but it may not be worse than other existing ones.

I don't see anyting right now.

No problem, that was just prospective.

[CodeFHD]

Update: I am so far satisfied with the general workflow and don't see any further conceptual actions.

What remains to sort out is the process on Windows of using freshly compiled binaries in BlendLuxCore (or the pyluxcore wheel in general).

macOS possibly as well, but unclear due to lack of dedicated, active developer.

[CodeFHD]

Update with the release of BlendLuxCore v2.10.0:
Carry the following item into a v2.11 roadmap:

• Check Blender's system for user writable directories
• Find out how it operates in order to incorporate into instructions
• Unify with the offline installation system (if feasible)