New use case for xpack

[mmomtchev]

I am exploring a potentially new use case for xpack - as a build system for native Node.js modules.

Let me summarize the problem:

• Node.js has a C++ API and supports the inclusion of native code in npm modules. These typically provide access to large existing complex C++ libraries that cannot be easily rewritten in JS - for example I maintain GDAL and ImageMagick. Since now we have WASM, emnapi and SWIG JSE (of which I am the author), these modules start to also work in the browser and this trend is expected to accelerate.
• When installing such a module, there will be either a precompiled binary for the target platform - or the module will be rebuilt using node-gyp by the npm install script. These modules are typically used by JS developers who expect them to simply work after npm install and who lack any C++ skills. Often, they won't have all the necessary tools installed. Often, they will have half-working versions that they never used.
• node-gyp is a hopelessly outdated system based on gyp which is no longer maintained and it is very difficult to integrate with other build systems. It was not a problem while the average Node.js addon contained some self-contained C++ code but with libraries such as ImageMagick and GDAL - which have a huge number of dependencies - a new build system is needed.

If found xpm by searching if there is someone who is maintaining a launch-and-forget meson-build + ninja npm package. I found much more than that - but alas - using it in its current state will impose xpm on the end-user - who is a web developer and does not have anything to do with C++. So naturally, I am wondering if there is any reason why xpm must be a separate package manager? Isn't it possible to provide most of its functionality via npm install / npm prepare scripts?

[ilg-ul]

I am wondering if there is any reason why xpm must be a separate package manager? Isn't it possible to provide most of its functionality via npm install / npm prepare scripts?

That's a very good question. Given that almost everything can be done with proper scripts, I guess that most of xpm functionality can be replicated via scripts.

For simple use cases this might be an acceptable solution.

However, for more complex use cases, the scripts are expected to be also quite complex, and including them in package.json might not be practical. For reusability reasons too, these scripts must be grouped in a separate package.

Also these scripts should be cross-platform, so probably something like shelljs must be used.

Personally I evaluated this path in the beginning, and found it not practical.

Please note that xpm can do a lot more than installing packages, for example managing multiple build configurations, use properties with substitutions, multi-line actions, etc.

Currently I use xpm to orchestrate multi-architecture multi-platform unit tests, with an insanely large selection of tools. For an example, see a real life package.json.

using it in its current state will impose xpm on the end-user - who is a web developer and does not have anything to do with C++

xpm is a npm package, so you can add it as a devDependency to be installed automatically, and invoke it in npm scripts, so that the end user does not need to know about it.

[ilg-ul]

Node.js has a C++ API and supports the inclusion of native code in npm modules.

My experience with npm modules which have to compile native code is very bad. In an early version of xpm I used some decompression code, but installing xpm was very problematic in some configuration and I had to give up and switch to JS modules.

Personally I cannot recommend using native code modules in public projects.

[mmomtchev]

xpm is a npm package, so you can add it as a devDependency to be installed automatically, and invoke it in npm scripts, so that the end user does not need to know about it.

This might be the path of the least resistance, I will experiment to see how it plays out.

My experience with npm modules which have to compile native code is very bad. In an early version of xpm I used some decompression code, but installing xpm was very problematic in some configuration and I had to give up and switch to JS modules.

Personally I cannot recommend using native code modules in public projects.

This is precisely what needs to change.

Especially since replacing libz is not that hard - but replacing software such as GDAL, ImageMagick and ffmpeg is a problem.

Still, there has been a huge improvement, since the early projects (node-sass was notorious for its compilation problems - you probably have used it). All of my projects - the bindings for GDAL, ImageMagick, ffmpeg and simdjson use node-pre-gyp which simply installs universal statically linked precompiled binaries for Windows x64, Linux x64 and macOS x64. Next step would be the fully self-contained and fully reproducible build - on all platforms.

[ilg-ul]

This is precisely what needs to change.

Yes, good luck with that.

I found running native builds especially difficult (read impossible) if npm requires sudo to install something in the global location, since it also runs the build with elevated rights, and this often leads to unexpected behaviours.

If npm is installed locally, for example with nvm, builds have a higher chance to pass, but still depend on the user environment, if the user has unexpected tools in the PATH, builds are not guaranteed to pass.

The reason I created xpm and the selection of tools in the xpack-dev-tools project is exactly to be able to create a reproducible build environment for my tests.