Builder Design

[skant7]

Here we are going to define the design of builder

Goals:

• implement a CLI with a separated API which provides a way to generate compact Docker images with metacall runtime
• it must be run under rootless and daemonless inside a Dockerized environment
• it has to be efficient and sandboxed producing compact images both in terms of size and dependencies

[viferga]

I am going to detail here all the images with dependencies, so we define a standard way to name them and also the meaning of each one. I am going to do this in Python.

First of all, we have the base image which is debian-slim, the version can be configurable but right now we are using debian:bullseye-slim.

Then we have the dev kind of image. This kind of images are used for developing the core itself (i.e building it):

• dev-deps-base: It includes dependencies like gcc, git, ... (https://github.com/metacall/core/blob/77f5a0e74d304ac69578f43bf47ede55e9a3063e/tools/metacall-environment.sh#L62), and also the cloned repo of MetaCall. It is the base of all dev images, needed for building any loader.
• dev-deps-py: This includes the dependencies for building py_loader.so (https://github.com/metacall/core/blob/77f5a0e74d304ac69578f43bf47ede55e9a3063e/tools/metacall-environment.sh#L92).
• dev-py: This is the previous image but with py_loader.so built.

Those images inherit one from each other, and we have the last two repeated for each language (they are independent between languages).

Another kind of images, which are independent to the compilation of metacall/core itself are the runners. Those do not need development dependencies (in terms of metacall/core development) but they need end user development dependencies. For example, in Python, they will need python3 and pip3 in order to install dependencies and run python code, but it won't need python3-dev package, instead it will be enough with python3-lib which is already included by python3. This is needed in order to have a minimal image in which to run installation scripts (ci/cd) of the end user projects. They do not need to inherit from dev-deps-base (usually). There's some exceptions, like for example, sometimes NodeJS has requirements like electron which need a compiler in order to install them, similar can happen in python although I haven't seen that in the same frequency.

Those images will be used as intermediate images in order to build the final image of the end user. They will execute pip install ... or npm install ... during "end-user build time" (i.e during when we are creating the end user images, not the images of metacall/builder itself).

Once we have all those images, we need the runtime images. Runtime images are for executing code based on metacall and they are based on debian-slim. For this we need the following (Python):

1. Runtime dependencies of Python: https://github.com/metacall/core/blob/77f5a0e74d304ac69578f43bf47ede55e9a3063e/tools/metacall-runtime.sh#L62
2. MetaCall runtime dependencies: py_loader.so

The 2) point can be taken from this operation: diff(dev-deps-py, dev-py) and excluding the build folder, because it will have .o files related to the compilation step.

The dev images are not really important for the end user, maybe we can find some use case for metacall/core development but in my opinion we can leave them for now... for development we can use the tools in metacall/core and compile it with multiple languages at once.

I am trying to write the build phase in docker syntax but it is not possible due to the limitations, maybe I can draw some graph instead. But basically we must have runtime images with only the dependencies for runtime, i.e a debian-slim with py_loader.so + libpython3. For each language merge then and copy the end user dependencies. This can be the first approach, avoiding runners. Once we have it, we can add runners by adding an intermediate phase where runners are executed, and the dependencies of the user are copied from the runners instead.

[skant7]

This design looks good to me, however I have few questions :

1. Does the dev-deps-base image needs to be built initially or does it gets built during building the loaders phase.
2. How do we decide the end user deps for each language or is it just minimal something like pip to mock the runners ?
3. Does the runner script change runtime dependencies or does it stay the same i.e ( libpy_loader.so + libpython) ?

[viferga]

Buildkit in deamonless+rootless mode (for testing):

docker run \
    -it \
    --rm \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    -e BUILDKITD_FLAGS=--oci-worker-no-process-sandbox \
    -v /path/to/dir:/tmp/work \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:master-rootless \
        build \
        --frontend \
        dockerfile.v0 \
        --local context=/tmp/work \
        --local dockerfile=/tmp/work

[viferga]

Check out how Moby creates his images: https://github.com/moby/buildkit/blob/354e4f6e6b2008a6690b1675279e6bd2e445d492/Dockerfile#L86

[ashpect]

Brief Documentation of Builder Design :

The base image is debian-slim. (You can specify the base image using the --image flag)
The other images currently supported are of dev, deps and runtime images. The specifics are mentioned in the above comment here

Here py is an example for python, you can pass multiple args to builder py c etc. as languages for building required image.

Deps images :

• deps base : contains the base for any image, basically base image + metacall cloned + dependencies like gcc, make etc.
• deps base py : deps base + dependencies for building py_loader.so

Dev image :

• dev base py : deps base + [ metacall_configure(py) + metacall_build ] (this builds the loaders for python)
• dev base. : des base + [metacall_configure("") + metacall_build ] (everything except the python loaders)

Taking diff(deps base, deps base py) will give us the dev dependencies for python.
and taking diff(runtime base, runtime py) will give us the runtime dependencies for python.

So, to form the final runtime image containing the python loaders as well as the python runtime.
We do merge(runtime base py, diff(dev base, dev base py) - the second part here i,e diff(dev py, dev base) are taken after removing the build folder in each to avoid conflicts in diffs if any. It then finally will basically give you the loaders for python.

[viferga]

@ashpect it would be interesting if you can document this, and we convert this issue into documentation so people can understand how it works internally. Once that's done, we can close this.