Skip to content

document OCI-mode SCIF support #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 17, 2023
Merged

document OCI-mode SCIF support #222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions build_a_container.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,8 @@ containers. The containers are decrypted at runtime entirely in kernel space,
meaning that no intermediate decrypted data is ever written to disk. See
:ref:`encrypted containers <encryption>` for more details.

.. _dockerfile:

*************************
Building from Dockerfiles
*************************
Expand Down Expand Up @@ -339,11 +341,11 @@ in the expected way.
.. code:: console

$ singularity exec --oci ./debian.oci.sif uname -a
Linux nueve 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 GNU/Linux
Linux myhost 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 GNU/Linux

$ singularity shell --oci ./debian.oci.sif uname
Singularity> uname -a
Linux nueve 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 GNU/Linux
Linux myhost 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 GNU/Linux
Singularity>

.. note::
Expand Down
7 changes: 7 additions & 0 deletions definition_files.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1191,6 +1191,13 @@ containers.
{Singularity} implements SCIF, and you can read more about how to use it
below.

.. note::

The OCI mode introduced in version 4 of {Singularity} also includes support
for SCIF, but in a way that is in line with the behavior of SCIF in other OCI
container runtimes (e.g. Docker). See the :ref:`documentation on SCIF in
OCI-mode <oci_scif>` for more information.

SCIF is not specific to {Singularity}. To learn more, take a look at the
project's site at https://sci-f.github.io/, which includes extended
tutorials, a detailed specification of the SCIF standard, and other
Expand Down
155 changes: 155 additions & 0 deletions oci_runtime.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,161 @@ files:
definition JSON files. If omitted,
default will be: /etc/cdi,/var/run/cdi

.. _oci_scif:

****************
SCIF in OCI mode
****************

`SCIF <https://sci-f.github.io/>`__ is a standard for encapsulating multiple
apps into a container. Support for SCIF in the native runtime is discussed
:ref:`here <apps>`; but the behavior of SCIF in OCI-mode is different, and is in
line with how SCIF is used in other OCI container runtimes, such as Docker, as
discussed & demonstrated in `this SCIF tutorial
<https://sci-f.github.io/tutorial-preview-install>`__.

In brief, SCIF in OCI containers relies on the container having the `scif
executable <https://pypi.org/project/scif/>`__ as its CMD / ENTRYPOINT, as shown
for example in this Dockerfile:

.. code:: console

$ cat Dockerfile.scif
FROM continuumio/miniconda3
RUN pip install scif
ADD my_recipe /
RUN scif install /my_recipe
CMD ["scif"]

.. note::

Starting with version 4.1, {Singularity} includes support for building OCI-SIF
images directly from Dockerfiles, and so a Dockerfile like the one above can
be compiled directly into an OCI-SIF image. (In this particular case, the
``my_recipe`` file would have to be present in the current directory and be a
well-formed SCIF recipe.) See :ref:`here <dockerfile>` for details on building
OCI-SIF images from Dockerfiles, and see the `SCIF documentation
<https://sci-f.github.io/tutorial-preview-install>`__ for more information on
SCIF recipes.

The main difference between SCIF support in native- and OCI-modes is the
location of the SCIF "recipe" (``%appinstall``, ``%appenv``, ``%apprun``,
``%apphelp`` and ``%applabels`` sections). In native mode, the SCIF recipe is
:ref:`part of the {Singularity} definition file <apps>`. In OCI mode, on the
other hand, the SCIF recipe is typically included in a separate file, and
processed using the ``scif install <recipefile>`` command inside the container,
to be executed *after* the ``scif`` executable has been installed (in this case,
using ``pip``).

Including the SCIF recipe as a separate file is not the only option, however.
The SCIF recipe file can be constructed on-the-fly as part of the OCI container
build, as well, as in the following example:

.. code::

$ cat Dockerfile.scif2
FROM continuumio/miniconda3

RUN pip install scif

RUN echo $'\n\
%apprun hello-world-one\n\
echo "'Hello world!'"\n\
\n\
%apprun hello-world-two\n\
echo "'Hello, again!'"\n\
' > /my_recipe

RUN scif install /my_recipe

CMD ["scif"]

Once you have built a SCIF-compliant OCI-SIF image, you can use {Singularity}'s
``--app`` option to interact with individual SCIF apps in the container using
the ``run / shell / exec`` commands:

.. code::

$ cat Dockerfile.scif
FROM continuumio/miniconda3
RUN pip install scif
ADD my_recipe /
RUN scif install /my_recipe
CMD ["scif"]

$ cat my_recipe
%appenv hello-world-echo
THEBESTAPP=$SCIF_APPNAME
export THEBESTAPP
%apprun hello-world-echo
echo "The best app is $THEBESTAPP"

%appinstall hello-world-script
echo "echo 'Hello World!'" >> bin/hello-world.sh
chmod u+x bin/hello-world.sh
%appenv hello-world-script
THEBESTAPP=$SCIF_APPNAME
export THEBESTAPP
%apprun hello-world-script
/bin/bash hello-world.sh

$ singularity build --oci scif.oci.sif Dockerfile.scif
INFO: Did not find usable running buildkitd daemon; spawning our own.
INFO: cfg.Root for buildkitd: /home/myuser/.local/share/buildkit
INFO: Using "crun" runtime for buildkitd daemon.
INFO: running buildkitd server on /run/user/1000/buildkit/buildkitd-8508905943414043.sock
[+] Building 1.8s (8/9)
[+] Building 1.9s (9/9) FINISHED
=> [internal] load build definition from Dockerfile.scif 0.0s
=> => transferring dockerfile: 206B 0.0s
=> [internal] load metadata for docker.io/continuumio/miniconda3 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/4] FROM docker.io/continuumio/miniconda3:latest@sha256:db9 0.0s
=> => resolve docker.io/continuumio/miniconda3:latest@sha256:db9 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 89B 0.0s
=> CACHED [2/4] RUN pip install scif 0.0s
=> CACHED [3/4] ADD my_recipe / 0.0s
=> CACHED [4/4] RUN scif install /my_recipe 0.0s
=> exporting to docker image format 1.2s
=> => exporting layers 0.0s
=> => exporting manifest sha256:5fa6d77d3e0f9190088d57782bbe52dc 0.0s
=> => exporting config sha256:a9ffa234dd97432b0bc74fa3ee7fa46bfd 0.0s
=> => sending tarball 1.2s
Getting image source signatures
Copying blob e67fdae35593 done |
Copying blob 62aa66a9c405 done |
Copying blob 129bc9a4304f done |
Copying blob 9eeb7d589f05 done |
Copying blob d4ef55d3a44b done |
Copying blob 81edcff80a6f done |
Copying config 2f162fba3f done |
Writing manifest to image destination
INFO: Converting OCI image to OCI-SIF format
INFO: Squashing image to single layer
INFO: Writing OCI-SIF image
INFO: Cleaning up.
INFO: Build complete: scif.oci.sif

$ singularity run --oci --app hello-world-script scif.oci.sif
[hello-world-script] executing /bin/bash /scif/apps/hello-world-script/scif/runscript
Hello World!

$ singularity exec --oci --app hello-world-script scif.oci.sif env | grep APPDATA
SCIF_APPDATA=/scif/data/hello-world-script
SCIF_APPDATA_hello_world_script=/scif/data/hello-world-script
SCIF_APPDATA_hello_world_echo=/scif/data/hello-world-echo

$ singularity shell --oci --app hello-world-script scif.oci.sif
[hello-world-script] executing /bin/bash
myuser@myhost:/scif/apps/hello-world-script$ echo $SCIF_APPNAME
hello-world-script
myuser@myhost:/scif/apps/hello-world-script$

See the `SCIF homepage <https://sci-f.github.io/>`__ for more information and
links to further documentation on SCIF itself.

.. _oci_command:

*****************
Expand Down