Low priority improvement: FORCE run with Singularity/Apptainer

[StephenSzwiec]

Observations

I have found that containerized FORCE may fail with the provided commands in documentation, if the user environment variables conflict with those of the container, especially on HPC clusters with non-standard filesystem paths, when Singularity or Apptainer is used.

The default isolation from either container system seems to be insufficient, in practice. A documentation tweak is possibly necessary short-term, with known solution detailed below, while solving the underlying behavior by having the container prepend canonical file locations to its environment a matter of QoL.

Expected behavior

singularity exec force.sif force-info runs and outputs list of available programs without major failures.

Testing/Reproduction

With singularity, building and running leads to failure to produce /home/docker inside the container:

# an older version, singularity is deprecated on my cluster, but used as an example
$ module load singularity/3.8.0 
$ singularity build force.sif docker://davidfrantz/force:latest
INFO:    Starting build...
# [...]
INFO:    Build complete: force.sif
$ singularity exec force.sif force-info
FATAL:   "force-info": executable file not found in $PATH
# diagnostic findings
$ singularity shell force.sif 
Singularity> force-info
bash: force-info: command not found
Singularity> echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/docker/bin
Singularity> echo $LD_LIBRARY_PATH
/usr/lib/R/lib:/.singularity.d/libs
Singularity> ls /home/docker/bin
ls: cannot access '/home/docker/bin': No such file or directory
Singularity> ls /home/docker
ls: cannot access '/home/docker': No such file or directory

apptainer runs of the same container find force-info in the correct location, but default isolation behavior does not link out properly, leading to failures:

$ module load apptainer/1.4.0
$ apptainer exec force.sif force-info 

###############################################################################

Hello stephen.szwiec! You are currently running FORCE v. 3.8.01

Framework for Operational Radiometric Correction for Environmental monitoring
Copyright (C) 2013-2024 David Frantz, david.frantz@uni-trier.de
  + many community contributions (https://github.com/davidfrantz/force)

FORCE is free software under the terms of the GNU General Public License as
  published by the Free Software Foundation, see http://www.gnu.org/licenses/

Thank you for using FORCE! This software is being developed in the hope that it
  will be helpful for you and your work.

However, it is requested that you to use the software in accordance with
  academic standards and fair usage. Without this, software like FORCE will not
  survive. This includes citation of the software and the scientific publica-
  tions, proper acknowledgement in any public presentation, or an offer of co-
  authorship of scientific articles in case substantial help in setting up,
  modifying or running the software is provided by the author(s).

At minimum, the citation of following paper is requested:
  Frantz, D. (2019): FORCE — Landsat + Sentinel-2 Analysis Ready Data and
  Beyond. Remote Sensing, 11, 1124

Each FORCE module will generate a "CITEME" file with suggestions for references
 to be cited. This list is based on the specific parameterization you are using

Usefule resources:
- Code, issues, discussions etc.: https://github.com/davidfrantz/force
- Documentation and tutorials: https://force-eo.readthedocs.io
- Docker images: https://hub.docker.com/r/davidfrantz/force/

##########################################################################

available programs:

force-cube: external command (gdalinfo) not found, terminating...
force-cube-init: Initialize a datacube definition
force-datacube-size: Display the size of your datacube
force-higher-level: Higher level processing (compositing, time series analysis, ...)
force-hist: Compute image histogram
force-import-modis: Import MODIS surface reflectance daily
force-info: Entrypoint, print short disclaimer, available modules etc.
force-init: Initialization of a new project
force-l2ps: Level 2 processing of single image
force-level1-csd: Cloud Storage Downloader
usage: force-level1-landsat [-h] [-v] {search,download} ...
force-level1-landsat: error: unrecognized arguments: -i
force-level2: external command (lockfile-create) not found, terminating...
force-level2-report: external command (R) not found, terminating...
force-lut-modis: Generation and maintenance of water vapor database using MODIS products
force-magic-parameters: Replace variables in parameterfile for bulk processing
During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_COLLATE failed, using "C"
3: Setting LC_TIME failed, using "C"
4: Setting LC_MESSAGES failed, using "C"
5: Setting LC_MONETARY failed, using "C"
6: Setting LC_PAPER failed, using "C"
7: Setting LC_MEASUREMENT failed, using "C"
force-map-accuracy: Compute map accuracy and area statistics
force-mdcp: Copy FORCE metadata from one file to another
force-mosaic: external command (gdalinfo gdalbuildvrt /home/docker/bin/force-mdcp) not found, terminating...
force-parameter: Generation of parameter files
force-procmask: external command (gdal_calc.py parallel) not found, terminating...
force-pyramid: external command (parallel) not found, terminating...
force-qai-inflate: Inflate QAI bit layers
During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_COLLATE failed, using "C"
3: Setting LC_TIME failed, using "C"
4: Setting LC_MESSAGES failed, using "C"
5: Setting LC_MONETARY failed, using "C"
6: Setting LC_PAPER failed, using "C"
7: Setting LC_MEASUREMENT failed, using "C"
force-sample-size: Suggest sample size for estimating area and accuracy of a classification
force-stack: Stack images, works with 4D data model
force-stratified-sample: Compute image histogram
usage: force-synthmix parameterFile
force-synthmix: error: the following arguments are required: parameterFile
force-tabulate-grid: Extract the data cube grid to shapefile
force-tile-extent: external command (/home/docker/bin/force-cube) not found, terminating...
force-tile-finder: Find the tile, pixel, and chunk of a given coordinate
force-train: Train (and validate) Machine Learning models
force-unit-testing: Run unit tests (DevOps only, not meant for users)

##########################################################################

This behavior persists after rebuilding with apptainer, as well. I use the default name to delineate:

$ apptainer shell force.sif
Singularity> ls /home/docker
bin  udf
Singularity> echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/docker/bin
Singularity> echo $LD_LIBRARY_PATH
/usr/lib/R/lib:/.singularity.d/libs
Singularity> exit
$ apptainer pull docker://davidfrantz/force:latest
INFO:    Using cached SIF image
$ apptainer shell ./force_latest.sif
Apptainer> ls /home/docker
bin  udf
Apptainer> echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/docker/bin
Apptainer> echo $LD_LIBRARY_PATH
/usr/lib/R/lib:/.singularity.d/libs
Apptainer> find /usr -name gdalinfo
/usr/bin/gdalinfo
/usr/share/bash-completion/completions/gdalinfo
Apptainer> /usr/bin/gdalinfo
ERROR 1: dataset_name: 1 argument(s) expected. 0 provided.
Usage: gdalinfo [--help] [--long-usage] [--help-general]
                [-json] [-mm]
                [[-stats]|[-approx_stats]]
                [-hist]
                [-nogcp] [-nomd] [-norat] [-noct] [-nofl] [-nonodata] [-nomask]
                [-checksum] [-listmdd] [-proj4]
                [-wkt_format <WKT1|WKT2|WKT2_2015|WKT2_2018|WKT2_2019>]
                [-sd <n>] [-oo <NAME>=<VALUE>]... [-if <format>]...
                [-mdd <domain>|all]
                <dataset_name>

Finally, I also tested with the develop flavored force:dev tagged container: while home location changes made to Dockerfile in the development branch prevent singularity from failing to find force-info in $PATH, it still fails similarly to apptainer.

Immediate Solution

For either singularity or apptainer, running with both --no-mount rootfs and --cleanenv solves this issue:

$ apptainer run --no-mount hostfs --cleanenv ./force_latest.sif
# [...] 
available programs:

force-cube: Ingestion of auxiliary data into datacube format
force-cube-init: Initialize a datacube definition
force-datacube-size: Display the size of your datacube
force-higher-level: Higher level processing (compositing, time series analysis, ...)
force-hist: Compute image histogram
force-import-modis: Import MODIS surface reflectance daily
force-info: Entrypoint, print short disclaimer, available modules etc.
force-init: Initialization of a new project
force-l2ps: Level 2 processing of single image
force-level1-csd: Cloud Storage Downloader
usage: force-level1-landsat [-h] [-v] {search,download} ...
force-level1-landsat: error: unrecognized arguments: -i
force-level2: Level 2 processing of image archive (bulk processing)
force-level2-report: Generate Level 2 processing report
force-lut-modis: Generation and maintenance of water vapor database using MODIS products
force-magic-parameters: Replace variables in parameterfile for bulk processing
force-map-accuracy: Compute map accuracy and area statistics
force-mdcp: Copy FORCE metadata from one file to another
force-mosaic: Mosaicking of image chips
force-parameter: Generation of parameter files
force-procmask: Processing masks from raster images
force-pyramid: Compute image pyramids
force-qai-inflate: Inflate QAI bit layers
force-sample-size: Suggest sample size for estimating area and accuracy of a classification
force-stack: Stack images, works with 4D data model
force-stratified-sample: Compute image histogram
usage: force-synthmix parameterFile
force-synthmix: error: the following arguments are required: parameterFile
force-tabulate-grid: Extract the data cube grid to shapefile
force-tile-extent: Compute processing extent based on AOI
force-tile-finder: Find the tile, pixel, and chunk of a given coordinate
force-train: Train (and validate) Machine Learning models
force-unit-testing: Run unit tests (DevOps only, not meant for users)

Suggestions and Requests for Comments

I am unsure if enough users are using FORCE on systems, such as HPC systems, which re-produce this behavior for --no-mount hostfs --cleanenv to be the suggested invocation in the documentation; perhaps a small environment wrapper or change to the Dockerfile is more appropriate here. I would be happy to contribute a PR for any of the above after consideration/discussion.

Setup

• Tested FORCE versions: docker tags force:latest [v3.8.01] and force:dev [v3.8.01-dev:::2025-09-23_12:49:36]
• Tested containerization software: apptainer v1.4.0, singularity v3.8.0
• OS and version: Red Hat Enterprise Linux 9.2 (Plow) x86_6
• Bare metal Linux provisioned with OpenHPC, using GPFS
• Attempts were made to stay within the confines of documented method
• AMD EPYC 7532 (32) @ 2.395GHz; 256447MiB memory

Please let me know if more information is necessary. Thank you for your time and consideration.