Add --source-file-name option to cap_data_download

[toniecrumley]

Summary

Some URLs don't end with an appropriate filename, especially those based on querystring parameters. The --source-file-name option allows a source filename to be provided.

Author Checklist

• This repository has a Lasseigne Lab ruleset configured.
  See instructions here.
• Only files relevant to this change are included; no temporary, generated,
  or unrelated files are part of this pull request. Use .gitignore to prevent
  irrelevant files from accidentally being included.
• All automated tests have passed.
• All automated code checks have passed. The "All checks have passed"
  message should show in the GitHub pull request status box above the
  "Merge pull request" button.
• No merge conflicts exist. Merge conflicts are indicated by the "This
  branch has conflicts that must be resolved" message in the GitHub pull
  request status box above the "Merge pull request" button.
• The README has been updated with documentation changes.
• The command script help documentation in the commands directory
  has been updated if necessary.
• The CHANGELOG has been updated with the planned version number.
• Setup instructions have been provided.
• Test instructions have been provided.
• Cleanup instructions have been provided.
• At least two reviewers have been requested to review this pull request.

Self/Peer Review Checklist (Coding Guidelines)

• Meaningful variable and function names
• File header comments
• Function comments
• In-line comments summarize logical sections of code by concisely explaining
  why, not what the code is doing. Avoid excessive or redundant commenting.
• Confirm that the automated tests pass.
• Confirm the code performs the intended functionality

Setup

Get the code for the pull request.

cd ~/bin/capture
git checkout main
git pull
git checkout download-file-name
cd project-template
git checkout main
git pull
git checkout version-1-0
cd ..

Test

Run the automated test suite.

cd ~/bin/capture
tests/install
tests/run

Test manually in scratch:

cd $USER_SCRATCH
cap new --owner lasseignelab "$USER-testing"

cd "$USER-testing"

Download files with and without the --source-file-name option. Examples below:

#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=16G
#SBATCH --cpus-per-task=1
#SBATCH --time=1:00:00
#SBATCH --partition=medium

cap_data_download \
  --unzip \
  --source-file-name "GSE185948_count_RNA.rds.gz" \
  "https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE185948&format=file&file=GSE185948%5Fcount%5FRNA%2Erds%2Egz"

cap_data_download \
  --source-file-name "foo.csv.gz" \
  --unzip \
  "https://ftp.ncbi.nlm.nih.gov/geo/series/GSE185nnn/GSE185948/suppl/GSE185948_metadata_RNA.csv.gz"

Cleanup

• Remove the test directory from scratch:

cd "$USER_SCRATCH"
rm -rf "$USER-testing"

• Delete the test repo from GitHub.
• Reset CAPTURE to the released version.

cap update

[github-actions]

✅⚠️MegaLinter analysis: Success with warnings

Descriptor	Linter	Files	Fixed	Errors	Warnings	Elapsed time
⚠️ BASH	bash-exec	32		29	0	0.34s
✅ BASH	shellcheck	32		0	0	1.14s
⚠️ BASH	shfmt	32		23	0	0.02s
⚠️ MARKDOWN	markdownlint	3		63	0	0.77s
✅ MARKDOWN	markdown-table-formatter	3		0	0	0.29s

Detailed Issues

⚠️ BASH / bash-exec - 29 errors

Results of bash-exec linter (version 5.3.3)
See documentation on https://megalinter.io/9.3.0/descriptors/bash_bash_exec/
-----------------------------------------------

✅ [SUCCESS] cap
❌ [ERROR] cap_completion.sh
    Error: File:[cap_completion.sh] is not executable

❌ [ERROR] commands/env.sh
    Error: File:[commands/env.sh] is not executable

❌ [ERROR] commands/help.sh
    Error: File:[commands/help.sh] is not executable

❌ [ERROR] commands/md5.sh
    Error: File:[commands/md5.sh] is not executable

❌ [ERROR] commands/new.sh
    Error: File:[commands/new.sh] is not executable

❌ [ERROR] commands/run.sh
    Error: File:[commands/run.sh] is not executable

❌ [ERROR] commands/update.sh
    Error: File:[commands/update.sh] is not executable

❌ [ERROR] commands/verify.sh
    Error: File:[commands/verify.sh] is not executable

❌ [ERROR] commands/version.sh
    Error: File:[commands/version.sh] is not executable

❌ [ERROR] configure.sh
    Error: File:[configure.sh] is not executable

❌ [ERROR] install.sh
    Error: File:[install.sh] is not executable

❌ [ERROR] lib/environment.sh
    Error: File:[lib/environment.sh] is not executable

❌ [ERROR] lib/environment_functions.sh
    Error: File:[lib/environment_functions.sh] is not executable

❌ [ERROR] lib/functions.sh
    Error: File:[lib/functions.sh] is not executable

❌ [ERROR] lib/functions/cap_array_value.sh
    Error: File:[lib/functions/cap_array_value.sh] is not executable

❌ [ERROR] lib/functions/cap_container.sh
    Error: File:[lib/functions/cap_container.sh] is not executable

❌ [ERROR] lib/functions/cap_data_download.sh
    Error: File:[lib/functions/cap_data_download.sh] is not executable

❌ [ERROR] lib/functions/commands/cap_check_environment.sh
    Error: File:[lib/functions/commands/cap_check_environment.sh] is not executable

❌ [ERROR] lib/functions/commands/cap_root_required.sh
    Error: File:[lib/functions/commands/cap_root_required.sh] is not executable

❌ [ERROR] lib/functions/verify/cap_verify_append.sh
    Error: File:[lib/functions/verify/cap_verify_append.sh] is not executable

❌ [ERROR] lib/functions/verify/cap_verify_md5.sh
    Error: File:[lib/functions/verify/cap_verify_md5.sh] is not executable

❌ [ERROR] tests/fixtures/env/.caprc
    Error: File:[tests/fixtures/env/.caprc] is not executable

❌ [ERROR] tests/fixtures/run/array_job.sh
    Error: File:[tests/fixtures/run/array_job.sh] is not executable

❌ [ERROR] tests/fixtures/run/job.sh
    Error: File:[tests/fixtures/run/job.sh] is not executable

❌ [ERROR] tests/fixtures/verify/verifications/test_cap_verify_md5.sh
    Error: File:[tests/fixtures/verify/verifications/test_cap_verify_md5.sh] is not executable

❌ [ERROR] tests/fixtures/verify/verifications/test_dry_run.sh
    Error: File:[tests/fixtures/verify/verifications/test_dry_run.sh] is not executable

❌ [ERROR] tests/fixtures/verify/verifications/test_name.sh
    Error: File:[tests/fixtures/verify/verifications/test_name.sh] is not executable

❌ [ERROR] tests/fixtures/verify/verifications/test_output_file.sh
    Error: File:[tests/fixtures/verify/verifications/test_output_file.sh] is not executable

✅ [SUCCESS] tests/install
✅ [SUCCESS] tests/run
❌ [ERROR] tests/setup_suite.bash
    Error: File:[tests/setup_suite.bash] is not executable

⚠️ MARKDOWN / markdownlint - 63 errors

.github/pull_request_template.md:1 error MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "## Summary"]
.github/pull_request_template.md:6:21 error MD059/descriptive-link-text Link text should be descriptive [Context: "[here]"]
.github/pull_request_template.md:38 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
.github/pull_request_template.md:51 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
.github/pull_request_template.md:58 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
.github/pull_request_template.md:63 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
.github/pull_request_template.md:69 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
.github/pull_request_template.md:76 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
CHANGELOG.md:1 error MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "## CAPTURE 1.0.0 (January xx, ..."]
README.md:1 error MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "CAPTURE"]
README.md:28 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:33 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:38 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:46 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:55 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:74 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:78 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:129 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:181 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:195 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:207 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:225 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:247 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:281 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:303 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:346 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:387 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:391 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:419 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:438 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:447 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:451 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:461 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:472 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:477 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:486 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:515 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:524 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:534 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:556 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:578 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:583:72 error MD026/no-trailing-punctuation Trailing punctuation in heading [Punctuation: ':']
README.md:587 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:594 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:613 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:617 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:630 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:649 error MD024/no-duplicate-heading Multiple headings with the same content [Context: "Examples for a verification na..."]
README.md:649:72 error MD026/no-trailing-punctuation Trailing punctuation in heading [Punctuation: ':']
README.md:652 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:656 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:663 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:667 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:672 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:676 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:697 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:710 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:715 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:724 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:728 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:734 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:741 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:746 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]

⚠️ BASH / shfmt - 23 errors

diff cap.orig cap
--- cap.orig
+++ cap
@@ -10,14 +10,14 @@
 
 # Load commands
 for file in "$CAP_INSTALL_PATH"/commands/*.sh; do
-    # shellcheck disable=SC1090
-    source "$file"
+	# shellcheck disable=SC1090
+	source "$file"
 done
 
 # Check if a parameter was provided
 if [ "$#" -eq 0 ]; then
-    cap_help
-    exit 1
+	cap_help
+	exit 1
 fi
 
 # Retrieve the function name from the first parameter
@@ -25,14 +25,14 @@
 namespaced_name="cap_$function_name"
 
 # Check if the function exists
-if declare -f "$namespaced_name" > /dev/null; then
-    # Extract parameters starting from the 2nd one
-    params=("${@:2}")
-
-    # Call the function with the extracted parameters
-    $namespaced_name "${params[@]}"
+if declare -f "$namespaced_name" >/dev/null; then
+	# Extract parameters starting from the 2nd one
+	params=("${@:2}")
+
+	# Call the function with the extracted parameters
+	$namespaced_name "${params[@]}"
 else
-    echo "Unknown command: $function_name"
-    echo "Usage: $0 {md5|sha256}"
-    exit 1
+	echo "Unknown command: $function_name"
+	echo "Usage: $0 {md5|sha256}"
+	exit 1
 fi
diff cap_completion.sh.orig cap_completion.sh
--- cap_completion.sh.orig
+++ cap_completion.sh
@@ -2,33 +2,33 @@
 
 # Function to handle tab completion for CAPTURE
 _cap_completion() {
-  local current subcommand subcommands
-  current="${COMP_WORDS[COMP_CWORD]}"
-  subcommand="${COMP_WORDS[1]}"
-  subcommands="env help md5 new run update verify version"
-
-  if [[ "${COMP_CWORD}" -eq 1 ]]; then
-    # Show the cap subcommand list.
-    mapfile -t COMPREPLY < <(compgen -W "${subcommands}" -- "${current}")
-  else
-    case "$subcommand" in
-      "help")
-        # Show the cap subcommand list.
-        if [[ "${COMP_CWORD}" == 2 ]]; then
-          mapfile -t COMPREPLY < <(compgen -W "${subcommands}" -- "${current}")
-        fi
-        ;;
-      "md5"|"run"|"verify")
-        # Tab compete with file and directory names.
-        compopt -o default -o plusdirs
-        ;;
-      *)
-        # Don't tab complete.
-        COMPREPLY=()
-    esac
-  fi
+	local current subcommand subcommands
+	current="${COMP_WORDS[COMP_CWORD]}"
+	subcommand="${COMP_WORDS[1]}"
+	subcommands="env help md5 new run update verify version"
+
+	if [[ "${COMP_CWORD}" -eq 1 ]]; then
+		# Show the cap subcommand list.
+		mapfile -t COMPREPLY < <(compgen -W "${subcommands}" -- "${current}")
+	else
+		case "$subcommand" in
+		"help")
+			# Show the cap subcommand list.
+			if [[ "${COMP_CWORD}" == 2 ]]; then
+				mapfile -t COMPREPLY < <(compgen -W "${subcommands}" -- "${current}")
+			fi
+			;;
+		"md5" | "run" | "verify")
+			# Tab compete with file and directory names.
+			compopt -o default -o plusdirs
+			;;
+		*)
+			# Don't tab complete.
+			COMPREPLY=()
+			;;
+		esac
+	fi
 }
 
 # Register the completion function for "mytool"
 complete -F _cap_completion cap
-
diff commands/env.sh.orig commands/env.sh
--- commands/env.sh.orig
+++ commands/env.sh
@@ -1,16 +1,16 @@
 #!/bin/bash
 
 cap_env_description() {
-  cat <<EOF
+	cat <<EOF
   Displays CAPTURE environment variables.
 EOF
 }
 
 cap_env_help() {
-  cap_env_description
-  echo
-
-  cat <<EOF
+	cap_env_description
+	echo
+
+	cat <<EOF
   The "env" command displays a list of all the CAPTURE environment variables
   along with their values. File and path values are displayed with full
   paths. This command must be executed from the project root directory.
@@ -40,68 +40,70 @@
 }
 
 cap_env() {
-  cap_root_required "env"
-  cap_env_parse_commandline_parameters "$@"
-
-  # Snapshot the incoming environment variables. The cap_env_before
-  # variable is created before the first snapshot so it will be in
-  # the before and after snapshots. Otherwise, it shows up in the
-  # cap_env_diff below.
-  local cap_env_before=""
-  cap_env_before="$(cap_env_snapshot)"
-
-  # Load the CAPTURE environment.
-  if [ -n "$environment_override" ]; then
-    CAP_ENVIRONMENT="$environment_override"
-  fi
-  source "$CAP_INSTALL_PATH/lib/environment.sh"
-  if [ -n "$environment_override" ]; then
-    CAP_ENVIRONMENT="$environment_override"
-  fi
-
-  # Snapshot after the CAPTURE environment as loaded.
-  cap_env_after="$(cap_env_snapshot)"
-
-  # Determine list of variables created during environment loading.
-  cap_env_diff="$(
-    comm -13 \
-      <(printf '%s\n' "$cap_env_before") \
-      <(printf '%s\n' "$cap_env_after")
-  )"
-
-  # Display CAPTURE environment variables.
-  echo
-  for v in $cap_env_diff; do
-    printf '%s=%q\n' "$v" "${!v}"
-  done
-  echo
+	cap_root_required "env"
+	cap_env_parse_commandline_parameters "$@"
+
+	# Snapshot the incoming environment variables. The cap_env_before
+	# variable is created before the first snapshot so it will be in
+	# the before and after snapshots. Otherwise, it shows up in the
+	# cap_env_diff below.
+	local cap_env_before=""
+	cap_env_before="$(cap_env_snapshot)"
+
+	# Load the CAPTURE environment.
+	if [ -n "$environment_override" ]; then
+		CAP_ENVIRONMENT="$environment_override"
+	fi
+	source "$CAP_INSTALL_PATH/lib/environment.sh"
+	if [ -n "$environment_override" ]; then
+		CAP_ENVIRONMENT="$environment_override"
+	fi
+
+	# Snapshot after the CAPTURE environment as loaded.
+	cap_env_after="$(cap_env_snapshot)"
+
+	# Determine list of variables created during environment loading.
+	cap_env_diff="$(
+		comm -13 \
+			<(printf '%s\n' "$cap_env_before") \
+			<(printf '%s\n' "$cap_env_after")
+	)"
+
+	# Display CAPTURE environment variables.
+	echo
+	for v in $cap_env_diff; do
+		printf '%s=%q\n' "$v" "${!v}"
+	done
+	echo
 }
 
 cap_env_parse_commandline_parameters() {
-  # Define the named commandline options
-  if ! OPTIONS=$(getopt -o e: --long environment: -- "$@"); then
-    echo "Use the 'cap help env' command for detailed help."
-    exit 1
-  fi
-  eval set -- "$OPTIONS"
-
-  # Set default values for the named parameters
-  environment_override=""
-
-  # Parse the optional named command line options
-  while true; do
-    case "$1" in
-      -e|--environment)
-        environment_override=$2
-        shift 2 ;;
-      --)
-        shift
-        break;;
-    esac
-  done
+	# Define the named commandline options
+	if ! OPTIONS=$(getopt -o e: --long environment: -- "$@"); then
+		echo "Use the 'cap help env' command for detailed help."
+		exit 1
+	fi
+	eval set -- "$OPTIONS"
+
+	# Set default values for the named parameters
+	environment_override=""
+
+	# Parse the optional named command line options
+	while true; do
+		case "$1" in
+		-e | --environment)
+			environment_override=$2
+			shift 2
+			;;
+		--)
+			shift
+			break
+			;;
+		esac
+	done
 }
 
 # Snapshot current environment variables.
 cap_env_snapshot() {
-  compgen -v | sort
+	compgen -v | sort
 }
diff commands/help.sh.orig commands/help.sh
--- commands/help.sh.orig
+++ commands/help.sh
@@ -1,16 +1,16 @@
 #!/bin/bash
 
 cap_help_description() {
-  cat <<EOF
+	cat <<EOF
   Shows help for the cap command line tool.
 EOF
 }
 
 cap_help_help() {
-  cap_help_description
-  echo
-
-  cat <<EOF
+	cap_help_description
+	echo
+
+	cat <<EOF
   The "help" command will display help for all the commands available for the
   cap command.
 
@@ -32,11 +32,11 @@
 }
 
 cap_help() {
-  echo
-
-  # Check if a parameter was provided
-  if [ "$#" -eq 0 ]; then
-    cat <<EOF
+	echo
+
+	# Check if a parameter was provided
+	if [ "$#" -eq 0 ]; then
+		cat <<EOF
   Usage: cap COMMAND ...
 
   Commands:
@@ -45,36 +45,36 @@
   COMMAND
 EOF
 
-    # Directory containing the scripts
-    CAP_COMMANDS_DIR="$CAP_INSTALL_PATH"/commands
-
-    {
-      # Loop through each script file in the directory
-      for script in "$CAP_COMMANDS_DIR"/*.sh; do
-        # Get the base name of the script (e.g., md5 for md5.sh)
-        script_name=$(basename "$script" .sh)
-
-        # Construct the function name
-        description_function="cap_${script_name}_description"
-
-        # Check if the function exists
-        if declare -f "$description_function" > /dev/null; then
-          printf '    %s:' "$script_name"
-
-          # Call the function
-          "$description_function"
-        else
-          echo "    Error:  Function $description_function not found in $script"
-        fi
-      done
-    } | column -t -s ':'
-  else
-    # Retrieve the command name from the first parameter
-    command_name=$1
-    # Construct the function name
-    help_function="cap_${command_name}_help"
-    # Call the function
-    "$help_function" | less -FX
-  fi
-  echo
+		# Directory containing the scripts
+		CAP_COMMANDS_DIR="$CAP_INSTALL_PATH"/commands
+
+		{
+			# Loop through each script file in the directory
+			for script in "$CAP_COMMANDS_DIR"/*.sh; do
+				# Get the base name of the script (e.g., md5 for md5.sh)
+				script_name=$(basename "$script" .sh)
+
+				# Construct the function name
+				description_function="cap_${script_name}_description"
+
+				# Check if the function exists
+				if declare -f "$description_function" >/dev/null; then
+					printf '    %s:' "$script_name"
+
+					# Call the function
+					"$description_function"
+				else
+					echo "    Error:  Function $description_function not found in $script"
+				fi
+			done
+		} | column -t -s ':'
+	else
+		# Retrieve the command name from the first parameter
+		command_name=$1
+		# Construct the function name
+		help_function="cap_${command_name}_help"
+		# Call the function
+		"$help_function" | less -FX
+	fi
+	echo
 }
diff commands/md5.sh.orig commands/md5.sh
--- commands/md5.sh.orig
+++ commands/md5.sh
@@ -1,16 +1,16 @@
 #!/bin/bash
 
 cap_md5_description() {
-  cat <<EOF
+	cat <<EOF
   Calculates a combined MD5 checksum for one or more files.
 EOF
 }
 
 cap_md5_help() {
-  cap_md5_description
-  echo
-
-  cat <<EOF
+	cap_md5_description
+	echo
+
+	cat <<EOF
   The "md5" command produces an MD5 checksum for each file specified and a
   combined MD5 checksum for all the files. The purpose of this command is to
   determine whether files downloaded or created are complete and accurate. If
@@ -99,21 +99,21 @@
 }
 
 cap_md5() {
-  cap_md5_parse_commandline_parameters "$@"
-
-  # Submit to slurm or run immediately
-  case "$slurm" in
-    batch)
-      # Create a temporary script and run it as a Slurm batch job.
-      current_path=$(pwd)
-      if [[ "$output_file" == "" ]]; then
-        output_file='cap-md5-%j.out'
-      fi
-
-      # Prepare a temporary script for running the command in Slurm.
-      # The temporary file was introduced to make the code testable by BATS.
-      temp_batch_script=$(mktemp)
-      cat <<EOF > "$temp_batch_script"
+	cap_md5_parse_commandline_parameters "$@"
+
+	# Submit to slurm or run immediately
+	case "$slurm" in
+	batch)
+		# Create a temporary script and run it as a Slurm batch job.
+		current_path=$(pwd)
+		if [[ "$output_file" == "" ]]; then
+			output_file='cap-md5-%j.out'
+		fi
+
+		# Prepare a temporary script for running the command in Slurm.
+		# The temporary file was introduced to make the code testable by BATS.
+		temp_batch_script=$(mktemp)
+		cat <<EOF >"$temp_batch_script"
 #!/bin/bash
 
 #################################### SLURM ####################################
@@ -127,62 +127,62 @@
 
 cap md5 $slurm_args "${md5_files[@]}"
 EOF
-      if [[ "$output_files_only" == "false" ]]; then
-        cat <<EOF >> "$temp_batch_script"
+		if [[ "$output_files_only" == "false" ]]; then
+			cat <<EOF >>"$temp_batch_script"
 echo "Ran from: $current_path"
 EOF
-      fi
-      sbatch "$temp_batch_script"
-
-      ;;
-    run)
-      # Prepare a temporary script for running the command in Slurm.
-      # The temporary file was introduced to make the code testable by BATS.
-      temp_run_script=$(mktemp)
-      cat <<EOF > "$temp_run_script"
+		fi
+		sbatch "$temp_batch_script"
+
+		;;
+	run)
+		# Prepare a temporary script for running the command in Slurm.
+		# The temporary file was introduced to make the code testable by BATS.
+		temp_run_script=$(mktemp)
+		cat <<EOF >"$temp_run_script"
 cap md5 $slurm_args ${md5_files[@]}
 EOF
 
-      srun \
-        --job-name=cap-md5 \
-        --ntasks=1 \
-        --cpus-per-task=1 \
-        --mem=32G \
-        --output="${output_file:-/dev/stdout}" \
-        --input="$temp_run_script" \
-        --export=ALL \
-        bash
-      ;;
-    *)
-      local temp_output_file
-      temp_output_file=$(mktemp)
-      {
-        if [[ "$dry_run" == "true" ]]; then
-          # shellcheck disable=SC2068
-          find -H -L ${md5_files[@]} "${ignore_filter[@]}" \( "${select_filter[@]}" \) -type f ! -path '*/\.*' | sort
-        else
-          # Compute checksums for all files
-          [[ "$output_files_only" == "false" ]] && echo -e '\nFiles included:'
-          checksums=$(cap_md5_find)
-          echo "$checksums"
-
-          # Compute single checksum based on the checksums of all files
-          if [[ "$output_files_only" == "false" ]]; then
-            echo -e '\nCombined MD5 checksum:'
-            echo "$checksums" | cut -d ' ' -f1 | md5sum | cut -d ' ' -f1
-            echo
-          fi
-        fi
-      } > "$temp_output_file"
-      if [[ "$normalize" == "true" && "$dry_run" == "false" ]]; then
-        cap_md5_normalize "$temp_output_file"
-      fi
-      if [[ "$append" == "false" ]]; then
-        : > "$output_file"
-      fi


(Truncated to 13333 characters out of 79536)

See detailed reports in MegaLinter artifacts

You could have the same capabilities but better runtime performances if you use a MegaLinter flavor:

• oxsecurity/megalinter/flavors/documentation@v9.3.0 (51 linters)
• oxsecurity/megalinter/flavors/ruby@v9.3.0 (52 linters)
• oxsecurity/megalinter/flavors/rust@v9.3.0 (52 linters)

Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)

• Documentation: Custom Flavors
• Command: npx mega-linter-runner@9.3.0 --custom-flavor-setup --custom-flavor-linters BASH_EXEC,BASH_SHELLCHECK,BASH_SHFMT,MARKDOWN_MARKDOWNLINT,MARKDOWN_MARKDOWN_TABLE_FORMATTER

[tsoelter]

I'm not sure if I like that if the file name includes the .gz extension but the --unzip argument is included, the provided file name does not match the downloaded file, since it got unzipped. I was initially confused and thought the --file-name option wasn't working, but it was, it just got affected by the unzip option. Is there a way to only provide the file name without any extensions that are affected by other arguments? Not sure if this made sense, but I can clarify in person if necessary. I'm also not sure how easy or difficult it would be to implement this.

[tchowton]

I'm not sure if I like that if the file name includes the .gz extension but the --unzip argument is included, the provided file name does not match the downloaded file, since it got unzipped. I was initially confused and thought the --file-name option wasn't working, but it was, it just got affected by the unzip option. Is there a way to only provide the file name without any extensions that are affected by other arguments? Not sure if this made sense, but I can clarify in person if necessary. I'm also not sure how easy or difficult it would be to implement this.

I think it is fine because the --unzip flag was used. I could see a world where someone copies and pastes the file name and they want it unzipped so they use the flag and forget to take the .gz off of the end of the file name. Providing the --unzip flag seems like a conscious decision.

[toniecrumley]

I'm not sure if I like that if the file name includes the .gz extension but the --unzip argument is included, the provided file name does not match the downloaded file, since it got unzipped. I was initially confused and thought the --file-name option wasn't working, but it was, it just got affected by the unzip option. Is there a way to only provide the file name without any extensions that are affected by other arguments? Not sure if this made sense, but I can clarify in person if necessary. I'm also not sure how easy or difficult it would be to implement this.

@tchowton @tsoelter After we all discussed and then discussed with Chat some more, we decided to use --source-file-name to clarify this issue.