Skip to content

CROM-6917 Optionally export MEM_SIZE and MEM_UNIT for all standard backends. #6766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

CROM-6917 Optionally export MEM_SIZE and MEM_UNIT for all standard backends. #6766

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 30 additions & 3 deletions backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ import org.apache.commons.lang3.exception.ExceptionUtils
import shapeless.Coproduct
import wom.callable.{AdHocValue, CommandTaskDefinition, ContainerizedInputExpression}
import wom.expression.WomExpression
import wom.format.MemorySize
import wom.graph.LocalName
import wom.values._
import wom.{CommandSetupSideEffectFile, InstantiatedCommand, WomFileMapper}
Expand Down Expand Up @@ -431,9 +432,25 @@ trait StandardAsyncExecutionActor
val errorOrGlobFiles: ErrorOr[List[WomGlobFile]] =
backendEngineFunctions.findGlobOutputs(call, jobDescriptor)

lazy val environmentVariables = instantiatedCommand.environmentVariables map { case (k, v) =>
s"""export $k="$v""""
} mkString ("", "\n", "\n")
lazy val environmentVariables = {
/*
Add `MEM_SIZE` and `MEM_UNIT` before the other environment variables on all backends that define a `memory`
runtime attribute. As of May 2022 some backends may expose these same environment variables via other means where
they are accessible elsewhere, for example within sidecar containers used for resource monitoring.
*/
val memoryEnvironmentVariables: List[(String, String)] =
runtimeMemoryOption.toList.flatMap(runtimeMemory =>
List(
"MEM_SIZE" -> runtimeMemory.amount.toString,
"MEM_UNIT" -> runtimeMemory.unit.toString
)
)
val environmentVariables: List[(String, String)] =
memoryEnvironmentVariables ++ instantiatedCommand.environmentVariables
environmentVariables map { case (key, value) =>
s"""export $key="$value""""
} mkString ("", "\n", "\n")
}

val shortId = jobDescriptor.workflowDescriptor.id.shortString
// Give the out and error FIFO variables names that are unlikely to conflict with anything the user is doing.
Expand Down Expand Up @@ -759,6 +776,16 @@ trait StandardAsyncExecutionActor
lazy val continueOnReturnCode: ContinueOnReturnCode =
RuntimeAttributesValidation.extract(ContinueOnReturnCodeValidation.instance, validatedRuntimeAttributes)

/**
* Returns the memory size for the job.
*
* @return the memory size for the job.
*/
lazy val runtimeMemoryOption: Option[MemorySize] = RuntimeAttributesValidation.extractOption(
runtimeAttributesValidation = MemoryValidation.instance(),
validatedRuntimeAttributes = validatedRuntimeAttributes
)

/**
* Returns the max number of times that a failed job should be retried, obtained by converting `maxRetries` to an Int.
*/
Expand Down
Loading