[ABLD-28] Add macOS build for AArch64/ARM64 #37676
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
team/agent-devx
team/agent-delivery
medium review
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
4 times, most recently
from
65be186 to
ec66bb6
Compare
rdesgroppes
changed the title
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
from
ec66bb6 to
de8c275
Compare
Contributor
Gitlab CI Configuration ChangesModified Jobs.lint_macos_gitlab .lint_macos_gitlab:
before_script:
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- export AWS_RETRY_MODE=standard
- export AWS_RETRY_MAX_ATTEMPTS=5
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
'
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- python3 -m pip install "git+https://github.com/DataDog/datadog-agent-dev.git@v$(cat
.dda/version)" --break-system-packages
- pyenv rehash
- python3 -m dda self dep sync -f legacy-tasks
- pyenv rehash
- DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
token)" || exit $?; export DD_API_KEY
- DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
token)" || exit $?; export DD_APP_KEY
- "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
.[0-9]+\\.(x|[0-9]+)$ ]]; then\n dda inv -- -e macos.report-versions -l all ||\
\ true\nfi\n"
- "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n echo Trying to remove inactive versions\n\
\ dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
\ || true\n dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
\ || true\nfi\n"
- dda inv -- -e rtloader.make
- dda inv -- -e rtloader.install
- dda inv -- -e install-tools
needs:
- go_deps
- go_tools_deps
+ rules:
+ - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
+ when: never
+ - when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache_tools.tar.xz
- dda inv -- -e linter.go --cpus 12 --debug --timeout 60
stage: lint.tests_macos_gitlab .tests_macos_gitlab:
+ after_script:
+ - export VAULT_ADDR=https://vault.us1.ddbuild.io
+ - vault login -method=aws -no-print
+ - 'export PATH="$(pyenv root)/shims:$PATH"
+
+ eval "$(pyenv init -)"
+
+ eval "$(pyenv virtualenv-init -)"
+
+ '
+ - "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
+ \ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
+ \ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
+ \ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
+ \ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
+ \ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
+ \ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
+ \ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
+ \\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
+ \\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
+ Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
+ \ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
+ \n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
+ \\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
+ \ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
+ \ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
+ \ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
+ fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
+ \n"
+ - echo "Using Python $PYTHON_VERSION..."
+ - "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
+ \ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
+ \ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
+ \ already exists. Skipping creation.\"\nfi\n"
+ - pyenv activate $VENV_NAME
+ - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
+ - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
+ $?; export CODECOV_TOKEN
+ - dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit: '**/junit-out-*.xml'
when: always
before_script:
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- export AWS_RETRY_MODE=standard
- export AWS_RETRY_MAX_ATTEMPTS=5
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
'
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- python3 -m pip install "git+https://github.com/DataDog/datadog-agent-dev.git@v$(cat
.dda/version)" --break-system-packages
- pyenv rehash
- python3 -m dda self dep sync -f legacy-tasks
- pyenv rehash
- DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
token)" || exit $?; export DD_API_KEY
- DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
token)" || exit $?; export DD_APP_KEY
- "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
.[0-9]+\\.(x|[0-9]+)$ ]]; then\n dda inv -- -e macos.report-versions -l all ||\
\ true\nfi\n"
- "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n echo Trying to remove inactive versions\n\
\ dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
\ || true\n dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
\ || true\nfi\n"
- dda inv -- -e rtloader.make
- dda inv -- -e rtloader.install
- dda inv -- -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache_tools.tar.xz
- dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
- dda inv -- -e invoke-unit-tests
stage: source_test
variables:
FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
TEST_OUTPUT_FILE: test_output.jsondeploy_packages_dmg-x64-a7 deploy_packages_dmg-x64-a7:
before_script:
- ls $OMNIBUS_PACKAGE_DIR
image: registry.ddbuild.io/ci/datadog-agent-buildimages/gitlab_agent_deploy$CI_IMAGE_GITLAB_AGENT_DEPLOY_SUFFIX:$CI_IMAGE_GITLAB_AGENT_DEPLOY
needs:
- agent_dmg-x64-a7
rules:
- if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
script:
- $S3_CP_CMD --recursive --exclude "*" --include "datadog-agent-7*.dmg" $OMNIBUS_PACKAGE_DIR
- $S3_RELEASE_ARTIFACTS_URI/dmg/x86_64/
? ^^^ ^^
+ $S3_RELEASE_ARTIFACTS_URI/dmg/$PACKAGE_ARCH/
? ^^^^^^^^ ^^^^
stage: deploy_packages
tags:
- arch:amd64
+ variables:
+ PACKAGE_ARCH: x86_64lint_macos_gitlab_arm64 lint_macos_gitlab_arm64:
before_script:
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- export AWS_RETRY_MODE=standard
- export AWS_RETRY_MAX_ATTEMPTS=5
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
'
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- python3 -m pip install "git+https://github.com/DataDog/datadog-agent-dev.git@v$(cat
.dda/version)" --break-system-packages
- pyenv rehash
- python3 -m dda self dep sync -f legacy-tasks
- pyenv rehash
- DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
token)" || exit $?; export DD_API_KEY
- DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
token)" || exit $?; export DD_APP_KEY
- "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
.[0-9]+\\.(x|[0-9]+)$ ]]; then\n dda inv -- -e macos.report-versions -l all ||\
\ true\nfi\n"
- "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n echo Trying to remove inactive versions\n\
\ dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
\ || true\n dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
\ || true\nfi\n"
- dda inv -- -e rtloader.make
- dda inv -- -e rtloader.install
- dda inv -- -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- - if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
+ - when: on_success
- - allow_failure: true
- when: manual
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache_tools.tar.xz
- dda inv -- -e linter.go --cpus 12 --debug --timeout 60
stage: lint
tags:
- macos:ventura-arm64
- specific:truetests_macos_gitlab_arm64 tests_macos_gitlab_arm64:
after_script:
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
$?; export CODECOV_TOKEN
- dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- export AWS_RETRY_MODE=standard
- export AWS_RETRY_MAX_ATTEMPTS=5
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
'
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- python3 -m pip install "git+https://github.com/DataDog/datadog-agent-dev.git@v$(cat
.dda/version)" --break-system-packages
- pyenv rehash
- python3 -m dda self dep sync -f legacy-tasks
- pyenv rehash
- DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
token)" || exit $?; export DD_API_KEY
- DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
token)" || exit $?; export DD_APP_KEY
- "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
.[0-9]+\\.(x|[0-9]+)$ ]]; then\n dda inv -- -e macos.report-versions -l all ||\
\ true\nfi\n"
- "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n echo Trying to remove inactive versions\n\
\ dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
\ || true\n dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
\ || true\nfi\n"
- dda inv -- -e rtloader.make
- dda inv -- -e rtloader.install
- dda inv -- -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
+ - if: $RUN_UNIT_TESTS == "off"
when: never
- - allow_failure: true
- when: manual
+ - if: $CI_COMMIT_BRANCH == "main"
+ variables:
+ COVERAGE_CACHE_FLAG: --push-coverage-cache
+ FAST_TESTS: 'false'
+ - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
+ variables:
+ COVERAGE_CACHE_FLAG: ''
+ FAST_TESTS: 'false'
+ - if: $CI_COMMIT_TAG != null
+ variables:
+ COVERAGE_CACHE_FLAG: ''
+ FAST_TESTS: 'false'
+ - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
+ variables:
+ COVERAGE_CACHE_FLAG: ''
+ FAST_TESTS: 'false'
+ - if: $RUN_UNIT_TESTS == "on"
+ variables:
+ COVERAGE_CACHE_FLAG: ''
+ FAST_TESTS: 'false'
+ - variables:
+ COVERAGE_CACHE_FLAG: --pull-coverage-cache
+ FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
|| exit 101
- rm -f modcache_tools.tar.xz
- dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
- dda inv -- -e invoke-unit-tests
stage: source_test
tags:
- macos:ventura-arm64
- specific:true
variables:
FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
TEST_OUTPUT_FILE: test_output.jsonAdded Jobs.agent_dmg.agent_dmg:
after_script:
- "if [ \"$SIGN\" = true ]; then\n security delete-keychain \"build.keychain\"\
\ || true\nfi\n"
- sudo umount /Volumes/Agent || true
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg/*.dmg
- omnibus/pkg/version-manifest.json
before_script:
- sudo umount /Volumes/Agent || true
- rm -rf "$OMNIBUS_GIT_CACHE_DIR" || true
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION
paths:
- omnibus/vendor/bundle
needs:
- go_mod_tidy_check
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: $COMPARE_TO_BRANCH
paths:
- comp/core/gui/guiimpl/systray/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: $COMPARE_TO_BRANCH
paths:
- omnibus/**/*
- .gitlab-ci.yml
- release.json
- .gitlab/package_build/**/*
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
- if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
== "true" || $DDR_WORKFLOW_ID != null
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- set -eo pipefail
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- python3 -m pip install "git+https://github.com/DataDog/datadog-agent-dev.git@v$(cat
.dda/version)" --break-system-packages
- pyenv rehash
- python3 -m dda self dep sync -f legacy-tasks
- pyenv rehash
- echo Setting up Go
- mkdir -p ~/go
- export GO_VERSION="$(cat .go-version)"
- eval "$(gimme $GO_VERSION)"
- export PATH="$PATH:$GOROOT/bin"
- echo Go version should be $GO_VERSION
- go version
- dda inv check-go-version
- DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
token)" || exit $?; export DD_API_KEY
- DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
token)" || exit $?; export DD_APP_KEY
- "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
.[0-9]+\\.(x|[0-9]+)$ ]]; then\n dda inv -- -e macos.report-versions -l all ||\
\ true\nfi\n"
- "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n echo Trying to remove inactive versions\n\
\ dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
\ || true\n dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
\ || true\nfi\n"
- sudo bash -c "rm -rf /var/cache/omnibus/src/*" || true
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- 'export GOMODCACHE=~/gomodcache
mkdir -p $GOMODCACHE
'
- bash .gitlab/package_build/build_agent_dmg.sh
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
timeout: 2h
variables:
INTEGRATION_WHEELS_CACHE_BUCKET: dd-agent-omnibus
INTEGRATION_WHEELS_SKIP_CACHE_UPLOAD: true
KEYCHAIN_NAME: build.keychain
NOTARIZATION_ATTEMPTS: 3
NOTARIZATION_TIMEOUT: 15m
NOTARIZATION_WAIT_TIME: 15s
S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
SIGN: true.deploy_packages_dmg.deploy_packages_dmg:
before_script:
- ls $OMNIBUS_PACKAGE_DIR
image: registry.ddbuild.io/ci/datadog-agent-buildimages/gitlab_agent_deploy$CI_IMAGE_GITLAB_AGENT_DEPLOY_SUFFIX:$CI_IMAGE_GITLAB_AGENT_DEPLOY
rules:
- if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
script:
- $S3_CP_CMD --recursive --exclude "*" --include "datadog-agent-7*.dmg" $OMNIBUS_PACKAGE_DIR
$S3_RELEASE_ARTIFACTS_URI/dmg/$PACKAGE_ARCH/
stage: deploy_packages
tags:
- arch:amd64agent_dmg-arm64-a7agent_dmg-arm64-a7:
after_script:
- "if [ \"$SIGN\" = true ]; then\n security delete-keychain \"build.keychain\"\
\ || true\nfi\n"
- sudo umount /Volumes/Agent || true
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg/*.dmg
- omnibus/pkg/version-manifest.json
before_script:
- sudo umount /Volumes/Agent || true
- rm -rf "$OMNIBUS_GIT_CACHE_DIR" || true
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION
paths:
- omnibus/vendor/bundle
needs:
- go_mod_tidy_check
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: $COMPARE_TO_BRANCH
paths:
- comp/core/gui/guiimpl/systray/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: $COMPARE_TO_BRANCH
paths:
- omnibus/**/*
- .gitlab-ci.yml
- release.json
- .gitlab/package_build/**/*
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
- if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
== "true" || $DDR_WORKFLOW_ID != null
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- set -eo pipefail
- export VAULT_ADDR=https://vault.us1.ddbuild.io
- vault login -method=aws -no-print
- 'export PATH="$(pyenv root)/shims:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
'
- "if [ -n \"$PYTHON_VERSION\" ]; then\n # 1. Use PYTHON_VERSION if set\n if !\
\ [[ \"$PYTHON_VERSION\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+$ ]]; then\n echo \"Invalid\
\ Python version format (should be X.Y.Z): $PYTHON_VERSION\" >& 2\n exit 1\n\
\ fi\n if ! [ -d \"$(pyenv root)/versions/$PYTHON_VERSION\" ]; then\n echo\
\ \"Python version $PYTHON_VERSION not found. Installing...\"\n pyenv install\
\ \"$PYTHON_VERSION\"\n fi\nelse\n # 2. Find appropriate python version from\
\ major version\n PYTHON_VERSION=\"$(cat .python-version)\"\n echo \"Find python\
\ version matching $PYTHON_VERSION\"\n PYTHON_VERSION_ESCAPED=\"${PYTHON_VERSION//./\\\
\\.}\"\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -n \"$versionmatches\" ]; then\n # 2.a.\n echo \"\
Found already existing python version for $PYTHON_VERSION\"\n else\n # 2.b.\n\
\ echo \"Installing Python $PYTHON_VERSION...\"\n pyenv install \"$PYTHON_VERSION\"\
\n versionmatches=\"$(ls \"$(pyenv root)/versions\" | grep -E \"^${PYTHON_VERSION_ESCAPED}\\\
\\.[0-9]+$\")\"\n if [ -z \"$versionmatches\" ]; then\n echo \"No matching\
\ Python version found for $PYTHON_VERSION even after installation.\" >& 2\n \
\ exit 1\n fi\n fi\n # If there are multiple matches, select the latest\
\ one\n PYTHON_VERSION=\"$(echo \"$versionmatches\" | sort -V | tail -n 1)\"\n\
fi\nVENV_NAME=\"datadog-agent-python-$PYTHON_VERSION\"\nVENV_PATH=\"$(pyenv root)/versions/$VENV_NAME\"\
\n"
- echo "Using Python $PYTHON_VERSION..."
- "# Check if the virtual environment directory exists\nif [ ! -d \"$VENV_PATH\"\
\ ]; then\n echo \"Creating virtual environment '$VENV_NAME'...\"\n pyenv virtualenv\
\ \"$PYTHON_VERSION\" \"$VENV_NAME\"\nelse\n echo \"Virtual environment '$VENV_NAME'\
\ already exists. Skipping creation.\"\nfi\n"
- pyenv activate $VENV_NAME
- python3 -m pip install "git+https://github.com/DataDog/datadog-agent-dev.git@v$(cat
.dda/version)" --break-system-packages
- pyenv rehash
- python3 -m dda self dep sync -f legacy-tasks
- pyenv rehash
- echo Setting up Go
- mkdir -p ~/go
- export GO_VERSION="$(cat .go-version)"
- eval "$(gimme $GO_VERSION)"
- export PATH="$PATH:$GOROOT/bin"
- echo Go version should be $GO_VERSION
- go version
- dda inv check-go-version
- DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
token)" || exit $?; export DD_API_KEY
- DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
token)" || exit $?; export DD_APP_KEY
- "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
.[0-9]+\\.(x|[0-9]+)$ ]]; then\n dda inv -- -e macos.report-versions -l all ||\
\ true\nfi\n"
- "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n echo Trying to remove inactive versions\n\
\ dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
\ || true\n dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
\ || true\nfi\n"
- sudo bash -c "rm -rf /var/cache/omnibus/src/*" || true
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- 'export GOMODCACHE=~/gomodcache
mkdir -p $GOMODCACHE
'
- bash .gitlab/package_build/build_agent_dmg.sh
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- macos:ventura-arm64
- specific:true
timeout: 2h
variables:
INTEGRATION_WHEELS_CACHE_BUCKET: dd-agent-omnibus
INTEGRATION_WHEELS_SKIP_CACHE_UPLOAD: true
KEYCHAIN_NAME: build.keychain
NOTARIZATION_ATTEMPTS: 3
NOTARIZATION_TIMEOUT: 15m
NOTARIZATION_WAIT_TIME: 15s
S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
SIGN: truedeploy_packages_dmg-arm64-a7deploy_packages_dmg-arm64-a7:
before_script:
- ls $OMNIBUS_PACKAGE_DIR
image: registry.ddbuild.io/ci/datadog-agent-buildimages/gitlab_agent_deploy$CI_IMAGE_GITLAB_AGENT_DEPLOY_SUFFIX:$CI_IMAGE_GITLAB_AGENT_DEPLOY
needs:
- agent_dmg-arm64-a7
rules:
- if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
script:
- $S3_CP_CMD --recursive --exclude "*" --include "datadog-agent-7*.dmg" $OMNIBUS_PACKAGE_DIR
$S3_RELEASE_ARTIFACTS_URI/dmg/$PACKAGE_ARCH/
stage: deploy_packages
tags:
- arch:amd64
variables:
PACKAGE_ARCH: arm64Changes Summary
ℹ️ Diff available in the job log. |
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
2 times, most recently
from
be58a55 to
f351c7c
Compare
rdesgroppes
added
the
qa/rc-required
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
2 times, most recently
from
54df5a3 to
c0e59af
Compare
rdesgroppes
changed the title
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
from
c0e59af to
6525223
Compare
github-actions
bot
added
long review
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
from
6525223 to
6842a0a
Compare
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 8c5d91d Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +3.66 | [+0.53, +6.78] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +3.66 | [+0.53, +6.78] | 1 | Logs |
| ➖ | ddot_logs | memory utilization | +0.39 | [+0.29, +0.48] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | +0.32 | [+0.24, +0.41] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.27 | [+0.23, +0.31] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | +0.25 | [+0.21, +0.28] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.16 | [+0.10, +0.23] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.07 | [-0.53, +0.66] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.06 | [-0.53, +0.66] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.03 | [-0.59, +0.66] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.01 | [-0.02, +0.04] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | +0.01 | [-0.13, +0.15] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.01 | [-0.30, +0.28] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.04 | [-0.69, +0.60] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | -0.18 | [-0.30, -0.06] | 1 | Logs |
| ➖ | ddot_metrics | memory utilization | -0.25 | [-0.37, -0.13] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | -0.31 | [-0.35, -0.27] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.54 | [-0.61, -0.48] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.96 | [-2.28, +0.35] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | -1.49 | [-4.24, +1.26] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | links |
|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | lost_bytes | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
from
6842a0a to
6c0838f
Compare
Contributor
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
|
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
2 times, most recently
from
d868f69 to
27250f8
Compare
rdesgroppes
commented
Jun 6, 2025
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
4 times, most recently
from
1705f74 to
ad25fb5
Compare
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
from
fd65f54 to
e8cde63
Compare
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: (both) - #37676 (comment) - #37676 (comment)
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
rdesgroppes
added a commit
that referenced
this pull request
Jul 21, 2025
This is to address: #37676 (comment)
This is the payload for adding macOS "native" build support for the AArch64/ARM64 architecture to the Agent, that is by removing the need for Rosetta 2 translation at runtime. Groundwork enabling the present payload: - DataDog/integrations-core#20455 - DataDog/integrations-core#20577 - DataDog/integrations-core#20699 - DataDog/omnibus-ruby#239 - DataDog/omnibus-ruby#240 - #38498 - #39082 - DataDog/agent-release-management#364 Some earlier scouting: - DataDog/integrations-core#19193 (credits to @Pythyu) - DataDog/omnibus-software#553 (credits to @Pythyu) - #31787 (credits to @Pythyu) - DataDog/ci-platform-machine-images#321 Some follow-ups: - DataDog/datadog-agent-dev#137 - DataDog/datadog-agent-buildimages#888 - #38446 - DataDog/integrations-core#20812 - DataDog/integrations-core#20815 - DataDog/integrations-core#20645 - DataDog/ci-platform-machine-images#343 - DataDog/agent-release-management#365
This is to address: #37676 (comment)
This is to address: (both) - #37676 (comment) - #37676 (comment)
This is to address: #37676 (comment)
This is to address: #37676 (comment)
This is to address: #37676 (comment)
rdesgroppes
force-pushed
the
regis.desgroppes/macos-aarch64
branch
from
e8cde63 to
7372415
Compare
Contributor
Serverless Benchmark Results
tl;drUse these benchmarks as an insight tool during development.
What is this benchmarking?The The benchmark is run using a large variety of lambda request payloads. In the charts below, there is one row for each event payload type. How do I interpret these charts?The charts below comes from The benchstat docs explain how to interpret these charts.
I need more helpFirst off, do not worry if the benchmarks are failing. They are not tests. The intention is for them to be a tool for you to use during development. If you would like a hand interpreting the results come chat with us in Benchmark stats |
robertjli
approved these changes
Jul 22, 2025
spencergilbert
approved these changes
Jul 22, 2025
This was referenced Jul 23, 2025
Merged
rdesgroppes
commented
Jul 25, 2025
rdesgroppes
added a commit
that referenced
this pull request
Jul 28, 2025
Add macOS build for AArch64/ARM64 This is the payload for adding macOS "native" build support for the AArch64/ARM64 architecture to the Agent, that is by removing the need for Rosetta 2 translation at runtime. Groundwork enabling the present payload: - DataDog/integrations-core#20455 - DataDog/integrations-core#20577 - DataDog/integrations-core#20699 - DataDog/omnibus-ruby#239 - DataDog/omnibus-ruby#240 - #38498 - #39082 - DataDog/agent-release-management#364 Some earlier scouting: - DataDog/integrations-core#19193 (credits to @Pythyu) - DataDog/omnibus-software#553 (credits to @Pythyu) - #31787 (credits to @Pythyu) - DataDog/ci-platform-machine-images#321 Some follow-ups: - DataDog/datadog-agent-dev#137 - DataDog/datadog-agent-buildimages#888 - #38446 - DataDog/integrations-core#20812 - DataDog/integrations-core#20815 - DataDog/integrations-core#20645 - DataDog/ci-platform-machine-images#343 - DataDog/agent-release-management#365
lucastemb
pushed a commit
that referenced
this pull request
Aug 7, 2025
Add macOS build for AArch64/ARM64 This is the payload for adding macOS "native" build support for the AArch64/ARM64 architecture to the Agent, that is by removing the need for Rosetta 2 translation at runtime. Groundwork enabling the present payload: - DataDog/integrations-core#20455 - DataDog/integrations-core#20577 - DataDog/integrations-core#20699 - DataDog/omnibus-ruby#239 - DataDog/omnibus-ruby#240 - #38498 - #39082 - DataDog/agent-release-management#364 Some earlier scouting: - DataDog/integrations-core#19193 (credits to @Pythyu) - DataDog/omnibus-software#553 (credits to @Pythyu) - #31787 (credits to @Pythyu) - DataDog/ci-platform-machine-images#321 Some follow-ups: - DataDog/datadog-agent-dev#137 - DataDog/datadog-agent-buildimages#888 - #38446 - DataDog/integrations-core#20812 - DataDog/integrations-core#20815 - DataDog/integrations-core#20645 - Datadog/ci-platform-machine-images#343 - DataDog/agent-release-management#365
10 tasks
rdesgroppes
added a commit
to DataDog/ansible-datadog
that referenced
this pull request
Oct 8, 2025
…#664) * Prepare deployment of Agent v7.70+ for macOS (arm64/x86_64) This change is meant to take into account how agent deliverables for macOS embed architecture qualifiers (name & metadata) starting from version 7.70 and some of the practical implications. A word of caution: I'm far from being fluent with `ansible`, let alone with the local setup and conventions, so the change is unlikely to work out of the box nor to be exhaustive. Following upstream changes might be of interest to reviewers: - architecture qualifiers: - DataDog/omnibus-ruby#240 - DataDog/agent-release-management#364 - new build for AArch64/ARM64: - DataDog/datadog-agent#37676 - DataDog/agent-release-management#365 * Fix CI-reported failures Altogether: - use method `GET` w/ `return_content: false` instead of `HEAD` (not necessarily implemented) - move `uri`-unsupported `retries` to upper level - pass `status_code: 200` to populate `datadog_macos_agent7_latest_arch_check.status` - make `ansible-lint` happy (at the expense of readability) * Apply Fanny's suggestion To prevent an undefined variable error. Co-authored-by: Fanny Jiang <fanny.jiang@datadoghq.com> * Make `ansible-lint` happy again Use a folded block scalar to overcome the 160-char line limitation. --------- Co-authored-by: Fanny Jiang <fanny.jiang@datadoghq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This is the payload for adding macOS "native" build support for the AArch64/ARM64 architecture to the Agent.
Motivation
Remove the need for Rosetta 2 translation at runtime.
Describe how you validated your changes
x86_64/AMD64 macOS jobs get here duplicated for AArch64/ARM64, bringing the same automated coverage to both architectures.
Colleagues running macOS Silicon are expected to try out the newly built Agent locally.
Possible Drawbacks / Trade-offs
Some follow-ups:
dmgbuildPython dependency for macOS datadog-agent-dev#137ddaversion to bringdmgbuildfor macOS datadog-agent-buildimages#888ddaversion to bringdmgbuildfor macOS #38446swiftcon AArch64/ARM64 (ABLD-28 follow-up) #39641pip install dmgbuildomnibus-ruby#241pip install dmgbuild(#241)" omnibus-ruby#242pip install dmgbuild" omnibus-ruby#243pyodbcdependency exclusion for macOS on AArch64/ARM64 (ABLD-28 follow-up) integrations-core#20812pymqidependency exclusion for macOS on AArch64/ARM64 (ABLD-28 follow-up) integrations-core#20815IBM MQfor macOS multiarch support (ABLD-28 follow-up) integrations-core#20645Additional Notes
Groundwork enabling the present payload:
delocatefor better macOS multiarch support integrations-core#20577osacriptbydmgbuildto build DMGs omnibus-ruby#239Some earlier scouting: