diff --git a/Templates/AzureDevOpsPipeline/README.md b/Templates/AzureDevOpsPipeline/README.md index dbffe812..20b8d4a0 100644 --- a/Templates/AzureDevOpsPipeline/README.md +++ b/Templates/AzureDevOpsPipeline/README.md @@ -1,42 +1,49 @@ -# Azure DevOps pipeline template +# Azure DevOps pipeline templates -This template provides an [azure-pipelines.yml](azure-pipelines.yml) definition file to setup an Azure DevOps Pipeline for applications managed in an Azure Git repository. +This template provides an [azure-pipelines.yml](azure-pipelines.yml) definition file to setup an Azure DevOps Pipeline for applications managed in an Azure Git repository along with the [azure-pipeline-deploy.yml](azure-pipeline-deploy.yml) definition file as a dedicated pipeline to run the deployment/installation of an existing package via IBM Wazi Deploy for installing into controlled environments. ## Overview and capabilities -This pipeline template is implementing the [Git-based process and branching model for mainframe development](https://ibm.github.io/z-devops-acceleration-program/docs/git-branching-model-for-mainframe-dev) within an Azure DevOps context. +This pipeline template is implementing the [Git-based process and branching model for mainframe development](https://ibm.github.io/z-devops-acceleration-program/docs/working-practices-intro) within an Azure DevOps context. It leverages the [Common Backend scripts](../Common-Backend-Scripts/) to implement the Setup, Build, Packaging and Deployment stages. -The pipeline implements the following stages +The **development pipeline** [azure-pipelines.yml](azure-pipelines.yml) implements the following stages -* `Setup` stage to [clone](../Common-Backend-Scripts/README.md#41---gitclonesh) the Git repository to a workspace directory on z/OS Unix System Services. +* `Setup` stage to [clone](../Common-Backend-Scripts/README.md#clone-repository-with-gitclonesh) the Git repository to a workspace directory on z/OS Unix System Services. * `Build` stage - * to invoke the zAppBuild [build](../Common-Backend-Scripts/README.md#42---dbbbuildsh) framework, - * to [prepare](../Common-Backend-Scripts/README.md#49---preparelogssh) the log files and publish them to the Azure build result. + * to invoke the [build](../Common-Backend-Scripts/README.md#build-stage) stage for either zBuilder or zAppBuild, + * to [prepare](../Common-Backend-Scripts/README.md#preparelogssh) the log files and publish them to the Azure build result. * `Packaging` stage - * to create a package (TAR file) based on the [PackageBuildOutputs script](../Common-Backend-Scripts/README.md#44---packagebuildoutputssh) + * to create a package (TAR file) based on the [PackageBuildOutputs script](../Common-Backend-Scripts/README.md#packaging-stage) * to publish the package file to the configured Artifact repository (Artifactory or Nexus, depending on the `publish` flag in CBS). - * (Alternatively - commented out) to create a new [UCD component version](../Common-Backend-Scripts/README.md#45---ucdpackagingsh) version + * (Alternatively - commented out) to create a new [UCD component version](../Common-Backend-Scripts/README.md#ucdpackagingsh) version * `Deployment` stage to deploy to the development test environment - * to run the Wazi Deploy [generate command](../Common-Backend-Scripts/README.md#47---wazideploy-generatesh) to download the package from the configured Artifact repository and generate the Deployment Plan. - * to deploy the package with the Wazi Deploy [deploy command](../Common-Backend-Scripts/README.md#48---wazideploy-deploysh) (Python-based) - * to run the Wazi Deploy [evidence command](../Common-Backend-Scripts/README.md#49---wazideploy-evidencesh) to generate deployment report and updating the evidence. - * to [prepare](../Common-Backend-Scripts/README.md#49---preparelogssh) the deployment log files and publish them to the Azure build result. -* `Deployment` stages to deploy to controlled test environments. Triggered a [release pipeline process](https://ibm.github.io/z-devops-acceleration-program/docs/branching-model-supporting-pipeline#the-release-pipeline-with-build-packaging-and-deploy-stages) via a manual pipeline request with the pipelineType set to release `release`: + * to run the Wazi Deploy [generate command](../Common-Backend-Scripts/README.md#wazideploy-generatesh) to download the package from the configured Artifact repository and generate the Deployment Plan. + * to deploy the package with the Wazi Deploy [deploy command](../Common-Backend-Scripts/README.md#wazideploy-deploysh) (Python-based) + * to run the Wazi Deploy [evidence command](../Common-Backend-Scripts/README.md#wazideploy-evidencesh) to generate deployment report and updating the evidence. + * to [prepare](../Common-Backend-Scripts/README.md#preparelogssh) the deployment log files and publish them to the Azure build result. +* `Deployment` stages to deploy to controlled test environments, if triggered the [release pipeline process](https://ibm.github.io/z-devops-acceleration-program/docs/branching-model-supporting-pipeline/#the-release-pipeline-with-build-package-and-deploy-stages) via a manual pipeline request with the pipelineType set to release `release`: * to create the release candidate Git tag using the [computeReleaseVersion script](../Common-Backend-Scripts/README.md#computereleaseversionsh). - * to run the Wazi Deploy [generate command](../Common-Backend-Scripts/README.md#47---wazideploy-generatesh) to download the package from the configured Artifact repository and generate the Deployment Plan. - * to deploy the package with the Wazi Deploy [deploy command](../Common-Backend-Scripts/README.md#48---wazideploy-deploysh) (Python-based) - * to run the Wazi Deploy [evidence command](../Common-Backend-Scripts/README.md#49---wazideploy-evidencesh) to generate deployment report and updating the evidence. - * to [prepare](../Common-Backend-Scripts/README.md#49---preparelogssh) the deployment log files and publish them to the Azure build result. + * to run the Wazi Deploy [generate command](../Common-Backend-Scripts/README.md#wazideploy-generatesh) to download the package from the configured Artifact repository and generate the Deployment Plan. + * to deploy the package with the Wazi Deploy [deploy command](../Common-Backend-Scripts/README.md#wazideploy-deploysh) (Python-based) into development and test regions + * to run the Wazi Deploy [evidence command](../Common-Backend-Scripts/README.md#wazideploy-evidencesh) to generate deployment report and updating the evidence. + * to [prepare](../Common-Backend-Scripts/README.md#preparelogssh) the deployment log files and publish them to the Azure build result. * `Cleanup` stage: - * to [delete the build workspace](../Common-Backend-Scripts/README.md#411---deleteworkspacesh) on z/OS Unix System Services. + * to [delete the build workspace](../Common-Backend-Scripts/README.md#deleteworkspacesh) on z/OS Unix System Services. -Depending on your selected deployment technology, review the definitions and (de-)/activate the appropriate steps. +The **deployment pipeline** template [azure-pipeline-deploy.yml](azure-pipeline-deploy.yml) is a manually-driven pipeline that allows teams to deploy an application package that has previously been created by the application pipeline into controlled environments, such as acceptance and production environments. -The pipeline uses the Azure concepts `Stage`, `Jobs` and `Tasks`, as well as [Azure DevOps templates](#supplied-azure-pipeline-templates). +It mandates the following input parameters: +* Target deployment environment selection +* Packaging information to retrieve the package from the artifact repository: + * Package type - either a preliminary or release build + * Package reference - either the branch name (like 'main') or release name (like 'rel-1.0.0') for the archive + * Package identifier - the unique identifier of the build. The ADO build number + +Based on the above information it kicks off the installation of the specified package. -![Azure Release pipeline](images/ado_releasePipeline.png) +The pipeline uses the Azure concepts `Stage`, `Jobs` and `Tasks`, as well as [Azure DevOps templates](#supplied-azure-pipeline-templates). ## Prerequisites @@ -44,7 +51,7 @@ To leverages this template, access to an Azure DevOps environment is required, a The template leverages the [SSH Task](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/ssh-v0?view=azure-pipelines) to invoke the [Common Backend scripts](../Common-Backend-Scripts/) via the configured SSH endpoint. Alternatively, the template can be modified to use the [CmdLine task](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/cmd-line-v2?view=azure-pipelines) to leverage an alternative communication technology such as Zowe CLI. -The Common Backend Scripts need to be configured for the selected deployment technologies to operate correctly. It is assumed that the Common Backend Scripts are configured to publish (upload) the package to the configured Artifact repository server. +The Common Backend Scripts need to be configured for the selected deployment technologies to operate correctly. It is assumed that the Common Backend Scripts are configured to publish (upload) the package to the configured Artifact repository server, from where the deployment pipeline will automatically pick it up. To tag important commits in the history of the application's Git repository, the Azure CLI is used and must be available on the Azure runner. Please follow the [documentation](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) to install the Azure CLI on all the agents within the agent pool that the pipelines will use. @@ -52,7 +59,7 @@ To tag important commits in the history of the application's Git repository, the **Note: Please work with your ADO pipeline specialist to review the below section.** -The `azure-pipeline.yml` can be dropped into the root folder of your Azure Git repository and will automatically provide pipelines for the specified triggers. Please review the definitions thoroughly with your Azure administrator. The use of Templates is recommended for a centralized control of definitions. +The `azure-pipeline.yml` and `azure-pipeline-deploy.yml` can be dropped into the root folder of your Azure Git repository and will automatically provide pipelines for the specified triggers. Please review the definitions thoroughly with your Azure administrator. The use of Templates is recommended for a centralized control of definitions. Following requirements need to be met: * An [Azure Agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops) is installed and can connect to the z/OS system where builds are occurring. @@ -73,15 +80,15 @@ Variable | Description ### Supplied Azure pipeline templates -The directory [templates](templates/) contains additional [Azure pipeline templates](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops&pivots=templates-includes#parameters-to-select-a-template-at-runtime) that implement actions used in the main pipeline template. This approach helps to keep central control over the tagging and release deployment process. Please review the templates with your Azure administrator and adjust the reference within the [azure-pipelines.yml](azure-pipelines.yml#L45) file. +The directory [templates](templates/) contains additional [Azure pipeline templates](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops&pivots=templates-includes#parameters-to-select-a-template-at-runtime) that implement actions used in the main pipeline templates. This approach helps to keep central control over the tagging and release deployment process. Please review the templates with your Azure administrator, store them at a central git repository, and adjust the reference within the [azure-pipelines.yml](azure-pipelines.yml#L33) file. -The tagging process implemented in the template leverages the [AZ REST command of the Azure DevOps CLI](templates/tagging/createReleaseCandidate.yml#L33) to connect with the existing permissions of the pipeline build user to the Azure repository to create the release candidate and the final release tag. Azure DevOps CLI needs to be available on the installed Azure agents. +The tagging process implemented in the template leverages the [AZ REST command of the Azure DevOps CLI](templates/tagging/createReleaseCandidate.yml#L33) to connect with the existing permissions of the pipeline build user to the Azure repository to create the release candidate and the final release tag. Azure DevOps CLI needs to be available on the installed Azure agents. ### Azure environments This template uses [Azure deployment jobs](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops) to trigger the deployment on the mainframe and document the deployment of packages to the defined [environments in Azure](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments?view=azure-devops). Please create the environments in advance, also consider configuring approvals and checks for environments as gates. -The template allows the development team to add additional environments where they would like to deploy the package to. Please see the [deployment section of the pipeline configuration](azure-pipelines.yml#L187). +The template allows the development team to add additional environments where they would like to deploy the package to. Please see the [deployment section of the pipeline configuration](azure-pipelines.yml#L309). ### Installing the private SSH key @@ -117,20 +124,26 @@ To upload the private key as a Secure File: * obtain the private SSH key from the location where you generated it. For instance, use `scp` to download the private key to your workstation. * upload the private SSH key as a Secure File and set the appropriate permissions that the pipeline can use it. For more information, please checkout the [ADO documentation on Secure Files](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops). -## Pipeline usage +## Usage overview + +This template implements the common build, package and deploy steps to process various configurations according to the conventions defined in the IBM zDevOps Guide. + +Please make yourself familiar with the [Git branching for mainframe development](https://ibm.github.io/z-devops-acceleration-program/docs/working-practices-intro) documentation. + + +### Development pipeline -The pipeline implements the common build, package and deploy steps to process various configurations according to the defined conventions. -It is a single Azure pipeline definition supporting various workflows. The [azure-pipelines.yml](azure-pipelines.yml) supports: +The development pipeline definition supporting various workflows. The [azure-pipelines.yml](azure-pipelines.yml) supports: * automated [build pipelines for feature branches](https://ibm.github.io/z-devops-acceleration-program/docs/branching-model-supporting-pipeline#pipeline-build-of-feature-branches) with a clone and build stage, * the [basic pipeline](https://ibm.github.io/z-devops-acceleration-program/docs/branching-model-supporting-pipeline#the-basic-build-pipeline-for-main-epic-and-release-branches) when changes are merged into the branch `main` and -* a [release pipeline](https://ibm.github.io/z-devops-acceleration-program/docs/branching-model-supporting-pipeline#the-release-pipeline-with-build-packaging-and-deploy-stages) to build and package the release candidate, installation to predefined environments including the production environment. +* a [release pipeline](https://ibm.github.io/z-devops-acceleration-program/docs/branching-model-supporting-pipeline#the-release-pipeline-with-build-packaging-and-deploy-stages) to build and package the release candidate, installation to predefined environments. -Please check the pipeline definition to understand the various triggers for which this pipeline is executed and also the conditions when stages, jobs or steps are executed. +The deployment pipeline definition supports teams to install an existing package that is stored in an artifact repository -Please make yourself familiar with the [Git branching for mainframe development](https://ibm.github.io/z-devops-acceleration-program/docs/git-branching-model-for-mainframe-dev/#characteristics-of-mainline-based-development-with-feature-branches) documentation. +Please check the pipeline definition to understand the various triggers for which this pipeline is executed and also the conditions when stages, jobs or steps are executed. -### Pipeline parameters +#### Development pipeline parameters In a default setup, the pipeline is triggered for each new commit. @@ -143,7 +156,9 @@ pipelineType | Pipeline type - either build, release or preview. (Default: b releaseType | Release type - major, minor, patch as input to compute the release version and to set the release candidate and release git tags. (Default: patch) verbose | boolean flag to control logging of build framework. (Default: false) -### Feature Branch pipeline +#### Supported workflows + +##### Feature branch pipeline The pipeline for feature branches executes the following steps: @@ -154,9 +169,9 @@ The pipeline for feature branches executes the following steps: Overview of the pipeline: -![Feature Branch pipeline](images/ado_featureBranchPipeline.png) +![Feature branch pipeline](images/ado_featureBranchPipeline.png) -### Basic build pipeline for Integration branches +##### Basic build pipeline for Integration branches The basic build pipeline for integration branches contains the following stages: * Clone @@ -169,7 +184,7 @@ Overview of the pipeline: ![Basic pipeline for integration branches](images/ado_basicBuildPipeline.png) -### Release pipeline +##### Release pipeline When the development team agrees to build a release candidate, the release pipeline type is triggered manually. @@ -178,7 +193,7 @@ It covers the followings steps: * Build and creation of Git tagging for the release candidate * Package & publish package * Deployment to the integration test environment -* Deployment to the controlled test environments including Production +* Deployment to the controlled test environments such as the Acceptance Test environment * Cleanup The development team manually requests the pipeline and specifies the *pipeline type* `release` as a parameter. Along with the *release type*, the pipeline will tag a release candidate and also the final release that is deployed to production. @@ -189,7 +204,39 @@ Overview of the release pipeline: ![Release pipeline](images/ado_releasePipeline.png) -### Additional notes +### Deployment pipeline + +A dedicated, manually invoked deployment pipeline template is provided to trigger the installation of an existing package from artifactory into additional test environments, that are not on the default route for the application teams, or to drive the deployment of release packages into the production environments. + +#### Pipeline parameters + +The pipeline is triggered manually by the team to request a deployment of an existing package. They need to specify the following information about the package: + +Parameter | Description +--- | --- +deploymentEnvironment | Target deployment environment - Environment where the package should be deployed to +packageType | Package Type - a preliminary build or release package (Default: build) +packageReference | Package Reference - either the branch name (like 'main') for preliminary build packages or the release name (like 'rel-1.5.0') for the release package type. +packageBuildIdentifier | Package Build Identifier - Identifier for the archive, like build number + +These information allow the CBS to find the package in the configured artifact repository. + +#### Pipeline stages + +The deployment pipeline contains following stages: +* Setup +* Deployment to selected runtime environment via IBM Wazi Deploy +* Cleanup + +Overview of the deploy pipeline request dialogue: + +![Deployment pipeline request dialogue](images/ado_deployPipeline_requestDialogue.png) + +Overview of the deployment pipeline stages: + +![Deployment pipeline run](images/ado_deployPipeline_run.png) + +### Usage notes Build, package and deployment logs are attached to the Azure pipeline run and can be accessed via UI: diff --git a/Templates/AzureDevOpsPipeline/azure-pipeline-deploy.yml b/Templates/AzureDevOpsPipeline/azure-pipeline-deploy.yml new file mode 100644 index 00000000..d899b17e --- /dev/null +++ b/Templates/AzureDevOpsPipeline/azure-pipeline-deploy.yml @@ -0,0 +1,141 @@ +# +# DEPLOYMENT PIPELINE +# +# This is a manually triggered pipeline that deploys a referenced archive of the application into the selected runtime environment +# using IBM Wazi Deploy +# +# It leverages the conventions of the Common Backend scripts, and assumes that the archive (a.k.a the package) +# has been uploaded to Artifactory or Nexus +# +# It relates to the environments, that have been configured for the project in the ADO project + + +# Configuration + + ### Variables to be created in Azure pipeline configuration + + # ) agentPool # Agent pool name for Azure Agents to connect to MVS + # ) zosSSHConnection # zOS - SSH connection name + # ) zosSSHKnownHost # Known host entry for secure shell connections + # ) zosSSHPrivateKeySecureFile # Reference to uploaded Private SSH Key that in ADO Pipeline/Libary/SecureFile that is + # that installed for sftp processes to fetch logs + # ) pipelineWorkspace # Root directory on z/OS Unix System services to perform build and deployments. E.g. `/u/ado/workspace` + # ) zosHostname # zOS - Host IP for SFTP connection + # ) zosSFTPUser # zOS - Host user for SFTP connection + + ### Environment variables to be added in z/OS Unix System Services $HOME/.profile + # To locate the backend scripts. See also Installation of backend scripts. + # PIPELINE_SCRIPTS: + # Ex: export PIPELINE_SCRIPTS=/var/dbb/Common-Backend-Scripts + # export PATH=$PIPELINE_SCRIPTS:$PATH + +trigger: none + +parameters: + - name: deploymentEnvironment + displayName: "Target deployment environment - Environment where the package should be deployed to" + type: string + default: Acceptance + values: + - Integration + - Acceptance + - Production + - name: archiveType + displayName: "Archive Type - a preliminary build or release candidate" + type: string + default: build + values: + - build + - release + - name: archiveReference + displayName: "Archive Reference - either the branch name (like 'main') or release name (like 'rel-1.0.0') for the archive" + type: string + default: main + - name: archiveBuildIdentifier + displayName: "Archive Identifier - Identifier for the archive, like build number or timestamp" + type: string + default: "build-identifier" + +# pipeline templates +resources: + repositories: + - repository: templates + type: git + name: PipelineCore/PipelineCore + +# Agent pool where the pipeline runs +pool: + name: $(agentPool) + +# Variables used in the pipeline +variables: + + ### Variables generated during run time + organization: $[ replace(variables['organization_tmp'], '/', '') ] # Azure organization name + project: $(System.TeamProject) # Project name + application: ${{ variables['Build.Repository.Name'] }} # Application name + repo: "git@ssh.dev.azure.com:v3/$(organization)/$(project)/$(application)" # Git repository path + uniqueWorkspaceId: $(pipelineWorkspace)/$(application)/deployments/${{ parameters.archiveType }}/${{ parameters.archiveReference }}/build-$(Build.BuildNumber) # Calculated workspace path + packageName: $(application) # default name of the Package + buildNumber: ${{ parameters.archiveBuildIdentifier }} # Build reference + pipelineType: ${{ parameters.archiveType }} + branch: ${{ parameters.archiveReference }} + +# Pipeline stages +stages: + - stage: Setup + jobs: + - job: Setup + continueOnError: false + steps: + - checkout: none + # Display CI-CD pipeline parms + - script: | + echo " " + echo " Deployment pipeline parameters" + echo "*******************************************" + echo "organization = $(organization)" + echo "project = $(project)" + echo "application = $(application)" + echo "repo = $(repo)" + echo " Archive information" + echo "archiveReference = $${{ parameters.archiveReference }}" + echo "archiveBuildIdentifier = $(buildNumber)" + echo "archiveType = $(pipelineType)" + echo " Connectivity" + echo "zosSSHConnection = $(zosSSHConnection)" + echo "zosHostname = $(zosHostname)" + echo "zosSFTPUser = $(zosSFTPUser)" + echo "ussWorkspaceRoot = $(pipelineWorkspace)" + echo "ussWorkspaceDir = $(uniqueWorkspaceId)" + echo " " + displayName: "Deploy pipeline parameters" + + # Production environment + - stage: Deploy + displayName: "Deploy Package" + dependsOn: + - Setup + jobs: + - template: deployment/deployPackage.yml@templates + parameters: + environmentName: ${{ parameters.deploymentEnvironment }} + packageReference: "${{ parameters.archiveReference }}" + packageIdentifier: "${{ parameters.archiveBuildIdentifier }}" + packageType: "${{ parameters.archiveType }}" + + # Cleanup + - stage: Cleanup + condition: always() + jobs: + - job: Cleanup + displayName: "Cleanup Deployment Workspace" + steps: + - checkout: none + - task: SSH@0 + inputs: + sshEndpoint: $(zosSSHConnection) + runOptions: "commands" + commands: ". ./.profile && deleteWorkspace.sh -w $(uniqueWorkspaceId)" + readyTimeout: "20000" + displayName: "Delete Deployment folder on USS" \ No newline at end of file diff --git a/Templates/AzureDevOpsPipeline/azure-pipelines.yml b/Templates/AzureDevOpsPipeline/azure-pipelines.yml index f71aff31..b38e2aa0 100644 --- a/Templates/AzureDevOpsPipeline/azure-pipelines.yml +++ b/Templates/AzureDevOpsPipeline/azure-pipelines.yml @@ -1,4 +1,4 @@ -# Branches that triggers this pipleline +# Branches that triggers this pipeline trigger: - feature/* - release/* @@ -6,7 +6,6 @@ trigger: - epic/* - main -# Pipeline input parameters to drive build and release pipelines parameters: - name: pipelineType displayName: "Pipeline Type" @@ -26,7 +25,7 @@ parameters: - patch - name: verboseLogging - displayName: "zAppBuild verbose logging" + displayName: "Verbose Logging" type: boolean default: false @@ -60,37 +59,31 @@ variables: # Ex: export PIPELINE_SCRIPTS=/var/dbb/Common-Backend-Scripts # export PATH=$PIPELINE_SCRIPTS:$PATH - ### Input parameter variables: - pipelineType: ${{ parameters.pipelineType }} - releaseType: ${{ parameters.releaseType }} - # Computation DBB verbose logging - ${{ if eq(parameters.verboseLogging, true) }}: - verbose: "-v" - ${{ if eq(parameters.verboseLogging, false) }}: - verbose: "" - ### Variables generated during run time ${{ if startsWith(variables['Build.SourceBranch'], 'refs/heads/') }}: # Extract branch name for push branch: $[ replace(variables['Build.SourceBranch'], 'refs/heads/', '') ] ${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}: # Extract branch name for pull request branch: $[ replace(variables['System.PullRequest.SourceBranch'], 'refs/heads/', '') ] buildNumber: $(Build.BuildNumber) # Build number - organization_tmp: $[ replace(variables['System.TeamFoundationCollectionUri'], 'https://dev.azure.com/', '') ] # Generate Azure repository uri for cloning - organization: $[ replace(variables['organization_tmp'], '/', '') ] # Azure organization name project: $(System.TeamProject) # Project name application: ${{ variables['Build.Repository.Name'] }} # Application name - repo: "git@ssh.dev.azure.com:v3/$(organization)/$(project)/$(application)" # Git repository path - uniqueWorkspaceId: $(pipelineWorkspace)/$(application)/$(branch)/build-$(buildNumber) # Calculated worksapce path - packageName: $(application) # Name of the Package + repo: ${{ variables['Build.Repository.Uri'] }}" # retrieves the http uri of the triggering project + uniqueWorkspaceId: $(pipelineWorkspace)/$(application)/$(branch)/build-$(buildNumber) # Calculated workspace path + # packageName: $(application) # Name of the Package + pipelineType: ${{ parameters.pipelineType }} + releaseType: ${{ parameters.releaseType }} isMain: $[contains(variables['Build.SourceBranch'], 'main')] # Variable to indicate pipeline for the main branch - - # Azure package name. Package names must contain only lowercase alphanumeric segments separated by a dash, dot or underscore. - # reformattedBranchName: $[ replace(variables['branch'], '/', '-')] - # lowerApplicationName: $[ lower(variables['application']) ] - # lowerReformattedBranchName: $[ lower(variables['reformattedBranchName']) ] - # azureArtifactName: $(lowerReformattedBranchName)-$(lowerApplicationName).tar - - + # Computation DBB verbose logging + ${{ if eq(parameters.verboseLogging, true) }}: + verbose: "-v" + ${{ if eq(parameters.verboseLogging, false) }}: + verbose: "" + # Azure artifact upload. Package names must contain only lowercase alphanumeric segments separated by a dash, dot or underscore. + # Kept for reference for users wanting to upload to Azure Artifacts instead of Artifactory/Nexus + #reformattedBranchName: $[ replace(variables['branch'], '/', '-')] + #lowerApplicatioName: $[ lower(variables['application']) ] + #lowerReformattedBranchName: $[ lower(variables['reformattedBranchName']) ] + #azureArtifactName: $(lowerReformattedBranchName)-$(lowerApplicatioName).tar # Pipeline stages stages: - stage: Setup @@ -112,7 +105,6 @@ stages: echo "verbose = ${{ parameters.verboseLogging }} " echo "branch = $(branch)" echo "buildNumber = $(buildNumber)" - echo "organization = $(organization)" echo "project = $(project)" echo "application = $(application)" echo "repo = $(repo)" @@ -133,7 +125,8 @@ stages: inputs: sshEndpoint: $(zosSSHConnection) runOptions: "commands" - commands: ". ./.profile && gitClone.sh -w $(uniqueWorkspaceId) -r $(repo) -b $(branch)" + # Showcases how the System.Accesstoken can be passed to authenticate the pipeline for cloning via HTTP + commands: ". ./.profile && gitClone.sh -c \"AUTHORIZATION: Bearer $(System.AccessToken)\" -w $(uniqueWorkspaceId) -r $(repo) -b $(branch)" readyTimeout: "20000" displayName: "Clone application repo" @@ -141,8 +134,6 @@ stages: # Invoke build and create a release candidate tag - stage: Build - dependsOn: - - Setup jobs: - job: Build displayName: "Build" @@ -152,8 +143,8 @@ stages: inputs: sshEndpoint: $(zosSSHConnection) runOptions: "commands" - commands: ". ./.profile && dbbBuild.sh -w $(uniqueWorkspaceId) -a $(application) -b $(branch) -p $(pipelineType) ${{variables.verbose}}" - #commands: ". ./.profile && zBuilder.sh -w $(uniqueWorkspaceId) -a $(application) -b $(branch) -p $(pipelineType) ${{variables.verbose}}" + #commands: ". ./.profile && dbbBuild.sh -w $(uniqueWorkspaceId) -a $(application) -b $(branch) -p $(pipelineType) ${{variables.verbose}}" + commands: ". ./.profile && zBuilder.sh -w $(uniqueWorkspaceId) -a $(application) -b $(branch) -p $(pipelineType) ${{variables.verbose}}" readyTimeout: "20000" failOnStdErr: false displayName: "Build application" @@ -215,30 +206,30 @@ stages: displayName: "Package Outputs" steps: - checkout: none - #- task: SSH@0 - # inputs: - # sshEndpoint: $(zosSSHConnection) - # runOptions: 'commands' - # commands: '. ./.profile && ucdPackaging.sh -v $(packageName) -c $(application) -w $(uniqueWorkspaceId) -b $(branch) -e $artifactRepoConfig' - # readyTimeout: '20000' - # displayName: 'Package the build artifacts for Deployment' - task: SSH@0 condition: succeeded() inputs: sshEndpoint: $(zosSSHConnection) runOptions: "commands" commands: ". ./.profile && packageBuildOutputs.sh -w $(uniqueWorkspaceId) -a $(application) -b $(branch) -p $(pipelineType) -r $(packageVersion) -i $(buildNumber)" - # For reference when creating a local tar file, skipping the upload - #commands: ". ./.profile && packageBuildOutputs.sh -w $(uniqueWorkspaceId) -t $(application).tar -a $(application) -b $(branch) -p $(pipelineType) -v $(packageVersion)" readyTimeout: "20000" - displayName: "Package the build artifacts for Deployment" + displayName: "Package the build artifacts" + # Alternative way to use DevOps Deploy packaging + #- task: SSH@0 + # inputs: + # sshEndpoint: $(zosSSHConnection) + # runOptions: 'commands' + # commands: '. ./.profile && ucdPackaging.sh -v $(packageName) -c $(application) -w $(uniqueWorkspaceId) -b $(branch) -e $artifactRepoConfig' + # readyTimeout: '20000' + # displayName: 'Package the build artifacts for Deployment' variables: ${{ if eq(variables['pipelineType'], 'release') }}: # Retrieve the version name from the release candidate packageVersion: $[ stageDependencies.Build.Tag.outputs['computeReleaseName.nextReleaseVersion'] ] - ${{ if not(eq(variables['pipelineType'], 'release')) }}: # Default for all other cases - packageVersion: $(packageName) + ${{ if not(eq(variables['pipelineType'], 'release')) }}: # For all other cases + packageVersion: $(branch) - # Alternate upload option to demonstrate to upload package to Azure DevOps Artifacts + # Demonstrate to upload package to Azure DevOps Artifacts + ## TODO - Requires more work to distinguish build and release packages # - job: PublishPackage # displayName: "Publish deployment package" # dependsOn: Package @@ -280,13 +271,16 @@ stages: Environment: Integration ${{ if eq(variables['pipelineType'], 'release') }}: # Retrieve the version name from the release candidate packageVersion: $[ stageDependencies.Build.Tag.outputs['computeReleaseName.nextReleaseVersion'] ] - ${{ if not(eq(variables['pipelineType'], 'release')) }}: # Default for all other cases - packageVersion: $(packageName) + ${{ if not(eq(variables['pipelineType'], 'release')) }}: # For all other cases + packageVersion: $(branch) + jobs: - template: deployment/deployPackage.yml@templates parameters: environmentName: ${{ variables.Environment }} - packageVersion: "$(packageVersion)" + packageReference: $(packageVersion) + packageIdentifier: $(buildNumber) + packageType: $(pipelineType) ## Deployment to higher test environments # deployments to these test environment can only happen when the development team requests a release pipeline @@ -303,33 +297,41 @@ stages: ${{ if eq(variables['pipelineType'], 'release') }}: # Retrieve the version name from the release candidate packageVersion: $[ stageDependencies.Build.Tag.outputs['computeReleaseName.nextReleaseVersion'] ] ${{ if not(eq(variables['pipelineType'], 'release')) }}: # For all other cases - packageVersion: $(packageName) + packageVersion: $(branch) jobs: - template: deployment/deployPackage.yml@templates parameters: environmentName: ${{ variables.Environment }} - packageVersion: "$(packageVersion)" + packageReference: $(packageVersion) + packageIdentifier: $(buildNumber) + packageType: $(pipelineType) # Add additional deployment stages as you need. + + # The below deployment is expected to be run through the deployment pipeline to drive + # deployments independent of the CI pipeline - # Production environment - - stage: Deploy_PRODUCTION - displayName: "Deploy Production" - dependsOn: - - Build - - Deploy_ACCEPTANCE - condition: and(succeeded(), eq(variables.isMain, true), eq(variables['pipelineType'], 'release')) - variables: - Environment: Production - ${{ if eq(variables['pipelineType'], 'release') }}: # Retrieve the version name from the release candidate - packageVersion: $[ stageDependencies.Build.Tag.outputs['computeReleaseName.nextReleaseVersion'] ] - ${{ if not(eq(variables['pipelineType'], 'release')) }}: # For all other cases - packageVersion: $(packageName) - jobs: - - template: deployment/deployPackage.yml@templates - parameters: - environmentName: ${{ variables.Environment }} - packageVersion: "$(packageVersion)" +# # Production environment +# - stage: Deploy_PRODUCTION +# displayName: "Deploy Production" +# dependsOn: +# - Build +# - Deploy_ACCEPTANCE +# condition: and(succeeded(), eq(variables.isMain, true), eq(variables['pipelineType'], 'release')) +# variables: +# Environment: Production +# ${{ if eq(variables['pipelineType'], 'release') }}: # Retrieve the version name from the release candidate +# packageVersion: $[ stageDependencies.Build.Tag.outputs['computeReleaseName.nextReleaseVersion'] ] +# ${{ if not(eq(variables['pipelineType'], 'release')) }}: # For all other cases +# packageVersion: $(branch) +# jobs: +# - template: deployment/deployPackage.yml@templates +# parameters: +# environmentName: ${{ variables.Environment }} +# packageVersion: $(packageVersion) +# packageReference: $(packageVersion) +# packageIdentifier: $(buildNumber) +# packageType: $(pipelineType) # Cleanup - stage: Cleanup diff --git a/Templates/AzureDevOpsPipeline/images/ado_basicBuildPipeline.png b/Templates/AzureDevOpsPipeline/images/ado_basicBuildPipeline.png index 195ad4af..90995490 100644 Binary files a/Templates/AzureDevOpsPipeline/images/ado_basicBuildPipeline.png and b/Templates/AzureDevOpsPipeline/images/ado_basicBuildPipeline.png differ diff --git a/Templates/AzureDevOpsPipeline/images/ado_deployPipeline_requestDialogue.png b/Templates/AzureDevOpsPipeline/images/ado_deployPipeline_requestDialogue.png new file mode 100644 index 00000000..be04f265 Binary files /dev/null and b/Templates/AzureDevOpsPipeline/images/ado_deployPipeline_requestDialogue.png differ diff --git a/Templates/AzureDevOpsPipeline/images/ado_deployPipeline_run.png b/Templates/AzureDevOpsPipeline/images/ado_deployPipeline_run.png new file mode 100644 index 00000000..180ff3f9 Binary files /dev/null and b/Templates/AzureDevOpsPipeline/images/ado_deployPipeline_run.png differ diff --git a/Templates/AzureDevOpsPipeline/images/ado_featureBranchPipeline.png b/Templates/AzureDevOpsPipeline/images/ado_featureBranchPipeline.png index fc85733e..357d79bb 100644 Binary files a/Templates/AzureDevOpsPipeline/images/ado_featureBranchPipeline.png and b/Templates/AzureDevOpsPipeline/images/ado_featureBranchPipeline.png differ diff --git a/Templates/AzureDevOpsPipeline/images/ado_releasePipeline.png b/Templates/AzureDevOpsPipeline/images/ado_releasePipeline.png index 1b304eab..1be6ff52 100644 Binary files a/Templates/AzureDevOpsPipeline/images/ado_releasePipeline.png and b/Templates/AzureDevOpsPipeline/images/ado_releasePipeline.png differ diff --git a/Templates/AzureDevOpsPipeline/images/ado_requestReleasePipeline.png b/Templates/AzureDevOpsPipeline/images/ado_requestReleasePipeline.png index 2343a687..9b20af04 100644 Binary files a/Templates/AzureDevOpsPipeline/images/ado_requestReleasePipeline.png and b/Templates/AzureDevOpsPipeline/images/ado_requestReleasePipeline.png differ diff --git a/Templates/AzureDevOpsPipeline/templates/deployment/createWaziDeployIndex.yml b/Templates/AzureDevOpsPipeline/templates/deployment/createWaziDeployIndex.yml new file mode 100644 index 00000000..8f2eb14e --- /dev/null +++ b/Templates/AzureDevOpsPipeline/templates/deployment/createWaziDeployIndex.yml @@ -0,0 +1,59 @@ +########### +# ADO template to save the Wazi Deploy evidence files on ADO runner environment +# +# This template: +# * Copies the evidences into a common evidences directory +# * Invoked from deployPackage.yml template + +parameters: + - name: evidenceFolder + type: string + default: "." # defaults for any parameters that specified with "." (current directory) + - name: environmentName + type: string + default: "." + +steps: + - bash: | + + # local Variables + rc=0 + PGM="deployment/createWaziDeployIndex.yml" + evidenceFile="$(Build.ArtifactStagingDirectory)/${{ parameters.evidenceFolder }}/evidences/evidence.yaml" + evidencesPath=/var/work/wazi_deploy_evidences_ado/$(application)/$(pipelineType)/${{ parameters.environmentName }} + + rc=0 + + # check evidence file + if [ ! -f "${evidenceFile}" ]; then + rc=8 + ERRMSG=$PGM": [ERROR] Evidence file (${evidenceFile}) was not found. rc="$rc + echo $ERRMSG + fi + + # create evidence dir + if [ $rc -eq 0 ]; then + + CMD="mkdir -p $evidencesPath" + ${CMD} + rc=$? + fi + + # Copy evidence File + if [ $rc -eq 0 ]; then + + CMD="cp ${evidenceFile} $evidencesPath/evidence-$(buildNumber).yaml" + ${CMD} + rc=$? + fi + + # Error Handling + if [ $rc -eq 0 ]; then + echo "${PGM}: [INFO] - Copy Evidence File (evidence-$(buildNumber).yaml) to Evidence Inventory ($evidencesPath) completed." + else + + echo "${PGM}: [WARNING] - Copy Evidence File to Evidence Inventory Failed" + exit 1 + fi + + displayName: "Copy Evidence File to Evidence Inventory" diff --git a/Templates/AzureDevOpsPipeline/templates/deployment/deployPackage.yml b/Templates/AzureDevOpsPipeline/templates/deployment/deployPackage.yml index 126b62bf..afe7b67c 100644 --- a/Templates/AzureDevOpsPipeline/templates/deployment/deployPackage.yml +++ b/Templates/AzureDevOpsPipeline/templates/deployment/deployPackage.yml @@ -1,11 +1,85 @@ +# This template is triggered either from an application or deployment pipeline +# It takes a set of arguments and triggers +# an IBM Wazi Deploy deployment process through the Common Backend scripts +# It assumes, that the package was upload to Artifactory or Nexus + parameters: - name: environmentName type: string - default: "." # defaults for any parameters that specified with "." (current directory) - - name: packageVersion + - name: packageReference + type: string + - name: packageIdentifier + type: string + - name: packageType type: string - default: "default" + jobs: + - deployment: DEPLOY_${{ parameters.environmentName }} + displayName: "Deploy to ${{ parameters.environmentName }}" + environment: "${{ parameters.environmentName }}" + strategy: + runOnce: + deploy: + steps: + - download: none + - task: InstallSSHKey@0 + inputs: + knownHostsEntry: $(zosSSHKnownHost) # Known Hosts Entry for build LPAR + sshKeySecureFile: $(zosSSHPrivateKeySecureFile) # Required.Reference to uploaded Private SSH Key that in ADO Pipeline/Libary/SecureFile that is installed for sftp processes + + # Generate Deployment Plan. + - task: SSH@0 + inputs: + sshEndpoint: $(zosSSHConnection) + runOptions: "commands" + commands: ". ./.profile && wazideploy-generate.sh -w $(uniqueWorkspaceId) -a $(application) -r deploy-logs-${{ parameters.environmentName }}/deployment-plan-report.html -I ${{ parameters.packageIdentifier }} -R ${{ parameters.packageReference }} -P ${{ parameters.packageType }}" + # For reference when creating a local tar file, and skipping the upload. In this case the pipeline needs to provide the reference to the tar file. + # commands: ". ./.profile && wazideploy-generate.sh -w $(uniqueWorkspaceId) -a $(application) -i $(application).tar -r deploy-logs/deployment-plan-report.html" + readyTimeout: "20000" + displayName: "Generate Deployment Plan with Wazi Deploy" + + + # Perform Deployment. + - task: SSH@0 + inputs: + sshEndpoint: $(zosSSHConnection) + runOptions: "commands" + commands: ". ./.profile && wazideploy-deploy.sh -w $(uniqueWorkspaceId) -e EOLEB7-$(application)-${{ parameters.environmentName }}.yaml -l deploy-logs-${{ parameters.environmentName }}/evidences/evidence.yaml" + readyTimeout: "20000" + displayName: "Deploy Package with Wazi Deploy" + + # Generate Evidence. + - task: SSH@0 + inputs: + sshEndpoint: $(zosSSHConnection) + runOptions: "commands" + commands: ". ./.profile && wazideploy-evidence.sh -w $(uniqueWorkspaceId) -l deploy-logs-${{ parameters.environmentName }}/evidence.yaml -o deploy-logs-${{ parameters.environmentName }}/deployment-report.html" + readyTimeout: "20000" + displayName: "Generate deployment report with Wazi Deploy" + + # Pull logs and publish them. + - task: CmdLine@2 + inputs: + script: | + rm -rf "$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}" + mkdir "$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}" + sftp -r $(zosSFTPUser)@$(zosHostname):$(uniqueWorkspaceId)/deployPkgDir/deploy-logs-${{ parameters.environmentName }}/* "\"$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}\"" + displayName: "Retrieve deployment reports" + + - task: PublishBuildArtifacts@1 + inputs: + PathtoPublish: "$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}/" + ArtifactName: "deploy-logs-${{ parameters.environmentName }}" + publishLocation: "Container" + displayName: "Publish deployment logs" + + # Store logs in evidence inventory + - template: createWaziDeployIndex.yml + parameters: + evidenceFolder: "deploy-logs-${{ parameters.environmentName }}" + environmentName: "${{ parameters.environmentName }}" + + # Left this as a template from a previous version showcasing the ADO package registry # - job: RetrievePackage # displayName: "Retrieve Deployment Package" @@ -73,69 +147,4 @@ jobs: # readyTimeout: "20000" # string. Required. SSH handshake timeout. Default: 20000. # overwrite: true # boolean. Overwrite. Default: true. # failOnEmptySource: true # boolean. Fail if no files found to copy. Default: false. -# displayName: "Upload Package to Deployment environment" - - - deployment: DEPLOY_${{ parameters.environmentName }} - displayName: "Deploy to ${{ parameters.environmentName }}" - environment: "${{ parameters.environmentName }}" - strategy: - runOnce: - deploy: - steps: - - download: none - - task: InstallSSHKey@0 - inputs: - knownHostsEntry: $(zosSSHKnownHost) # Known Hosts Entry for build LPAR - sshKeySecureFile: $(zosSSHPrivateKeySecureFile) # Required.Reference to uploaded Private SSH Key that in ADO Pipeline/Libary/SecureFile that is installed for sftp processes - - # Generate Deployment Plan. - - task: SSH@0 - inputs: - sshEndpoint: $(zosSSHConnection) - runOptions: "commands" - commands: ". ./.profile && wazideploy-generate.sh -w $(uniqueWorkspaceId) -a $(application) -b $(branch) -r deploy-logs-${{ parameters.environmentName }}/deployment-plan-report.html -I $(buildNumber) -R ${{ parameters.packageVersion }} -P $(pipelineType)" - # For reference when creating a local tar file, and skipping the upload. In this case the pipeline needs to provide the reference to the tar file. - # commands: ". ./.profile && wazideploy-generate.sh -w $(uniqueWorkspaceId) -a $(application) -i $(application).tar -r deploy-logs/deployment-plan-report.html" - readyTimeout: "20000" - displayName: "Generate Deployment Plan with Wazi Deploy" - - - # Perform Deployment. - - task: SSH@0 - inputs: - sshEndpoint: $(zosSSHConnection) - runOptions: "commands" - commands: ". ./.profile && wazideploy-deploy.sh -w $(uniqueWorkspaceId) -e EOLEB7-$(application)-${{ parameters.environmentName }}.yaml -l deploy-logs-${{ parameters.environmentName }}/evidences/evidence.yaml" - readyTimeout: "20000" - displayName: "Deploy Package with Wazi Deploy" - - # Generate Evidence. - - task: SSH@0 - inputs: - sshEndpoint: $(zosSSHConnection) - runOptions: "commands" - commands: ". ./.profile && wazideploy-evidence.sh -w $(uniqueWorkspaceId) -l deploy-logs-${{ parameters.environmentName }}/evidence.yaml -o deploy-logs-${{ parameters.environmentName }}/deployment-report.html" - readyTimeout: "20000" - displayName: "Generate deployment report with Wazi Deploy" - - # Pull logs and publish them. - - task: CmdLine@2 - inputs: - script: | - rm -rf "$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}" - mkdir "$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}" - sftp -r $(zosSFTPUser)@$(zosHostname):$(uniqueWorkspaceId)/deployPkgDir/deploy-logs-${{ parameters.environmentName }}/* "\"$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}\"" - displayName: "Retrieve deployment reports" - - - task: PublishBuildArtifacts@1 - inputs: - PathtoPublish: "$(Build.ArtifactStagingDirectory)/deploy-logs-${{ parameters.environmentName }}/" - ArtifactName: "deploy-logs-${{ parameters.environmentName }}" - publishLocation: "Container" - displayName: "Publish deployment logs" - - # Store logs in evidence inventory - - template: createWaziDeployIndex.yml - parameters: - evidenceFolder: "deploy-logs-${{ parameters.environmentName }}" - environmentName: "${{ parameters.environmentName }}" +# displayName: "Upload Package to Deployment environment" \ No newline at end of file diff --git a/Templates/AzureDevOpsPipeline/templates/tagging/createReleaseCandidate.yml b/Templates/AzureDevOpsPipeline/templates/tagging/createReleaseCandidate.yml index 1511e2f0..e6ad5e71 100644 --- a/Templates/AzureDevOpsPipeline/templates/tagging/createReleaseCandidate.yml +++ b/Templates/AzureDevOpsPipeline/templates/tagging/createReleaseCandidate.yml @@ -74,15 +74,15 @@ steps: # Set new Git tag if [ $rc -eq 0 ]; then - newVersionTag="${nextReleaseVersion}-$(buildNumber)" + newVersionTag="${nextReleaseVersion}.$(buildNumber)" echo "${PGM}: [INFO] Creating new release candidate tag with name: ${newVersionTag}" az rest \ --method post \ --headers "Authorization=Bearer $AZURE_DEVOPS_EXT_PAT" \ --url $(System.CollectionUri)/$(System.TeamProjectId)/_apis/git/repositories/$(application)/annotatedtags\?api-version\=7.1-preview.1 \ - --body "{ \"name\": \"${newVersionTag}\", \"taggedObject\": { \"objectId\": \"$(Build.SourceVersion)\" }, \"message\": \"Release Candidate created by pipeline $(Build.BuildNumber) \"} " \ - --resource 499b84ac-1321-427f-aa17-267ca6975798 + --body "{ \"name\": \"${newVersionTag}\", \"taggedObject\": { \"objectId\": \"$(Build.SourceVersion)\" }, \"message\": \"Release Candidate created by pipeline $(Build.BuildNumber) \"} " + # catch RC of rest command rc=$? diff --git a/Templates/Common-Backend-Scripts/README.md b/Templates/Common-Backend-Scripts/README.md index e60197a9..5c56d633 100644 --- a/Templates/Common-Backend-Scripts/README.md +++ b/Templates/Common-Backend-Scripts/README.md @@ -164,6 +164,7 @@ CLI parameter | Description -r `` | **Git repository URL**, can either be SSH or HTTPS-based. Example: `-r git@github.com:Organization/MortgageApplication.git` -b `` | **Git branch** that should be checked out. Example: `-b main` -a `` | (Optional) **Application name** to specify the directory into which git will clone. This is required for instance in the scenario if the repository name differs from the application name that is used in the build phase. Example: `-a CBSA`. +-c `` | (Optional) **http extra header parms** passed from the pipeline orchestrator that are added to the `-c http.extraHeader` option. This can be used to pass in an authentication token. **Dealing with private repositories** @@ -274,111 +275,6 @@ baselines: buildid: "build-7656" ``` -### dbbBuild.sh for zAppBuild framework - -This script implements the invocation of the [zAppBuild](https://github.com/IBM/dbb-zappbuild) framework. - -By design, the script implements the recommended working practice. It makes use of the [baselineRef sub-option](https://github.com/IBM/dbb-zappbuild/blob/main/docs/BUILD.md#perform-impact-build-by-providing-baseline-reference-for-the-analysis-of-changed-files) provided by zAppBuild to set the baseline Git hash. This is used to identify all the committed changes for the upcoming deliverable (that can be a planned release, a emergency fix, or a significant development initiative) - -The computation of the build configuration is performed by the [dbbBuildUtils.sh](utilities/dbbBuildUtils.sh) script. It leverages the [application baseline configuration](samples/baselineReference.config) file which is expected to be present in the `application-conf` directory in order to compute the baseline reference. - -#### Invocation - -The `dbbBuild.sh` script can be invoked as follows: - -``` -dbbBuild.sh -w MortApp/main/build-1 -a MortgageApplication -b main -p build -``` - -CLI parameter | Description ----------- | ---------------------------------------------------------------------------------------- --w `` | **Workspace directory**, an absolute or relative path that represents unique directory for this pipeline definition, that needs to be consistent through multiple steps. --a `` | **Application name** to be built, which is passed to zAppBuild as the `--application` parameter. --b `` | **Git branch** that is built. Used to compute various build properties such as the `--hlq` and build type. --p `` | (Optional) **Pipeline Type** to indicate a `build` pipeline (build only with test/debug options) or a `release` pipeline (build for optimized load modules), or if it runs in `preview` mode. --v | (Optional) zAppBuild verbose tracing flag. --t `` | (Optional) **zAppBuild Build Type** to specify the build type, such as `--fullBuild`, or `--impactBuild`. Arguments must be provided between quotes - e.g.: `-t '--fullBuild'`. Providing this parameter overrides the computation of the build type in the backend scripts. For instance can be used to initialize the DBB Metadatastore. --q `` |(Optional) **HLQ prefix**. Default is retrieved from the [pipelineBackend.config](pipelineBackend.config) file, if the configuration file is not modified - the default value is set to the user executing the script. - -**Pipeline type** - -The type of pipeline (`-p` option), is used to modify the operational behavior of the build framework on producing executables: -* `build` configures the build options for test/debug options. This is the **default**. -* `release` used to indicate to produce executables with the flag for performance-optimized runtime modules. This is required for the release pipelines which include release candidate packages. -* `preview` configures the build process to execute without producing any outputs. It is used to preview what the build will do. The pipeline should not have any subsequent actions. - -#### Output - -The section below contains the output that is produced by the `dbbBuild.sh` script. - -
- Script Output - -``` -dbbBuild.sh -w MortApp/main/build-1 -a MortgageApplication -b main -p build - -$ dbbBuild.sh -w MortApp/main/build-1 -a MortgageApplication -b main -p b - - -#### Script utility - dbbBuildUtils.sh - -The [dbbBuildUtils](utilities/dbbBuildUtils.sh) script is a core utility script providing the `computeBuildConfiguration()` method to compute the zAppBuild's options and parameters according to the branch naming conventions. For instance - -* `build type`, such as the `--impactBuild` zAppBuild build option, - * the baseline reference, `--baselineRef xxx`, where *xxx* is retrieved from the baselineReference.config file for integration branches, -* the configured topic branch build behavior (see parameter `featureBranchBuildBehaviour` in pipelineBackend.config), that can either be configured as - * `merge-base` (default) for cumulative builds that include all the changes added to the feature branch that flow to the integration branch. This setting automatically computes the merge-base commit, which defines the commit when the feature branch was forked. - * `incremental` for standard zAppBuild `--impactBuild` behavior. - * `cumulative` for computing all the differences between the topic branch and the integration branch by passing the `--baselineRef`. -* flag to produce test modules (`--debug` in zAppBuild) or modules improved for performance (production runtime modules). -* the `mainBuildBranch` to configure feature branch pipelines to clone the corresponding DBB dependency metadata collections. - ### zBuilder.sh for DBB zBuilder This script implements the invocation of the [zBuilder](https://www.ibm.com/docs/en/dbb/3.0?topic=building-zos-applications-zbuilder) framework. @@ -489,6 +385,111 @@ The [dbbzBuilderUtils](utilities/dbbzBuilderUtils.sh) script is a core utility s * the `mainBuildBranch` to configure feature branch pipelines to clone the corresponding DBB dependency metadata collections by generating a config.yaml that is passed into zBuilder. +### dbbBuild.sh for zAppBuild framework + +This script implements the invocation of the [zAppBuild](https://github.com/IBM/dbb-zappbuild) framework. + +By design, the script implements the recommended working practice. It makes use of the [baselineRef sub-option](https://github.com/IBM/dbb-zappbuild/blob/main/docs/BUILD.md#perform-impact-build-by-providing-baseline-reference-for-the-analysis-of-changed-files) provided by zAppBuild to set the baseline Git hash. This is used to identify all the committed changes for the upcoming deliverable (that can be a planned release, a emergency fix, or a significant development initiative) + +The computation of the build configuration is performed by the [dbbBuildUtils.sh](utilities/dbbBuildUtils.sh) script. It leverages the [application baseline configuration](samples/baselineReference.config) file which is expected to be present in the `application-conf` directory in order to compute the baseline reference. + +#### Invocation + +The `dbbBuild.sh` script can be invoked as follows: + +``` +dbbBuild.sh -w MortApp/main/build-1 -a MortgageApplication -b main -p build +``` + +CLI parameter | Description +---------- | ---------------------------------------------------------------------------------------- +-w `` | **Workspace directory**, an absolute or relative path that represents unique directory for this pipeline definition, that needs to be consistent through multiple steps. +-a `` | **Application name** to be built, which is passed to zAppBuild as the `--application` parameter. +-b `` | **Git branch** that is built. Used to compute various build properties such as the `--hlq` and build type. +-p `` | (Optional) **Pipeline Type** to indicate a `build` pipeline (build only with test/debug options) or a `release` pipeline (build for optimized load modules), or if it runs in `preview` mode. +-v | (Optional) zAppBuild verbose tracing flag. +-t `` | (Optional) **zAppBuild Build Type** to specify the build type, such as `--fullBuild`, or `--impactBuild`. Arguments must be provided between quotes - e.g.: `-t '--fullBuild'`. Providing this parameter overrides the computation of the build type in the backend scripts. For instance can be used to initialize the DBB Metadatastore. +-q `` |(Optional) **HLQ prefix**. Default is retrieved from the [pipelineBackend.config](pipelineBackend.config) file, if the configuration file is not modified - the default value is set to the user executing the script. + +**Pipeline type** + +The type of pipeline (`-p` option), is used to modify the operational behavior of the build framework on producing executables: +* `build` configures the build options for test/debug options. This is the **default**. +* `release` used to indicate to produce executables with the flag for performance-optimized runtime modules. This is required for the release pipelines which include release candidate packages. +* `preview` configures the build process to execute without producing any outputs. It is used to preview what the build will do. The pipeline should not have any subsequent actions. + +#### Output + +The section below contains the output that is produced by the `dbbBuild.sh` script. + +
+ Script Output + +``` +dbbBuild.sh -w MortApp/main/build-1 -a MortgageApplication -b main -p build + +$ dbbBuild.sh -w MortApp/main/build-1 -a MortgageApplication -b main -p b + + +#### Script utility - dbbBuildUtils.sh + +The [dbbBuildUtils](utilities/dbbBuildUtils.sh) script is a core utility script providing the `computeBuildConfiguration()` method to compute the zAppBuild's options and parameters according to the branch naming conventions. For instance + +* `build type`, such as the `--impactBuild` zAppBuild build option, + * the baseline reference, `--baselineRef xxx`, where *xxx* is retrieved from the baselineReference.config file for integration branches, +* the configured topic branch build behavior (see parameter `featureBranchBuildBehaviour` in pipelineBackend.config), that can either be configured as + * `merge-base` (default) for cumulative builds that include all the changes added to the feature branch that flow to the integration branch. This setting automatically computes the merge-base commit, which defines the commit when the feature branch was forked. + * `incremental` for standard zAppBuild `--impactBuild` behavior. + * `cumulative` for computing all the differences between the topic branch and the integration branch by passing the `--baselineRef`. +* flag to produce test modules (`--debug` in zAppBuild) or modules improved for performance (production runtime modules). +* the `mainBuildBranch` to configure feature branch pipelines to clone the corresponding DBB dependency metadata collections. + ## Packaging stage Depending on the Deployment Manager tool you are using, you can choose from either creating a package with the [PackageBuildOutputs](#packagebuildoutputssh) script that can be used with IBM Wazi Deploy, or the [UCD packaging](#ucdpackagingsh) script that creates the UCD shiplist and UCD component version. diff --git a/Templates/Common-Backend-Scripts/gitClone.sh b/Templates/Common-Backend-Scripts/gitClone.sh index ea9bb6fa..42da5b6f 100755 --- a/Templates/Common-Backend-Scripts/gitClone.sh +++ b/Templates/Common-Backend-Scripts/gitClone.sh @@ -30,6 +30,8 @@ # 2023/07/06 TLD 1.0.0 Initial Release # 2023/09/25 DB 1.1.0 Initial Release # 2024/03/01 DB 1.2.0 Updated Error Handling +# 2025/11/28 DB 1.3.0 Support for http.extraHeader +# argument to support Bearer tokens #=================================================================================== Help() { @@ -67,13 +69,18 @@ Help() { echo " -b - Name of the Branch to be cloned. " echo " Default=None, Required. " echo " " + echo " Ex: main " + echo " " echo " Optional: " echo " -a - Folder name to clone the " echo " application git repo " echo " " echo " Ex: MortgageApplication " echo " " - echo " Ex: refs/heads/main " + echo " -c - Bearer Token configuration " + echo " " + echo " Ex: 'Authorization: Bearer abc' " + echo " " echo " " exit 0 } @@ -82,7 +89,7 @@ Help() { # Customization # Configuration file leveraged by the backend scripts # Either an absolute path or a relative path to the current working directory -SCRIPT_HOME="`dirname "$0"`" +SCRIPT_HOME="$(dirname "$0")" pipelineConfiguration="${SCRIPT_HOME}/pipelineBackend.config" # Customization - End @@ -92,7 +99,7 @@ pipelineConfiguration="${SCRIPT_HOME}/pipelineBackend.config" #export BASH_XTRACEFD=1 # Write set -x trace to file descriptor PGM=$(basename "$0") -PGMVERS="1.2.0" +PGMVERS="1.3.0" USER=$(whoami) SYS=$(uname -Ia) @@ -101,11 +108,12 @@ ERRMSG="" Repo="" WorkDir="" Branch="" +GitConfigurationParameters=() application="" HELP=$1 if [ "$HELP" = "?" ]; then - Help + Help fi # Validate Shell environment @@ -137,14 +145,14 @@ fi # Get Options if [ $rc -eq 0 ]; then - while getopts "h:r:a:w:b:" opt; do + while getopts "h:r:a:w:b:c:" opt; do case $opt in h) Help ;; r) argument="$OPTARG" - nextchar="$(expr substr $argument 1 1)" + nextchar="${argument:0:1}" if [ -z "$argument" ] || [ "$nextchar" = "-" ]; then rc=4 ERRMSG=$PGM": [WARNING] Git Repository URL is required. rc="$rc @@ -155,7 +163,7 @@ if [ $rc -eq 0 ]; then ;; w) argument="$OPTARG" - nextchar="$(expr substr $argument 1 1)" + nextchar="${argument:0:1}" if [ -z "$argument" ] || [ "$nextchar" = "-" ]; then rc=4 ERRMSG=$PGM": [WARNING] Build Workspace Folder Name is required. rc="$rc @@ -166,7 +174,7 @@ if [ $rc -eq 0 ]; then ;; a) argument="$OPTARG" - nextchar="$(expr substr $argument 1 1)" + nextchar="${argument:0:1}" if [ -z "$argument" ] || [ "$nextchar" = "-" ]; then rc=4 ERRMSG=$PGM": [WARNING] Application Name is required. rc="$rc @@ -177,7 +185,7 @@ if [ $rc -eq 0 ]; then ;; b) argument="$OPTARG" - nextchar="$(expr substr $argument 1 1)" + nextchar="${argument:0:1}" if [ -z "$argument" ] || [ "$nextchar" = "-" ]; then rc=4 ERRMSG=$PGM": [WARNING] Branch Name is required. rc="$rc @@ -186,6 +194,17 @@ if [ $rc -eq 0 ]; then fi Branch="$argument" ;; + c) + argument="$OPTARG" + nextchar="${argument:0:1}" + if [ -z "$argument" ] || [ "$nextchar" = "-" ]; then + rc=4 + ERRMSG=$PGM": [WARNING] Git configuration parameter for http.extraHeader required. rc="$rc + echo $ERRMSG + break + fi + GitConfigurationParameters+=("${argument}") + ;; \?) Help rc=1 @@ -203,38 +222,37 @@ fi # # Validate Options -validateOptions(){ +validateOptions() { if [ -z "${Repo}" ]; then rc=8 ERRMSG=$PGM": [ERROR] Application Repository is required. rc="$rc echo $ERRMSG fi - + if [ -z "${Workspace}" ]; then rc=8 ERRMSG=$PGM": [ERROR] Unique Workspace Path is required. rc="$rc echo $ERRMSG else - if [[ ${Workspace:0:1} != "/" ]] ; then + if [[ ${Workspace:0:1} != "/" ]]; then if [ ! -d "${buildRootDir}" ]; then rc=8 ERRMSG=$PGM": [ERROR] Workspace Directory (${buildRootDir}) was not found. rc="$rc echo $ERRMSG fi - fi - fi - + fi + fi + if [ -z "${Branch}" ]; then rc=8 ERRMSG=$PGM": [ERROR] Branch Name is required. rc="$rc echo $ERRMSG fi - + } # - # Call validate Options if [ $rc -eq 0 ]; then validateOptions @@ -261,7 +279,7 @@ if [ $rc -eq 0 ]; then echo $PGM": [INFO] ** WorkDir:" $(getWorkDirectory) if [ ! -z "${application}" ]; then echo $PGM": [INFO] ** GitDir:" ${application} - else + else echo $PGM": [INFO] ** GitDir:" ${GitDir} fi echo $PGM": [INFO] ** Ref:" ${Branch} "->" ${BranchID} @@ -271,8 +289,8 @@ fi # # Set up to perform the clone of the Repo if [ $rc -eq 0 ]; then - - if [[ ${Workspace:0:1} != "/" ]] ; then + + if [[ ${Workspace:0:1} != "/" ]]; then cd ${buildRootDir} rc=$? fi @@ -302,16 +320,30 @@ fi # Clone the Repo to z/OS UNIX System Services with a re-Direct of STDERR to STDOUT if [ $rc -eq 0 ]; then - echo $PGM": [INFO] Preforming Git Clone of Repo ${Repo}, Ref ${BranchID} to $(getWorkDirectory)" - if [ ! -z "${application}" ]; then - CMD="git clone -b ${BranchID} ${Repo} ${application}" - else - CMD="git clone -b ${BranchID} ${Repo}" - fi + echo $PGM": [INFO] Performing Git Clone of Repo ${Repo}, Ref ${BranchID} to $(getWorkDirectory)" - echo $PGM": [INFO] ${CMD}" - ${CMD} 2>&1 - rc=$? + # Construct Clone cmd + if [ ! -z "${GitConfigurationParameters[@]}" ]; then + if [ ! -z "${application}" ]; then + echo $PGM": [INFO] git -c http.extraHeader= clone -b ${BranchID} ${Repo} ${application}" + git -c http.extraHeader="${GitConfigurationParameters[@]}" clone -b ${BranchID} ${Repo} ${application} 2>&1 + rc=$? + else + echo $PGM": [INFO] git -c http.extraHeader= clone -b ${BranchID} ${Repo}" + git -c http.extraHeader="${GitConfigurationParameters[@]}" clone -b ${BranchID} ${Repo} 2>&1 + rc=$? + fi + else + if [ ! -z "${application}" ]; then + echo $PGM": [INFO] git -clone -b ${BranchID} ${Repo} ${application}" + git clone -b ${BranchID} ${Repo} ${application} 2>&1 + rc=$? + else + echo $PGM": [INFO] git clone -b ${BranchID} ${Repo}" + git clone -b ${BranchID} ${Repo} 2>&1 + rc=$? + fi + fi if [ $rc -ne 0 ]; then ERRMSG=$PGM": [ERROR] Unable to Clone Repo ${Repo}, Ref ${BranchID}. rc="$rc @@ -326,7 +358,7 @@ if [ $rc -eq 0 ]; then if [ ! -z "${application}" ]; then cd ${application} - else + else cd ${GitDir} fi rc=$? @@ -363,7 +395,7 @@ fi if [ $rc -eq 0 ]; then ERRMSG=$PGM": [INFO] Clone Repository Complete. rc="$rc echo $ERRMSG -else +else ERRMSG=$PGM": [ERROR] Clone Repository Failed. Check Log. rc="$rc echo $ERRMSG fi