diff --git a/docs/2.0/docs/accountfactory/architecture/index.md b/docs/2.0/docs/accountfactory/architecture/index.md index 37ddb2ed91..d3c94c9050 100644 --- a/docs/2.0/docs/accountfactory/architecture/index.md +++ b/docs/2.0/docs/accountfactory/architecture/index.md @@ -28,13 +28,14 @@ sequenceDiagram Infra Live Repository ->> Pipelines: Trigger Account Added Pipelines ->> Core Accounts: Execute terragrunt to baseline account ``` + ## IAM roles Newly created accounts include IAM policies that define the scope of changes Pipelines is authorized to perform within AWS. Pipelines automatically assumes the necessary roles for each account when it detects changes. Detailed information about the provisioned roles can be found [here](/2.0/docs/pipelines/architecture/security-controls#roles-provisioned-by-devops-foundations). ## Delegated repositories -Delegated repositories enhance the architecture of infrastructure management by introducing additional layers of access control. When delegated repositories are created, Pipelines continues to manage new account security baselines within the `infrastructure-live-root` repository, while other infrastructure resources are managed in a new repository specific to the delegated account(s). +Delegated repositories enhance the architecture of infrastructure management by introducing additional layers of access control. When delegated repositories are created, Pipelines continues to manage new account security baselines within the `infrastructure-live-root` repository, while other infrastructure resources are managed in a new repository specific to the delegated account(s). Pipelines uses IAM roles from the `infrastructure-live-access-control` repository to deploy infrastructure in these delegated repositories. This setup enables the central platform team to define and restrict the scope of changes individual teams can make via Pipelines in delegated repositories. diff --git a/docs/2.0/docs/accountfactory/architecture/repository-topology.md b/docs/2.0/docs/accountfactory/architecture/repository-topology.md new file mode 100644 index 0000000000..f282218b50 --- /dev/null +++ b/docs/2.0/docs/accountfactory/architecture/repository-topology.md @@ -0,0 +1,149 @@ +# Repository Topology + +Gruntwork Account Factory provides an opinionated (but flexible) repository structure that supports organizations as they scale their infrastructure management across multiple AWS accounts. This approach is designed to help teams graduate from managing a handful of accounts with difficulty to being able to conveniently manage hundreds of accounts, all while maintaining high standards for security, compliance, and developer productivity. + +The repository topology is designed around a core principle: **centralized governance with distributed ownership**. Your platform team maintains control over critical security and compliance infrastructure, while application teams get the autonomy they need to move fast within well-defined guardrails. + +Understanding this repository structure will help you leverage Account Factory effectively and set your organization up for sustainable growth. + +## `infrastructure-live-root` + +Think of `infrastructure-live-root` as your organization's infrastructure command center. This repository, built from the [infrastructure-live-root-template](https://github.com/gruntwork-io/infrastructure-live-root-template), is where your platform team manages the foundational elements that every other AWS account depends on, and where your Account Factory workflow lives. + +This repository is the only repository with access to the AWS management account, and is trusted by IAM roles provisioned in all AWS accounts so that your platform team is able to provision infrastructure in them as necessary to prepare them for workloads. This is also where your platform team can provision new AWS accounts with consistent baselines whenever teams need them. You'll also manage critical organization-wide infrastructure like your AWS Landing Zone, central logging, and security services in this repository. + +Access to this repository is intentionally restricted to your most trusted platform team members. Every other piece of infrastructure in your organization can trace back to the foundational services configured here. + +### `infrastructure-live-root` workflows + +- **Account Factory:** (GitHub only) This is your self-service account vending machine. When someone needs a new AWS account (e.g. for a new application, environment, or team), they can trigger this workflow as the entrypoint for the account vending workflow. This workflow is triggered via [repository dispatch](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#repository_dispatch), which is a feature of GitHub Actions, and makes it possible to trigger the workflow from outside the repository using the GitHub API. + + The workflow accepts a simple JSON payload (there's even a handy customizable HTML form included to make this easy) and creates a pull request with all the necessary infrastructure code to provision and baseline the new account. Organizations typically customize this form to capture additional metadata like additional tags, cost center codes or conditional creation of potentially expensive security services like Macie or GuardDuty. + + :::tip + + The included HTML form is just a starting point. Organizations typically customize this form to capture additional metadata like additional tags, cost center codes or conditional creation of potentially expensive security services like Macie or GuardDuty. + + Once you have a good grasp of how the form works, and how it generates the JSON payload, you can even opt not to use the form at all, and instead trigger the workflow using the GitHub API directly from your internal platforms like ServiceNow or Jira. + + You can learn more about this in the ["Using the Account Factory Workflow" guide](/2.0/docs/accountfactory/guides/vend-aws-account). + + ::: + +- **Pipelines:** This is where Account Factory integrates with Gruntwork Pipelines to drive infrastructure changes via GitOps workflows. With Gruntwork Pipelines, every infrastructure change goes through a proper review process with pull requests, approvals, and controlled deployments. Your platform team gets the confidence of peer review while maintaining the ability to rapidly deploy critical infrastructure changes. + +:::tip + +While you can rename `infrastructure-live-root` during setup, keeping the name consistent with our documentation makes life easier for your team. You also *can* create multiple root repositories for complex organizational structures, but be sure that it's worth the additional complexity for your organization. It can be a significant source of operational overhead, and you might be better off delegating some infrastructure management to a separate repository with a [delegated infrastructure-live repository](#infrastructure-live-delegated). + +::: + +## `infrastructure-live-access-control` + +This is where you solve one of the biggest challenges in scaling infrastructure management: **How do you give teams the access they need (and only give them the exact access they need) to manage their own infrastructure?** + +:::tip + +This repository is optional for most users (Enterprise customers must provision this repository for delegated repository access control), but is a highly recommended best-practice for all customers. + +::: + +The `infrastructure-live-access-control` repository is your organization's permission control center. It manages all the IAM roles, policies, and permissions that determine what each team can do in their AWS accounts outside of the `infrastructure-live-root` repository. It provides a central place where application engineers and the platform team can collaborate to define and iterate on the access control policies for roles that can be assumed by [delegated infrastructure-live repositories](#infrastructure-live-delegated). + +Your application teams can _request_ the access they need here through pull requests, but your platform team maintains oversight by reviewing and approving these changes, and branch protection rules can ensure that they have final say in the approval process. No more bottlenecks where platform teams have to manually create every single IAM policy (and determine the appropriate level of access for each team), and no more security risks from teams having overly broad permissions. + +:::info + +**Delegated infrastructure management** is the practice of allowing developers to manage infrastructure in a self-service fashion. + +Instead of having your platform team manually provision every resource required for your entire organization (which doesn't always scale), you can let application teams manage their own infrastructure within clearly defined boundaries. Your platform team still entirely controls critical resources (e.g. AWS accounts, VPCs, security policies) but developers can deploy and update their applications without opening a ticket and waiting on the platform team. + +Most organizations find success with a hybrid approach: centralized control for anything that affects security or compliance, delegated management for everything else. Where you draw that line depends on your risk tolerance, team maturity, and the complexity of your organization. + +::: + +:::tip + +You can name the `infrastructure-live-access-control` repository whatever makes sense for your organization. Just keep it descriptive so future team members (and your future self) know exactly what it does. If you can keep the name similar to `infrastructure-live-access-control`, you probably should so that it's easier for your team to learn more about it when reading Gruntwork documentation. + +While you *could* split access control across multiple repositories for very large organizations, remember that multiple sources of truth for permissions can quickly become a security and operational nightmare. Start with one repository and only consider splitting if you have a compelling organizational reason. + +::: + +### `infrastructure-live-access-control` workflows + +- **Pipelines** - Every permission change goes through the same GitOps workflow as your other infrastructure. When someone proposes new IAM policies or role changes, the workflow runs a plan to show exactly what will change. Once approved and merged, it automatically applies those changes across your AWS accounts. + + This means your access control changes are auditable, reversible, and follow the same quality gates as the rest of your infrastructure. No more wondering who changed what permissions or scrambling to fix a misconfigured IAM policy. + +## `infrastructure-catalog` + +The `infrastructure-catalog` repository is your organization's internal infrastructure component library. This is where you build and maintain the custom Terragrunt/OpenTofu/Terraform resources that are specific to your organization's needs and standards, and can be reused throughout your organization. + +While Gruntwork provides battle-tested OpenTofu/Terraform modules for common infrastructure patterns, every organization has unique requirements. Maybe you need a special monitoring setup, custom networking configurations, or specific compliance controls. This repository is where those organization-specific modules live, get tested, and evolve alongside your infrastructure needs. + +The result? Instead of every team reinventing the wheel, they can leverage proven, tested components that follow your organization's best practices, and your engineers can work together to learn from each other and build on each other's work. + +:::tip + +Starting with a single `infrastructure-catalog` repository makes discoverability much easier (e.g. your teams won't have to guess where to find the standardized "database" module that follows your organization's best practices for security and cost savings). You can always split it later if your organization grows large enough that centralized module management becomes unwieldy. This can be the case if your central catalog starts to receive so many git tag updates that it becomes difficult to determine when a version bump in a module is a breaking change or not. + +Some large organizations also benefit from separate module repositories for different domains (security modules, application modules, etc.) or business units. Just make sure the benefits outweigh the complexity of managing multiple sources of truth. + +::: + +### `infrastructure-catalog` workflows + +- **Tests:** This is where your team can validate that your reusable infrastructure patterns are, in-fact, reliably reproducible. Every module gets automatically tested by spinning up real AWS resources, running comprehensive tests with [Terratest](https://terratest.gruntwork.io/), and then cleaning everything after the tests are run. This means your teams can trust that modules actually work consistently before they use them in production. This also means that you have a sandbox for ephemeral infrastructure that you can use to test out experimental changes to your infrastructure patterns before you commit to running them in the long-term in production. + +## `infrastructure-live-delegated` + +This is where you can start to empower more of your organization outside your central platform team to start managing their own infrastructure independently. **Delegated repositories** are how your organization grows from a small platform team managing everything to hundreds of developers deploying infrastructure independently while maintaining security and compliance best practices. + +:::tip + +Typical use cases for delegated repositories include: + +- Allowing a separate team to independently manage infrastructure relevant to a specific account (e.g. a mobile app team to manage their own database and application infrastructure). +- Enabling a GitHub Actions workflow in a repository to make restricted changes to infrastructure in a specific account (e.g. a repository with application code may need to build and push a container image to AWS ECR before it's picked up by ArgoCD in the cluster). +- Allowing a repository's GitHub Actions workflows to have read-only access to select resources within a specific account (e.g. a data science team may need to be granted read-only access to an S3 bucket in an AWS account to run their ML pipelines against real production data). + +::: + +These repositories represent individual teams or applications that have been granted specific, limited permissions to manage their own infrastructure. Think of them as specialized workshops where each team has exactly the tools they need for their job, but can't accidentally (or intentionally) mess with anyone else's work. + +The permissions for each delegated repository are carefully controlled by your `infrastructure-live-access-control` repository. Maybe the mobile app team needs to deploy containers and manage their databases, while the data science team needs different permissions for their ML pipelines. Each team gets exactly what they need, no more and no less. + +For Enterprise customers, Account Factory can even automatically create these delegated repositories as part of the account vending process. Request a new AWS account (or set of AWS accounts) for your team, and you automatically get a corresponding repository with all the right permissions to manage infrastructure in those account(s). + +## How it all fits together + +Here's how these repositories work together to create a scalable, secure infrastructure management system: + +```mermaid +erDiagram + infra-live-root ||--o| infra-live-access-control : "Delegated Access Control" + infra-live-access-control ||--o{ infra-live-delegated : "Delegated Infrastructure Management" + infra-live-root ||--o{ infra-live-delegated : "Vended (Enterprise)" + infra-live-root ||--o| infra-catalog : "" + infra-live-access-control ||--o| infra-catalog: "" + infra-live-delegated }o--o| infra-catalog: "" +``` + +:::note + +We've abbreviated `infrastructure` as `infra` in the diagram for brevity. + +::: + +**The flow in practice:** + +1. **Foundations first:** Your `infrastructure-live-root` repository sets up the foundational AWS infrastructure that everything else depends on: accounts, networking, security services. + +2. **Shared components:** The `infrastructure-catalog` provides reusable, tested modules that any team can use, ensuring consistency and reducing duplicate work across your organization. + +3. **Permissions next:** The `infrastructure-live-access-control` repository defines who can do what in each AWS account, creating the guardrails that keep your infrastructure secure as it scales. + +4. **Teams get autonomy:** Individual `infrastructure-live-delegated` repositories give teams the ability to manage their own infrastructure within the boundaries set by access control policies. + +This topology grows with you: start simple with just the root repository, build out your shared components in your catalog, add access control as you scale, introduce delegated repositories as teams need more autonomy. diff --git a/docs/2.0/docs/accountfactory/architecture/security-controls.md b/docs/2.0/docs/accountfactory/architecture/security-controls.md new file mode 100644 index 0000000000..d41576a336 --- /dev/null +++ b/docs/2.0/docs/accountfactory/architecture/security-controls.md @@ -0,0 +1,250 @@ +# Controls + +Gruntwork Account Factory employs a defense-in-depth approach to secure workflows across both GitHub and GitLab platforms. This document outlines the controls Account Factory uses to ensure that only infrastructure written in code and approved by a reviewer can be deployed to your AWS accounts. + +Account Factory relies on Pipelines to drive infrastructure changes via GitOps workflows, so make sure to read the [Pipelines security controls](/2.0/docs/pipelines/architecture/security-controls) for more details on how Pipelines secures workflows. + +## Least privilege principle + +Account Factory adheres to the principle of least privilege, configuring the AWS IAM roles vended as part of Account Factory onboarding to grant only the necessary permissions for infrastructure actions relevant for Account Factory to operate correctly, and to only trust the `infrastructure-live-root` repository for role assumption. + +By default, the only repository/group required to interact with infrastructure using Pipelines in Account Factory is the `infrastructure-live-root` repository/group. This contains the Infrastructure as Code for `management`, `logs`, `security`, and `shared` accounts. Access should be limited to a small, trusted group responsible for defining critical infrastructure, similar to the role of the `root` user in Unix systems. + +The roles used by the `infrastructure-live-root` repository are divided by responsibility: + +- Plan roles: Every account is provisioned with an AWS IAM role that `infrastructure-live-root` is trusted to assume from any branch that has read-only permissions to the resources in that account. +- Apply roles: Every account is provisioned with an AWS IAM role that `infrastructure-live-root` is trusted to assume on the deploy branch (e.g. `main`) with read-write permissions to the resources in that account. + +## Platform-Specific Access Controls + +import Tabs from "@theme/Tabs" +import TabItem from "@theme/TabItem" + + + + +- The AWS IAM role assumed via OIDC when pull requests are opened or updated has a trust policy that restricts access to the repository itself and provides read-only permissions +- The AWS IAM role assumed via OIDC when pull requests are merged into the `main` branch has a trust policy limiting access to the repository's `main` branch and granting write permissions +- Branch protection rules can be configured to require reviews and status checks +- GitHub App or machine user authentication options available + + + + +- The AWS IAM role assumed via OIDC when merge requests are opened or updated has a trust policy that restricts access to the group itself and provides read-only permissions +- The AWS IAM role assumed via OIDC when merge requests are merged into the `main` branch has a trust policy limiting access to the group's `main` branch and granting write permissions +- Protected branches can be configured to require approvals and pipeline success +- Machine user authentication required with group-level access configuration + + + + +## Infrastructure access control + +An optional `infrastructure-live-access-control` repository/group can manage access control for infrastructure provisioned in AWS accounts. Using this is a best practice for centralized and auditable access management. + +- Access to the `main` branch should be restricted to a small, trusted group managing infrastructure access +- The same least privilege principles apply: roles assumed for pull/merge requests have read-only permissions, while roles for merged changes have write permissions + +Unlike the infrastructure-live-root repository, this repository focuses on managing access control rather than defining infrastructure. You might grant write access to a broader group for managing access while maintaining tight control over the main branch. Encourage collaboration between platform teams and application engineers to review and refine access control continuously. + +## CI/CD Token Strategy + + + + +There are two ways GitHub users can configure Source Control Management (SCM) authentication for Account Factory: + +- The [Gruntwork.io GitHub App](https://github.com/apps/gruntwork-io) +- [GitHub Machine Users](/2.0/docs/pipelines/installation/viamachineusers) + +In general, we recommend using the Gruntwork.io GitHub App when possible, as it provides a more feature-rich, reliable and secure experience. + +Reasons you might need to set up Machine Users include: + +- Your organization does not allow installation of third-party GitHub apps. +- You are using Self-hosted GitHub Enterprise, and cannot use third-party GitHub apps due to your server settings. +- You are using a different SCM platform (e.g. GitLab). +- You want a fallback mechanism in case the Gruntwork.io GitHub App is temporarily unavailable. + +### GitHub App Installation Strategy (Recommended) + +No long-lived tokens are stored when using the Gruntwork.io GitHub App. Instead, short-lived tokens are generated at runtime on-demand using the Gruntwork.io GitHub App for authentication with GitHub. + +### Machine Users Installation Strategy + +Requires the following tokens be created: + +- `PIPELINES_READ_TOKEN`: Classic PAT with read access to required repositories +- `INFRA_ROOT_WRITE_TOKEN`: Fine-grained PAT with read/write access to infrastructure repositories +- `ORG_REPO_ADMIN_TOKEN`: Fine-grained PAT with admin access for repository management + +See [Setup via Machine Users](/2.0/docs/pipelines/installation/viamachineusers.md) for more details. + + + + +Requires the following tokens be created: + +- `PIPELINES_GITLAB_TOKEN`: A GitLab access token with `api` scope +- `PIPELINES_GITLAB_READ_TOKEN`: A GitLab access token with `read_repository` scope + +See [Setup via Machine Users](/2.0/docs/pipelines/installation/viamachineusers) for more details. + +Pipelines will also require access to Gruntwork's GitHub repositories, however those tokens are generated at runtime via the Gruntwork Management Portal. + + + + +## AWS credentials + +Pipelines requires IAM roles configured with trust policies to use OpenID Connect (OIDC) with your CI/CD platform. This eliminates the need for long-lived AWS credentials stored as secrets. + +### OpenID Connect Configuration + +Pipelines provisions an OpenID Connect identity provider in AWS IAM for each account, setting GitHub/GitLab as the provider and restricting the audience to AWS STS and your GitHub/GitLab organization. The Pipelines IAM role's trust policy ensures: + +- Only a single repository in your GitHub/GitLab organization can assume the role for plans. +- Only a single branch can assume the role for applies/destroys. + +For more details, see the [official AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html). Below is an example of a trust policy used by Pipelines. + + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "", + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::0123456789012:oidc-provider/token.actions.githubusercontent.com" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringLike": { + "token.actions.githubusercontent.com:sub": "repo:acme/infrastructure-live-root:ref:*" + } + } + } + ] +} +``` + + + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "", + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::0123456789012:oidc-provider/gitlab.com" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringLike": { + "gitlab.com:sub": "project_path:acme/projectprefix*:*" + } + } + } + ] +} + + +``` + + + + +Refer to [Configuring OpenId Connect in Amazon Web Services](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) for additional details. + +### Roles provisioned by Account Factory + +Pipelines automatically provisions specific roles in AWS accounts to support required infrastructure operations. These roles follow the naming pattern `-pipelines-`. + +For example: + +- The `root-pipelines-plan` role is used by Pipelines to plan changes in the `infrastructure-live-root` repository. + +These roles are designed to operate in a single repository and include a trust policy that only permits GitHub Actions workflows triggered by that repository to assume the role. Each role is provisioned in pairs: + +- `plan` roles, with read-only permissions, are used to execute Terragrunt plans for open pull requests. +- `apply` roles, with read/write permissions, are used to apply or destroy infrastructure changes for merged pull requests or direct pushes to the deploy branch (commonly `main`). + +This separation ensures that controls like [branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) and [CODEOWNERS files](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) can effectively govern infrastructure changes. + +#### `root-pipelines-plan` + +A read-only plan role for the `infrastructure-live-root` repository. + +- This role is one of the first created when setting up Account Factory. It is provisioned manually by the customer during the platform setup process. +- It exists in all accounts and handles tasks necessary for setting up AWS accounts. +- These roles are highly permissive among read-only roles as they manage foundational AWS account setups. + +#### `root-pipelines-apply` + +A read/write apply role for the `infrastructure-live-root` repository. + +- Like the plan role, this is one of the initial roles created during setup. +- It is broadly permissive to support foundational account setups and bootstrapping. + +#### `access-control-pipelines-plan` + +A read-only plan role for the `infrastructure-live-access-control` repository. + +- These roles are provisioned for new accounts but are not included in core accounts such as `management`, `logs`, `security`, or `shared`. +- They manage IAM roles and policies for vended accounts, facilitating infrastructure access control. + +#### `access-control-pipelines-apply` + +A read/write apply role for the `infrastructure-live-access-control` repository. + +- Similar to the plan role, these roles are provisioned for vended accounts but excluded from core accounts. +- They have permissions to manage IAM roles and policies for the accounts where they are provisioned. + +#### `delegated-pipelines-plan` + +A read-only plan role for delegated repositories, used by Pipelines Enterprise customers. + +- These roles are pre-configured to have minimal permissions, primarily for managing OpenTofu/Terraform state. +- A pull request will be opened in `infrastructure-live-access-control` during provisioning include documentation for adding additional permissions if necessary. +- Users should ensure that only the necessary _read-only_ permissions are granted for the specific delegated repository. + +:::note + +These roles have almost no permissions by default. They are pre-configured by default to only have access to OpenTofu/Terraform state, and the pull requests that are opened to provision them include documentation on how to add additional permissions as appropriate. + +It is up to the user provisioning these roles to ensure that this role has only the necessary _read-only_ permissions required to manage infrastructure changes relevant to the delegated repository. + +::: + +#### `delegated-pipelines-apply` + +A read/write apply role for delegated repositories. + +- Similar to the plan role, these roles are pre-configured with minimal permissions and are intended for managing OpenTofu/Terraform state. +- A pull request will be opened in `infrastructure-live-access-control` during provisioning include documentation for adding additional permissions if necessary. +- Users must ensure that the role has only the necessary _read/write_ permissions required for the delegated repository. + +:::note +The `delegated-pipelines-plan` and `delegated-pipelines-apply` roles are automatically provisioned for new delegated accounts. Enterprise customers will see pull requests created in the `infrastructure-live-access-control` repository to vend these roles with proper configurations. +::: + +## Trust boundaries + +A critical aspect of Pipelines' architecture is understanding its trust model. Since Pipelines runs within a CI/CD system, it has privileged access to your infrastructure resources (e.g. AWS accounts, VPCs, EC2 instances, etc.). + +Anyone with the ability to edit code in the `main` branch of your repositories inherently has the authority to make corresponding changes in your infrastructure resources. For this reason, it is important to follow the [Repository Access](/2.0/docs/pipelines/installation/viamachineusers#repository-access) guidelines to ensure appropriate access control. + +:::tip + +Each AWS IAM role provisioned through setup of [Gruntwork Account Factory](https://docs.gruntwork.io/account-factory/overview) is configured to trust a single repository (and, for apply roles, a single branch). If a role's permissions become overly broad, consider creating a new role with more granular permissions tailored to the specific use case. Use the `infrastructure-live-access-control` repository to define and manage these roles. + +::: diff --git a/docs/2.0/docs/accountfactory/guides/vend-aws-account.md b/docs/2.0/docs/accountfactory/guides/vend-aws-account.md index cad08f666f..204a60cc6d 100644 --- a/docs/2.0/docs/accountfactory/guides/vend-aws-account.md +++ b/docs/2.0/docs/accountfactory/guides/vend-aws-account.md @@ -42,6 +42,12 @@ The JSON payload approach provides greater flexibility for account vending, over #### Step 1 - Download the file +:::note + +This guide focuses on non-delegated repositories. Enterprise customers can also [use Account Factory to create new Delegated Repositories](/2.0/docs/accountfactory/guides/delegated-repositories). + +::: + Locate the inputs web page in your `infrastructure-live-root` repository at `.github/workflows/account-factory-inputs.html` and download it to your local machine. #### Step 2 - Populate the values diff --git a/docs/2.0/docs/pipelines/architecture/audit-logs.md b/docs/2.0/docs/pipelines/architecture/audit-logs.md index 3fc94094b4..264ad474e4 100644 --- a/docs/2.0/docs/pipelines/architecture/audit-logs.md +++ b/docs/2.0/docs/pipelines/architecture/audit-logs.md @@ -1,8 +1,11 @@ # Audit Logs -Gruntwork Pipelines provides an audit log that records which user performed specific operations in your AWS accounts as a result of a [Pipelines Action](/2.0/docs/pipelines/architecture/actions.md). +For certain cloud environments (for now, only AWS), Gruntwork Pipelines provides an audit log that records which user performed specific operations in your AWS accounts as a result of a [Pipelines Action](/2.0/docs/pipelines/architecture/actions.md). Pipelines does this via integration with native tooling for the cloud provider. + +## AWS Accessing AWS environments from a CI/CD system often involves assuming temporary credentials using OpenID Connect (OIDC). For platform-specific documentation, see: + - [GitHub OIDC Configuration](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) - [GitLab OIDC Configuration](https://docs.gitlab.com/ee/ci/cloud_services/aws/) @@ -10,11 +13,11 @@ Shared credentials can complicate tracking who performed specific actions in AWS ## How it works -Gruntwork Pipelines creates an audit log that tracks which user performed what action in which AWS account. It does this by setting the [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session name to include the initiating username, the Pipelines name, and the merge request/pull request or branch that triggered the action. Logging is handled through [AWS CloudTrail](https://aws.amazon.com/cloudtrail/), where session names appear in the `User name` field, making it easy to identify which user performed an action. For information on locating logs, see [where you can find logs](#where-you-can-find-logs) and [querying data](#querying-data). +Gruntwork Pipelines creates an audit log that tracks which user performed what action in which AWS account. It does this by setting the [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session name to include the initiating username, the Pipelines name, and the merge/pull request or branch that triggered the action. Logging is handled through [AWS CloudTrail](https://aws.amazon.com/cloudtrail/), where session names appear in the `User name` field, making it easy to identify which user performed an action. For information on locating logs, see [where you can find logs](#where-you-can-find-logs) and [querying data](#querying-data). ### What gets logged -Logs are generated for all operations performed by Gruntwork Pipelines across every AWS account. These logs leverage [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session names to clearly label sessions with the username that requested the change and the associated merge request/pull request or branch. +Logs are generated for all operations performed by Gruntwork Pipelines in AWS across every AWS account. These logs leverage [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session names to clearly label sessions with the username that requested the change and the associated merge/pull request or branch. Each CloudTrail event linked to API calls from Pipelines [Actions](/2.0/docs/pipelines/architecture/actions.md) includes the session name in the `userIdentity` field. For example, if the user `SomeUserInYourOrg` initiated the 123rd request in your repository, the `userIdentity` field in a corresponding CloudTrail event would provide details such as the following. @@ -58,13 +61,17 @@ By combining this data with a [query service](#querying-data), you can analyze t Pipelines employs a naming scheme that integrates the user who triggered the Pipelines [Action](/2.0/docs/pipelines/architecture/actions.md) along with the request or branch that initiated the action. The AWS STS session name is formatted as follows: `-via-GWPipelines@(PR-|)`. -#### For merge request/pull request events +#### For merge/pull request events + When Pipelines runs in response to a request event (opened, updated, or reopened), the session name includes the user who made the most recent commit on the branch and the request number. For instance: + - If the user `SomeUserInYourOrg` created request number `123`, the session name would be: `SomeUserInYourOrg-via-GWPipelines@PR-123`. #### For merged requests + When Pipelines runs after a request is merged, the session name reflects the user who performed the merge and the deploy branch name (e.g., `main`). For example: + - If the user `SomeUserInYourOrg` merged a request to the branch `main`, the session name would be: `SomeUserInYourOrg-via-GWPipelines@main`. diff --git a/docs/2.0/docs/pipelines/architecture/ci-workflows.md b/docs/2.0/docs/pipelines/architecture/ci-workflows.md index e941dd3f97..65ca5b4d55 100644 --- a/docs/2.0/docs/pipelines/architecture/ci-workflows.md +++ b/docs/2.0/docs/pipelines/architecture/ci-workflows.md @@ -21,6 +21,7 @@ jobs: include: - component: gitlab.com/gruntwork-io/pipelines-workflows/pipelines@3 ``` + ## Workflow versioning @@ -42,33 +43,32 @@ If you [fork the Gruntwork Workflows](https://docs.gruntwork.io/2.0/docs/pipelin The `pipelines-workflows` repository includes the following reusable workflows: -- `pipelines-drift-detection.yml` - Used for [Pipelines Drift Detection](/2.0/docs/pipelines/concepts/drift-detection) in all repositories with Drift Detection installed. -- `pipelines-root.yml` - The core Pipelines workflow for the `infrastructure-live-root` repository, providing core plan/apply functionality and account vending. -- `pipelines-unlock.yml` - Used to manually unlock state files in all repositories. +- `pipelines-drift-detection.yml` - (Enterprise only) Used for [Pipelines Drift Detection](/2.0/docs/pipelines/concepts/drift-detection) in all repositories with Drift Detection installed. +- `pipelines-root.yml` - (Account Factory only) The core Pipelines workflow for the `infrastructure-live-root` repository, providing core plan/apply functionality and account vending. +- `pipelines-unlock.yml` - (AWS only) Used to manually unlock state files in all repositories. - `pipelines.yml` - The core Pipelines workflow for `infrastructure-live-access-control` and delegated repositories, supporting plan/apply operations. +If you are using [Gruntwork Account Factory](/2.0/docs/accountfactory/concepts/), the following workflows are typically present: -In your repositories, the following workflows are typically present: - -#### infrastructure-live-root +### infrastructure-live-root - `account-factory.yml` - A standalone workflow independent of `pipelines-workflows`. - `pipelines-drift-detection.yml` (Enterprise only) - Uses the Gruntwork `pipelines-drift-detection.yml` workflow. - `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow. - `pipelines.yml` - Uses `pipelines-root.yml`. -#### infrastructure-live-access-control + +### infrastructure-live-access-control - `pipelines-drift-detection.yml` (Enterprise only) - Uses the Gruntwork `pipelines-drift-detection.yml` workflow. -- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow. +- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow (AWS only). - `pipelines.yml` - Uses `pipelines.yml`. -#### infrastructure-live-delegated ([Vended Delegated Repositories](/2.0/docs/accountfactory/guides/delegated-repositories)) +### infrastructure-live-delegated ([Vended Delegated Repositories](/2.0/docs/accountfactory/guides/delegated-repositories)) - `pipelines-drift-detection.yml` - Uses the Gruntwork `pipelines-drift-detection.yml` workflow. - `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow. - `pipelines.yml` - Uses `pipelines.yml`. - diff --git a/docs/2.0/docs/pipelines/architecture/components.md b/docs/2.0/docs/pipelines/architecture/components.md deleted file mode 100644 index 902c4c2338..0000000000 --- a/docs/2.0/docs/pipelines/architecture/components.md +++ /dev/null @@ -1,23 +0,0 @@ -# Components Architecture - -Pipelines consists of two main components: the orchestrator and the executor. The orchestrator identifies necessary jobs, while the executor performs those tasks and updates AWS resources accordingly. - -## Orchestrator - -The orchestrator analyzes each infrastructure change in a pull request or git commit, categorizes the type of change (e.g., `AccountsAdded`, `ModuleChanged`, `EnvCommonChanged`), and identifies the appropriate pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to execute based on the type of change. - -## Executor - -The executor receives a pipeline action and an infrastructure change as input and executes the specified action on the change. For example, when responding to `AccountsAdded` events on merge, the executor may create a follow-up pull request in the `infrastructure-live-root` repository to include additional IaC code for baselining the newly added accounts. - -## Execution flow - -Pipelines begins with an event in GitHub/GitLab, such as the creation, update, or reopening of a merge request/pull request, or a push to `main` (e.g., merging a pull request). The orchestrator determines the set of infrastructure changes (`infra-change set`) and selects the appropriate action for each change. For every change in the set, the executor performs the necessary action and logs the results in GitHub/GitLab, attaching them to the merge request/pull request that triggered the workflow. - -## Trust boundaries - -A critical aspect of Pipelines' architecture is understanding its trust model. Since Pipelines runs within a CI/CD system, it has privileged access to your AWS accounts. - -Anyone with the ability to edit code in the `main` branch of your repositories inherently has the authority to make corresponding changes in your AWS accounts. For this reason, it is important to follow the [Repository Access](/2.0/docs/pipelines/installation/viamachineusers#repository-access) guidelines to ensure appropriate access control. - -Additionally, each AWS IAM role provisioned through DevOps Foundations is configured to trust a single repository (and, for apply roles, a single branch). If a role's permissions become overly broad, consider creating a new role with more granular permissions tailored to the specific use case. Use the `infrastructure-live-access-control` repository to define and manage these roles. diff --git a/docs/2.0/docs/pipelines/architecture/execution-flow.md b/docs/2.0/docs/pipelines/architecture/execution-flow.md new file mode 100644 index 0000000000..b7cbbbc47b --- /dev/null +++ b/docs/2.0/docs/pipelines/architecture/execution-flow.md @@ -0,0 +1,15 @@ +# Execution flow + +Pipelines begins doing work in response to an event in GitHub/GitLab, such as the creation, update, or merging of a pull/merge request, or a push to a deploy branch (e.g., `main` or `master`). Pipelines does this in the native CI/CD feature offered by Source Control Management (SCM) platforms (GitHub Actions for GitHub, GitLab CI/CD Pipelines for GitLab). + +The flow of this work consists of two main stages: the orchestrator and the executor. The orchestrator identifies and categorizes work into a set of infrastructure changes (`infra-change set`) based on the contents of a pull/merge request or push to the deploy branch, while the executor performs those tasks and updates infrastructure accordingly. + +## Orchestrator + +The orchestrator analyzes each infrastructure change in a pull request or git commit, categorizes the type of change (e.g., `AccountsAdded`, `ModuleChanged`, `EnvCommonChanged`), and identifies the appropriate pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to execute based on the type of change. + +## Executor + +The executor receives as inputs a pipeline action (e.g. `terragrunt plan`) and a specific unit of infrastructure that has been changed (e.g. `/path/to/changed-unit/terragrunt.hcl`) and executes the specified action on the specified unit. + +For example, when responding to a `ModuleUpdated` event for `/some/unit/terragrunt.hcl`, the executor might execute a `terragrunt apply` on `/some/unit/terragrunt.hcl`. Or when responding to `AccountsAdded` events on merge, the executor may create a follow-up pull request in the `infrastructure-live-root` repository to include additional IaC code for baselining the newly added accounts. diff --git a/docs/2.0/docs/pipelines/architecture/index.md b/docs/2.0/docs/pipelines/architecture/index.md index c810f39522..e75a7905b7 100644 --- a/docs/2.0/docs/pipelines/architecture/index.md +++ b/docs/2.0/docs/pipelines/architecture/index.md @@ -1,129 +1,218 @@ # Architecture -Gruntwork Pipelines is designed to provide flexibility, enabling you to utilize the components you need to manage your infrastructure in a way that aligns with your organization's requirements. +Gruntwork Pipelines, at its core, is a single binary that customers can download using their Gruntwork subscription to "do the right thing" when it comes to managing Infrastructure as Code using Terragrunt using GitOps workflows. Pipelines runs in customer CI/CD pipelines (not Gruntwork servers or any other dedicated server you have to maintain) to handle all the complexity of performing IaC CI/CD. +Outside of the main binary, Pipelines has several other components that work together to make it all work. These components are all provided by Gruntwork, and we work with our customers to ensure that they are configured correctly to work together. -Understanding the components and their structure will help you use Pipelines and associated Infrastructure as Code (IaC) effectively. +## CI/CD pipelines -## `infrastructure-live-root` +By design, customers run the binary as part of their CI/CD pipelines (e.g. GitHub Actions, GitLab CI, etc.). As such, Gruntwork provides out-of-the-box CI/CD configurations for supported platforms when customers sign up for Gruntwork Pipelines. -The `infrastructure-live-root` repository serves as the root of your infrastructure and is provisioned using the [infrastructure-live-root-template](https://github.com/gruntwork-io/infrastructure-live-root-template) template repository. +We likewise provide CI/CD configurations for [Gruntwork Account Factory](https://docs.gruntwork.io/account-factory/overview). -This repository is where you manage sensitive resources such as the Landing Zone and Organization services for AWS. Typically, access to this repository is restricted to a small group of trusted users. +When using Gruntwork Pipelines without Gruntwork Account Factory, customers are responsible for configuring their repositories to use the appropriate CI/CD configuration for that platform (see [Adding Pipelines to an Existing Repository](/2.0/docs/pipelines/installation/addingexistingrepo) for more information). This code is typically fairly minimal, and the majority of the work is done by reusable workflows made available by Gruntwork, and the binary itself. -All other infrastructure managed with Gruntwork software ultimately depends on resources configured in this repository. +## Cloud resources -### Workflows +When Pipelines is used to manage infrastructure resources in cloud environments, it needs to be able to authenticate to the cloud provider. Pipelines supports authentication out of the box for the following cloud providers: -- **Account Factory:** (GitHub only) Provides an API for interacting with the Gruntwork Account Factory. It uses a [repository dispatch](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#repository_dispatch) to create AWS account requests. +- [AWS](/2.0/docs/pipelines/concepts/cloud-auth/aws) +- [Azure](/2.0/docs/pipelines/concepts/cloud-auth/azure) +- [Custom](/2.0/docs/pipelines/concepts/cloud-auth/custom) (build your own support for your cloud provider) - This workflow uses a [repository dispatch](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#repository_dispatch) to create a standard AWS account creation request in the repository. +The way in which Pipelines authenticates to these cloud providers (with the exception of Custom authentication) is via OpenID Connect (OIDC). - The workflow payload is a JSON object, which can be constructed using the sample HTML file included in the repository. This file can be customized for organizational needs, such as adding tagging fields or additional context. +With OIDC, Pipelines is able to generate temporary credentials for granular authentication to cloud providers, using the context of the pull request or push to the deploy branch. - :::tip +e.g. When a pull request is opened, Pipelines might determine that the IaC being edited is in an [environment](/2.0/reference/pipelines/configurations-as-code#environment-blocks) that is configured to authenticate to AWS using OIDC, and generate temporary read-only credentials (as that's what's required for plans in pull requests) to authenticate with AWS using the role defined for that environment. - This HTML file can be customized as needed to fit the needs of your organization. You may want to add additional fields to the form for tagging purposes, or to provide additional context to the person who will be approving the account vend. +## SCM Authentication - You don't even need to use the form at all if you don't want to. An alternative would be to use a service like ServiceNow to populate the requisite fields, then trigger the workflow using the GitHub API. +In addition to authenticating to cloud providers, Pipelines also needs to be able to authenticate to Source Control Management (SCM) platforms (e.g. GitHub, GitLab) to fetch resources (e.g. IaC code, reusable CI/CD code and the Pipelines binary itself). - You can learn more about this [here](/2.0/docs/accountfactory/guides/vend-aws-account). +Pipelines supports authentication to the following SCM platforms: - ::: +- GitHub +- GitLab -- **Pipelines:** Manages infrastructure changes in a GitOps fashion. While the workflow permissions are mostly read-only for proposing changes, they include write access to apply infrastructure changes upon merging. +The way in which Pipelines authenticates to these SCM platforms differs slightly, due to the different ways in which they support authentication, and the needs of customers. -:::tip +### Gruntwork.io GitHub App Authentication -The `infrastructure-live-root` repository can be renamed during the bootstrap process, but giving it a similar name to `infrastructure-live-root` is recommended for clarity when using Gruntwork documentation. Multiple root repositories can be created if needed, but this increases complexity and operational overhead. -It also doesn't have to be the only root repository in your organization. You can have multiple root repositories if you have a complex organization structure that requires it. Make sure to evaluate the trade-offs of having multiple root repositories before doing so. It can be a significant source of complexity and operational overhead. +Customers using Github as their SCM platform can install the [Gruntwork.io GitHub App](https://github.com/apps/gruntwork-io) to authenticate with Github. This app provides a more feature-rich and secure experience, and is the recommended method of authentication for most customers if they can use it. -::: +When using the Gruntwork.io GitHub App, users are able to avoid the need to provision any machine users or long-lived Personal Access Tokens (PATs) to authenticate with Github to download the Pipelines binary or access other repositories they want to access in their CI/CD pipelines. Control of the permissions the Gruntwork.io GitHub App grants when installed is done via the [Gruntwork Developer Portal](#gruntwork-developer-portal). -## `infrastructure-live-access-control` +### GitHub Machine Users Authentication -The `infrastructure-live-access-control` repository manages access control for your infrastructure and is provisioned during the bootstrap process in the `infrastructure-live-root` repository. While only necessary for Enterprise customers, it is recommended for all users. +Customers using GitHub as their SCM platform, but are unable (or don't wish to) use the Gruntwork.io GitHub App, and customers using a different SCM platform (e.g. GitLab) use GitHub Machine Users to authenticate with GitHub. -This repository handles IAM roles, policies, and permissions for delegated infrastructure management. It allows application engineers to propose access control changes, while a central platform team reviews and approves them. +All customers must authenticate with GitHub in some way for the core requirement to download the Pipelines binary using their Gruntwork subscription. Customers that are using GitHub as their SCM platform, but aren't using the Gruntwork.io GitHub App also use GitHub Machine Users to authenticate with GitHub to download IaC code and reusable GitHub Actions code. Customers using a different SCM platform (e.g. GitLab) only use GitHub Machine Users to authenticate with GitHub to download the Pipelines binary. -More access can be granted to this repository than the `infrastructure-live-root` repository, but it should still be treated as a sensitive resource. Organizations typically find it useful to have restrictions on who can approve and merge changes to this repository, and allow for users to propose changes in a self-service fashion. This allows for application workload engineers to propose changes to the access control their workflows need, while allowing a central platform team to review and approve those changes instead of having to author the changes themselves. +### GitLab Machine Users Authentication -:::info +Customers using GitLab as their SCM platform use GitLab Machine Users to authenticate with GitLab to download reusable GitLab CI/CD code, and access other IaC repositories. -Delegated infrastructure management is the practice of allowing developers to manage infrastructure in a self-service fashion. +## Gruntwork Developer Portal -This is in contrast to centralized infrastructure management, where a small number of people manage all the infrastructure for the entire organization. +The Gruntwork Developer Portal (hosted at [app.gruntwork.io](https://app.gruntwork.io)) is a web-based interface that customers use to manage their Gruntwork subscription. This includes the ability to install the Gruntwork.io GitHub App and associate it with a Gruntwork organization, and manage the access that the app has to relevant GitHub resources in customer GitHub organizations. -Organizations frequently use a combination of both centralized and delegated infrastructure management to balance the need for control and security with the need for agility and speed. +It is also used to manage the access that GitHub Machine Users have to relevant Gruntwork owned repositories, made available via customer Gruntwork subscriptions. -e.g. Centralized management for core infrastructure like AWS accounts, VPCs, OIDC providers and powerful IAM roles, and delegated management for application infrastructure like container images, AWS ECS services, and S3 buckets. +## Architecture Overview -The exact balance will depend on your organization's needs. +The Gruntwork Pipelines architecture varies depending on your Source Control Management (SCM) platform, authentication method, and cloud provider. -::: +Below are simplified diagrams for each supported configuration, to provide a high-level overview of the architecture: -:::tip - -The `infrastructure-live-access-control` repository does not have to be named "infrastructure-live-access-control". You can name it whatever you like. It is highly recommended that the repository is named something similar to `infrastructure-live-access-control` to make it clear what it is when reading Gruntwork documentation, however. - -It also doesn't have to be the only access control repository in your organization. You can have multiple access control repositories if you have a complex organization structure that requires it. Make sure to evaluate the trade-offs of having multiple access control repositories before doing so. It can be a significant source of complexity and operational overhead. Also note that doing this will sacrifice some of the benefits of having a single source of truth for access control. - -::: - -### Workflows - -- **Pipelines** - This workflow will be used by the `infrastructure-live-access-control` repository to manage access control infrastructure in response to changes in the repository in a GitOps fashion. - - While the permissions for this workflow are largely read-only when proposing access control changes, the workflow also has the ability to make changes to relevant access control infrastructure when the changes are merged. - -## `infrastructure-catalog` - -The `infrastructure-catalog` repository stores OpenTofu/Terraform modules created for internal use. It is optionally provisioned during the bootstrap process of the `infrastructure-live-root` repository. - -This repository is optional but recommended for managing reusable infrastructure code. Customers often combine Gruntwork modules with custom modules stored here to extend functionality. - -:::tip - -While `infrastructure-catalog` can be renamed, keeping a consistent name is recommended for clarity in documentation. Multiple module repositories can be created if necessary, but consider the trade-offs between centralized and decentralized approaches. - -It can be advantageous to have one repository for all modules to make it easier to find and share modules across your organization. However, it can also be advantageous to have multiple repositories if you have different teams that need to manage their own modules, or want to have different modules available to different teams within your organization. - -::: - -### Workflows - -- **Tests:** Validates module functionality by provisioning them in real environments, running [Terratests](https://github.com/gruntwork-io/terratest), and tearing them down. This workflow ensures modules work as expected. - - -## `infrastructure-live-delegated` - -One of the primary benefits of IaC Foundations is the streamlined delegation of infrastructure management. For the sake of discussion in Gruntwork documentation, we refer to repositories that have been granted delegated permissions to interact with infrastructure as "delegated repositories", or `infrastructure-live-delegated`. Their permissions are granted by the `infrastructure-live-access-control` repository. - -These repositories can be created manually by customers for specific purposes. For example, an application repository may need permissions to build and push a container image to AWS ECR, or to deploy an update to an ECS service. These permissions can be delegated by the `infrastructure-live-access-control` repository to a specific repository that needs those permissions. - -Enterprise customers can also expect the creation and management of delegated repositories centrally in the `infrastructure-live-root` repository. This is an Enterprise-only feature that allows for the creation of delegated repositories with largely the same degree of infrastructure management as the `infrastructure-live-root` repository itself. This is useful for organizations that want to allow large degrees of infrastructure management (e.g. entire AWS accounts) without having to manually provision and manage the repositories that need those permissions. -## Entity relationship diagram +### GitHub with Gruntwork.io App (Recommended) ```mermaid -erDiagram - infra-live-root ||--o| infra-live-access-control : "Delegated Access Control" - infra-live-access-control ||--o{ infra-live-delegated : "Delegated Infrastructure Management" - infra-live-root ||--o{ infra-live-delegated : "Vended (Enterprise)" - infra-live-root ||--o| infra-modules : "" - infra-live-access-control ||--o| infra-modules: "" - infra-live-delegated }o--o| infra-modules: "" +graph TB + subgraph "Customer GitHub" + GH[GitHub Repository] + GHA[GitHub Actions] + IaC[Infrastructure as Code] + end + + subgraph "Gruntwork Services" + GDP[Gruntwork Developer Portal
app.gruntwork.io] + GApp[Gruntwork.io GitHub App] + end + + subgraph "Pipelines Runtime" + Binary[Gruntwork Pipelines Binary] + end + + subgraph "Cloud Provider" + OIDC[OpenID Connect] + Cloud[Cloud Resources
AWS/Azure/Custom] + end + + %% Main flow + GH --> IaC + IaC --> GHA + GHA --> Binary + + %% Authentication + GDP --> GApp + GApp --> GH + Binary --> GApp + + %% Cloud access + Binary --> OIDC + OIDC --> Cloud + + classDef customer fill:#b3e5fc,stroke:#01579b,stroke-width:2px,color:#000000 + classDef gruntwork fill:#e1bee7,stroke:#4a148c,stroke-width:2px,color:#000000 + classDef runtime fill:#f8bbd9,stroke:#880e4f,stroke-width:2px,color:#000000 + classDef cloud fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px,color:#000000 + + class GH,GHA,IaC customer + class GDP,GApp gruntwork + class Binary runtime + class OIDC,Cloud cloud ``` -:::note - -The term `infrastructure` is abbreviated as `infra` in the diagram for simplicity. +### GitHub with Machine Users -::: - -The `infrastructure-live-root` repository serves as the central hub for managing all infrastructure. - -- Users who choose to delegate access control can use the `infrastructure-live-root` repository to provision the necessary `pipelines-access-control` roles in AWS accounts. These roles allow access control to be managed within the `infrastructure-live-access-control` repository. +```mermaid +graph TB + subgraph "Customer GitHub" + GH[GitHub Repository] + GHA[GitHub Actions] + IaC[Infrastructure as Code] + end + + subgraph "Gruntwork Services" + GDP[Gruntwork Developer Portal
app.gruntwork.io] + GMU[GitHub Machine User
Long-lived PAT] + end + + subgraph "Pipelines Runtime" + Binary[Gruntwork Pipelines Binary] + end + + subgraph "Cloud Provider" + OIDC[OpenID Connect] + Cloud[Cloud Resources
AWS/Azure/Custom] + end + + %% Main flow + GH --> IaC + IaC --> GHA + GHA --> Binary + + %% Authentication + GDP --> GMU + GMU --> GH + Binary --> GMU + + %% Cloud access + Binary --> OIDC + OIDC --> Cloud + + classDef customer fill:#b3e5fc,stroke:#01579b,stroke-width:2px,color:#000000 + classDef gruntwork fill:#e1bee7,stroke:#4a148c,stroke-width:2px,color:#000000 + classDef runtime fill:#f8bbd9,stroke:#880e4f,stroke-width:2px,color:#000000 + classDef cloud fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px,color:#000000 + + class GH,GHA,IaC customer + class GDP,GMU gruntwork + class Binary runtime + class OIDC,Cloud cloud +``` -- Users who opt to delegate infrastructure management can use the `infrastructure-live-access-control` repository to provision the required `pipelines-delegated` roles in AWS accounts. These roles enable infrastructure management to be handled through `infrastructure-live-delegated` repositories. +### GitLab -- Users who prefer to centralize module management can utilize the `infrastructure-catalog` repository to create and maintain reusable, vetted modules. These modules can be shared across the organization and accessed by any repository as needed. +```mermaid +graph TB + subgraph "Customer GitLab" + GL[GitLab Repository] + GLC[GitLab CI] + IaC[Infrastructure as Code] + end + + subgraph "Gruntwork Services" + GDP[Gruntwork Developer Portal
app.gruntwork.io] + GMU[GitHub Machine User
For Pipelines Binary] + GLMU[GitLab Machine User
Access Token] + end + + subgraph "Pipelines Runtime" + Binary[Gruntwork Pipelines Binary] + end + + subgraph "Cloud Provider" + OIDC[OpenID Connect] + Cloud[Cloud Resources
AWS/Azure/Custom] + end + + %% Main flow + GL --> IaC + IaC --> GLC + GLC --> Binary + + %% Authentication - GitLab + GDP --> GLMU + GLMU --> GL + Binary --> GLMU + + %% Authentication - GitHub (for binary download) + GDP --> GMU + Binary --> GMU + + %% Cloud access + Binary --> OIDC + OIDC --> Cloud + + classDef customer fill:#b3e5fc,stroke:#01579b,stroke-width:2px,color:#000000 + classDef gruntwork fill:#e1bee7,stroke:#4a148c,stroke-width:2px,color:#000000 + classDef runtime fill:#f8bbd9,stroke:#880e4f,stroke-width:2px,color:#000000 + classDef cloud fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px,color:#000000 + + class GL,GLC,IaC customer + class GDP,GMU,GLMU gruntwork + class Binary runtime + class OIDC,Cloud cloud +``` diff --git a/docs/2.0/docs/pipelines/architecture/security-controls.md b/docs/2.0/docs/pipelines/architecture/security-controls.md index 8f26b5e596..6b88e281b8 100644 --- a/docs/2.0/docs/pipelines/architecture/security-controls.md +++ b/docs/2.0/docs/pipelines/architecture/security-controls.md @@ -1,12 +1,10 @@ # Controls -Gruntwork Pipelines employs a defense-in-depth approach to secure workflows across both GitHub and GitLab platforms. This document outlines the controls Pipelines uses to ensure that only infrastructure written in code and approved by a reviewer can be deployed to your AWS accounts. +Gruntwork Pipelines employs a defense-in-depth approach to secure workflows across both GitHub and GitLab platforms. This document outlines the controls Pipelines uses to ensure that only infrastructure written in code and approved by a reviewer can be deployed to your cloud environments (e.g. AWS accounts). ## Least privilege principle -Pipelines adheres to the principle of least privilege, granting only the necessary permissions for infrastructure actions. - -By default, the only repository/group required to interact with infrastructure using Pipelines in DevOps Foundations is the `infrastructure-live-root` repository/group. This contains infrastructure code for management, logs, security, and shared accounts. Access should be limited to a small, trusted group responsible for defining critical infrastructure, similar to the role of the `root` user in Unix systems. +Pipelines adheres to the principle of least privilege, granting only the necessary permissions for infrastructure actions using the context of a pull request or push to the deploy branch to determine the environment(s) to authenticate to, and how to authenticate to them. ## Platform-Specific Access Controls @@ -16,43 +14,35 @@ import TabItem from "@theme/TabItem" -- The AWS IAM role assumed via OIDC when pull requests are opened or updated has a trust policy that restricts access to the repository itself and provides read-only permissions -- The AWS IAM role assumed via OIDC when pull requests are merged into the `main` branch has a trust policy limiting access to the repository's `main` branch and granting write permissions +- The credentials assumed via OIDC (e.g. STS Tokens for AWS IAM Roles) when pull requests are opened or updated are intended to have trust policies that restrict access to the repository itself and provides read-only permissions. +- The credentials assumed via OIDC when pull requests are merged into the deploy branch (e.g. `main`) are intended to have trust policies that limiting access to the repository's deploy branch and granting write permissions. - Branch protection rules can be configured to require reviews and status checks - GitHub App or machine user authentication options available -- The AWS IAM role assumed via OIDC when merge requests are opened or updated has a trust policy that restricts access to the group itself and provides read-only permissions -- The AWS IAM role assumed via OIDC when merge requests are merged into the `main` branch has a trust policy limiting access to the group's `main` branch and granting write permissions +- The credentials assumed via OIDC (e.g. STS Tokens for AWS IAM Roles) when merge requests are opened or updated are intended to have trust policies that restrict access to the group itself and provides read-only permissions. +- The credentials assumed via OIDC when merge requests are merged into the deploy branch (e.g. `main`) are intended to have trust policies that limiting access to the group's deploy branch and granting write permissions. - Protected branches can be configured to require approvals and pipeline success - Machine user authentication required with group-level access configuration -## Infrastructure access control - -An optional `infrastructure-live-access-control` repository/group can manage access control for infrastructure provisioned in AWS accounts. Using this is a best practice for centralized and auditable access management. - -- Access to the `main` branch should be restricted to a small, trusted group managing infrastructure access -- The same least privilege principles apply: roles assumed for pull/merge requests have read-only permissions, while roles for merged changes have write permissions - -Unlike the infrastructure-live-root repository, this repository focuses on managing access control rather than defining infrastructure. You might grant write access to a broader group for managing access while maintaining tight control over the main branch. Encourage collaboration between platform teams and application engineers to review and refine access control continuously. - - - ## Token Strategy ### GitHub App Installation Strategy (Recommended) -No tokens are required when using the GitHub App. + +No long-lived tokens are required when using the GitHub App. ### Machine Users Installation Strategy -Requires the following tokens be created: + +Requires that the following tokens are created: + - `PIPELINES_READ_TOKEN`: Classic PAT with read access to required repositories - `INFRA_ROOT_WRITE_TOKEN`: Fine-grained PAT with read/write access to infrastructure repositories - `ORG_REPO_ADMIN_TOKEN`: Fine-grained PAT with admin access for repository management @@ -61,7 +51,9 @@ See [Setup via Machine Users](/2.0/docs/pipelines/installation/viamachineusers.m -Requires the following tokens be created: + +Requires that the following tokens are created: + - `PIPELINES_GITLAB_TOKEN`: A GitLab access token with `api` scope - `PIPELINES_GITLAB_READ_TOKEN`: A GitLab access token with `read_repository` scope @@ -72,136 +64,14 @@ Pipelines will also require access to Gruntwork's GitHub repositories, however t -## AWS credentials - -Pipelines requires IAM roles configured with trust policies to use OpenID Connect (OIDC) with your CI/CD platform. This eliminates the need for long-lived AWS credentials stored as secrets. - -### OpenID Connect Configuration -Pipelines provisions an OpenID Connect identity provider in AWS IAM for each account, setting GitHub/GitLab as the provider and restricting the audience to AWS STS and your GitHub/GitLab organization. The Pipelines IAM role's trust policy ensures: -- Only a single repository in your GitHub/GitLab organization can assume the role for plans. -- Only a single branch can assume the role for applies/destroys. - -For more details, see the [official AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html). Below is an example of a trust policy used by Pipelines. - - - - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "", - "Effect": "Allow", - "Principal": { - "Federated": "arn:aws:iam::0123456789012:oidc-provider/token.actions.githubusercontent.com" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringLike": { - "token.actions.githubusercontent.com:sub": "repo:acme/infrastructure-live-root:ref:*" - } - } - } - ] -} -``` - - - - - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "", - "Effect": "Allow", - "Principal": { - "Federated": "arn:aws:iam::0123456789012:oidc-provider/gitlab.com" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringLike": { - "gitlab.com:sub": "project_path:acme/projectprefix*:*" - } - } - } - ] -} - - -``` - - - - - -Refer to [Configuring OpenId Connect in Amazon Web Services](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) for additional details. - -### Roles provisioned by DevOps Foundations - -Pipelines automatically provisions specific roles in AWS accounts to support required infrastructure operations. These roles follow the naming pattern `-pipelines-`. - -For example: -- The `root-pipelines-plan` role is used by Pipelines to plan changes in the `infrastructure-live-root` repository. - -These roles are designed to operate in a single repository and include a trust policy that only permits GitHub Actions workflows triggered by that repository to assume the role. Each role is provisioned in pairs: -- `plan` roles, with read-only permissions, are used to execute Terragrunt plans for open pull requests. -- `apply` roles, with read/write permissions, are used to apply or destroy infrastructure changes for merged pull requests or direct pushes to the deploy branch (commonly `main`). - -This separation ensures that controls like [branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) and [CODEOWNERS files](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) can effectively govern infrastructure changes. - -#### `root-pipelines-plan` - -A read-only plan role for the `infrastructure-live-root` repository. -- This role is one of the first created when setting up DevOps Foundations. It is provisioned manually by the customer during the platform setup process. -- It exists in all accounts and handles tasks necessary for setting up AWS accounts. -- These roles are highly permissive among read-only roles as they manage foundational AWS account setups. - -#### `root-pipelines-apply` - -A read/write apply role for the `infrastructure-live-root` repository. -- Like the plan role, this is one of the initial roles created during setup. -- It is broadly permissive to support foundational account setups and bootstrapping. - -#### `access-control-pipelines-plan` - -A read-only plan role for the `infrastructure-live-access-control` repository. -- These roles are provisioned for new accounts but are not included in core accounts such as `management`, `logs`, `security`, or `shared`. -- They manage IAM roles and policies for vended accounts, facilitating infrastructure access control. - -#### `access-control-pipelines-apply` - -A read/write apply role for the `infrastructure-live-access-control` repository. -- Similar to the plan role, these roles are provisioned for vended accounts but excluded from core accounts. -- They have permissions to manage IAM roles and policies for the accounts where they are provisioned. - -#### `delegated-pipelines-plan` - -A read-only plan role for delegated repositories, used by Pipelines Enterprise customers. - -- These roles are pre-configured to have minimal permissions, primarily for managing OpenTofu/Terraform state. -- A pull request will be opened in `infrastructure-live-access-control` during provisioning include documentation for adding additional permissions if necessary. -- Users should ensure that only the necessary _read-only_ permissions are granted for the specific delegated repository. - - -:::note - -These roles have almost no permissions by default. They are pre-configured by default to only have access to OpenTofu/Terraform state, and the pull requests that are opened to provision them include documentation on how to add additional permissions as appropriate. +## Cloud Authentication -It is up to the user provisioning these roles to ensure that this role has only the necessary _read-only_ permissions required to manage infrastructure changes relevant to the delegated repository. +Pipelines supports multiple different authentication methods for different cloud providers, and will use the appropriate method based on the context of the pull request or push to the deploy branch. -::: +To learn more about how Pipelines authenticates to the cloud, read the [Cloud Authentication](/2.0/docs/pipelines/concepts/cloud-auth/index.md) documentation. -#### `delegated-pipelines-apply` +## Trust boundaries -A read/write apply role for delegated repositories. -- Similar to the plan role, these roles are pre-configured with minimal permissions and are intended for managing OpenTofu/Terraform state. -- A pull request will be opened in `infrastructure-live-access-control` during provisioning include documentation for adding additional permissions if necessary. -- Users must ensure that the role has only the necessary _read/write_ permissions required for the delegated repository. +A critical aspect of Pipelines' architecture is understanding its trust model. Since Pipelines runs within a CI/CD system, it has privileged access to your infrastructure resources (e.g. AWS accounts, VPCs, EC2 instances, etc.). -:::note -The `delegated-pipelines-plan` and `delegated-pipelines-apply` roles are automatically provisioned for new delegated accounts. Enterprise customers will see pull requests created in the `infrastructure-live-access-control` repository to vend these roles with proper configurations. -::: +Anyone with the ability to edit code in the `main` branch of your repositories inherently has the authority to make corresponding changes in your infrastructure resources. For this reason, it is important to follow the [Repository Access](/2.0/docs/pipelines/installation/viamachineusers#repository-access) guidelines to ensure appropriate access control. diff --git a/docs/2.0/docs/pipelines/architecture/usage-data.md b/docs/2.0/docs/pipelines/architecture/usage-data.md index 7bc78674b6..90971ad0ad 100644 --- a/docs/2.0/docs/pipelines/architecture/usage-data.md +++ b/docs/2.0/docs/pipelines/architecture/usage-data.md @@ -1,3 +1,3 @@ # Usage Data -Gruntwork Pipelines collects usage data to gain insights into how customers interact with the product. This data includes information such as the duration of pipeline runs, the number of jobs executed, the customer name, and any application errors encountered during execution. \ No newline at end of file +Gruntwork Pipelines collects usage data to gain insights into how customers interact with the product. This data includes information such as the duration of pipeline runs, the number of jobs executed, the customer name, and any application errors encountered during execution. diff --git a/sidebars/docs.js b/sidebars/docs.js index c17033d4af..0de3c67e1f 100644 --- a/sidebars/docs.js +++ b/sidebars/docs.js @@ -174,9 +174,9 @@ const sidebar = [ }, items: [ { - label: "Components", + label: "Execution flow", type: "doc", - id: "2.0/docs/pipelines/architecture/components", + id: "2.0/docs/pipelines/architecture/execution-flow", }, { label: "Actions", @@ -480,6 +480,11 @@ const sidebar = [ type: "doc", id: "2.0/docs/accountfactory/architecture/network-topology", }, + { + label: "Repository Topology", + type: "doc", + id: "2.0/docs/accountfactory/architecture/repository-topology", + }, ], }, { diff --git a/src/redirects.js b/src/redirects.js index c22eaac168..f2b86849a2 100644 --- a/src/redirects.js +++ b/src/redirects.js @@ -30,11 +30,11 @@ export const redirects = [ from: "/developer-portal/link-github-id" }, { - to: "/2.0/docs/pipelines/architecture/components", + to: "/2.0/docs/pipelines/architecture/execution-flow", from: "/pipelines/architecture/index" }, { - to: "/2.0/docs/pipelines/architecture/components", + to: "/2.0/docs/pipelines/architecture/execution-flow", from: "/pipelines/architecture" }, {