diff --git a/doc/user/content/get-started/install.md b/doc/user/content/get-started/install.md index 6622e519a2c9b..8b28c07556d95 100644 --- a/doc/user/content/get-started/install.md +++ b/doc/user/content/get-started/install.md @@ -12,4 +12,14 @@ aliases: - /self-managed/v25.1/get-started/install/ --- -{{< include-md file="shared-content/self-managed/install-landing-page.md" >}} +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-intro" %}} + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-guides-helm" %}} + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-guides-unified" %}} + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-guides-legacy" %}} diff --git a/doc/user/content/installation/_index.md b/doc/user/content/installation/_index.md deleted file mode 100644 index 5fdae97374c0f..0000000000000 --- a/doc/user/content/installation/_index.md +++ /dev/null @@ -1,195 +0,0 @@ ---- -title: "Install/Upgrade (Self-Managed)" -description: "Installation and upgrade guides for Self-Managed Materialize." -disable_list: true -menu: - main: - identifier: "installation" - weight: 5 -aliases: - - /self-managed/v25.1/installation/ ---- - -{{< include-md file="shared-content/self-managed/install-landing-page.md" >}} - -## Upgrade - -{{< include-md file="shared-content/self-managed/general-rules-for-upgrades.md" >}} - -### Upgrade guides - -The following upgrade guides are available: - -| | Notes | -| ------------- | -------| -| [Upgrade on kind](/installation/install-on-local-kind/upgrade-on-local-kind/) | -| [Upgrade on AWS](/installation/install-on-aws/upgrade-on-aws/) | Uses Materialize provided Terraform | -| [Upgrade on Azure Kubernetes Service (AKS)](/installation/install-on-azure/upgrade-on-azure/) | Uses Materialize provided Terraform | -| [Upgrade on Google Kubernetes Engine (GKE)](/installation/install-on-gcp/upgrade-on-gcp/) | Uses Materialize provided Terraform | - - -### General notes for upgrades - -The following provides some general notes for upgrades. For specific examples, -see the [Upgrade guides](#upgrade-guides) - - -#### Upgrading the Helm Chart and Kubernetes Operator - -{{< important >}} - -When upgrading Materialize, always upgrade the operator first. - -{{}} - -The Materialize Kubernetes operator is deployed via Helm and can be updated through standard Helm upgrade commands. - -```shell -helm upgrade my-materialize-operator materialize/misc/helm-charts/operator -``` - -If you have custom values, make sure to include your values file: - -```shell -helm upgrade my-materialize-operator materialize/misc/helm-charts/operator -f my-values.yaml -``` - -#### Upgrading Materialize Instances - -In order to minimize unexpected downtime and avoid connection drops at critical -periods for your application, changes are not immediately and automatically -rolled out by the Operator. Instead, the upgrade process involves two steps: -- First, staging spec changes to the Materialize custom resource. -- Second, applying the changes via a `requestRollout`. - -When upgrading your Materialize instances, you'll first want to update the `environmentdImageRef` field in the Materialize custom resource spec. - -##### Updating the `environmentdImageRef` -To find a compatible version with your currently deployed Materialize operator, check the `appVersion` in the Helm repository. - -```shell -helm list -n materialize -``` - -Using the returned version, we can construct an image ref. -We always recommend using the official Materialize image repository -`docker.io/materialize/environmentd`. - -``` -environmentdImageRef: docker.io/materialize/environmentd:v26.0.0 -``` - -The following is an example of how to patch the version. -```shell -# For version updates, first update the image reference -kubectl patch materialize \ - -n \ - --type='merge' \ - -p "{\"spec\": {\"environmentdImageRef\": \"materialize/environmentd:v26.0.0\"}}" -``` - -##### Applying the changes via `requestRollout` - -To apply changes and kick off the Materialize instance upgrade, you must update the `requestRollout` field in the Materialize custom resource spec to a new UUID. -Be sure to consult the [Rollout Configurations](#rollout-configuration) to ensure you've selected the correct rollout behavior. -```shell -# Then trigger the rollout with a new UUID -kubectl patch materialize \ - -n \ - --type='merge' \ - -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\"}}" -``` - - -It is possible to combine both operations in a single command if preferred: - -```shell -kubectl patch materialize \ - -n materialize-environment \ - --type='merge' \ - -p "{\"spec\": {\"environmentdImageRef\": \"materialize/environmentd:v26.0.0\", \"requestRollout\": \"$(uuidgen)\"}}" -``` - -##### Using YAML Definition - -Alternatively, you can update your Materialize custom resource definition directly: - -```yaml -apiVersion: materialize.cloud/v1alpha1 -kind: Materialize -metadata: - name: 12345678-1234-1234-1234-123456789012 - namespace: materialize-environment -spec: - environmentdImageRef: materialize/environmentd:v26.0.0 # Update version as needed - requestRollout: 22222222-2222-2222-2222-222222222222 # Generate new UUID - forceRollout: 33333333-3333-3333-3333-333333333333 # Optional: for forced rollouts - inPlaceRollout: false # In Place rollout is deprecated and ignored. Please use rolloutStrategy - rolloutStrategy: WaitUntilReady # The mechanism to use when rolling out the new version. Can be WaitUntilReady or ImmediatelyPromoteCausingDowntime - backendSecretName: materialize-backend -``` - -Apply the updated definition: - -```shell -kubectl apply -f materialize.yaml -``` - -#### Rollout Configuration - -##### Forced Rollouts - -If you need to force a rollout even when there are no changes to the instance: - -```shell -kubectl patch materialize \ - -n materialize-environment \ - --type='merge' \ - -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\", \"forceRollout\": \"$(uuidgen)\"}}" -``` - -##### Rollout Strategies - -The behavior of the new version rollout follows your `rolloutStrategy` setting: - -`WaitUntilReady` (default): - -New instances are created and all dataflows are determined to be ready before cutover and terminating the old version, temporarily requiring twice the resources during the transition. - -`ImmediatelyPromoteCausingDowntime`: - -Tears down the prior version before creating and promoting the new version. This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources. - -##### In Place Rollout - -The `inPlaceRollout` setting has been deprecated and will be ignored. - -### Verifying the Upgrade - -After initiating the rollout, you can monitor the status field of the Materialize custom resource to check on the upgrade. - -```shell -# Watch the status of your Materialize environment -kubectl get materialize -n materialize-environment -w - -# Check the logs of the operator -kubectl logs -l app.kubernetes.io/name=materialize-operator -n materialize -``` -### Version Specific Upgrade Notes - -#### Upgrading to `v26.1` and later versions -{{< include-md file="shared-content/self-managed/upgrade-notes/v26.1.md" >}} - -#### Upgrading to `v26.0` - -{{< include-md file="shared-content/self-managed/upgrade-notes/v26.0.md" >}} - -#### Upgrading between minor versions less than `v26` - - Prior to `v26`, you must upgrade at most one minor version at a time. For - example, upgrading from `v25.1.5` to `v25.2.16` is permitted. - -## See also - -- [Materialize Operator Configuration](/installation/configuration/) -- [Troubleshooting](/installation/troubleshooting/) -- [FAQ](/installation/faq/) diff --git a/doc/user/content/installation/appendix-aws-production-guide.md b/doc/user/content/installation/appendix-aws-production-guide.md deleted file mode 100644 index 2049efd0a2543..0000000000000 --- a/doc/user/content/installation/appendix-aws-production-guide.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: "Production guide" -description: "Production guide for Self-Managed Materialize AWS deployment." -draft: true - ---- - -The sample Materialize on AWS Terraform module are provided as a starting point -for evaluation. The Terraform module: - -- Does not use encryption for object storage. -- Does not use TLS for traffic. -- Does not implement RBAC (Role-Based Access Control). -- Does not include explicit OS version upgrade logic. -- Does not include Kubernetes upgrade logic. -- Relies on port forwarding for access. - - - diff --git a/doc/user/content/installation/appendix-materialize-crd-field-descriptions.md b/doc/user/content/installation/appendix-materialize-crd-field-descriptions.md deleted file mode 100644 index d674786dfbdad..0000000000000 --- a/doc/user/content/installation/appendix-materialize-crd-field-descriptions.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: "Appendix: Materialize CRD Field Descriptions" -description: "Reference page on Materialize CRD Fields" -menu: - main: - parent: "installation" - identifier: "appendix-materialize-crd-field-descriptions" - weight: 97 ---- - -{{% self-managed/materialize-crd-descriptions %}} diff --git a/doc/user/content/installation/appendix-terraforms.md b/doc/user/content/installation/appendix-terraforms.md deleted file mode 100644 index cfb6ae78dd999..0000000000000 --- a/doc/user/content/installation/appendix-terraforms.md +++ /dev/null @@ -1,53 +0,0 @@ - ---- -title: "Appendix: Terraforms" -description: "List of template Terraform modules that are available as a -starting point." -menu: - main: - parent: "installation" - identifier: "appendix-terraforms" - weight: 95 ---- - -To help you get started, Materialize provides some template Terraforms. - -{{< important >}} -These modules are intended for evaluation/demonstration purposes and for serving -as a template when building your own production deployment. The modules should -not be directly relied upon for production deployments: **future releases of the -modules will contain breaking changes.** Instead, to use as a starting point for -your own production deployment, either: - -- Fork the repo and pin to a specific version; or - -- Use the code as a reference when developing your own deployment. - -{{}} - -{{< yaml-table data="self_managed/terraform_list" >}} - -## Releases - -### Materialize on AWS Terraform module - -{{< yaml-table data="self_managed/aws_terraform_versions" >}} - -{{% self-managed/aws-terraform-upgrade-notes %}} - -See also [Upgrade Notes]( -https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#upgrade-notes) -for release-specific upgrade notes. - - -### Materialize on Azure Terraform module - -{{< yaml-table data="self_managed/azure_terraform_versions" >}} - -{{% self-managed/azure-terraform-upgrade-notes %}} - -### Materialize on GCP Terraform module - -{{< yaml-table data="self_managed/gcp_terraform_versions" >}} - -{{% self-managed/gcp-terraform-upgrade-notes %}} diff --git a/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md b/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md deleted file mode 100644 index f8f8252b35695..0000000000000 --- a/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: "Appendix: AWS deployment guidelines" -description: "AWS environment setup/deployment guidelines" -aliases: - - /self-hosted/operational-guidelines/ -menu: - main: - parent: "install-on-aws" - identifier: "aws-deployment-guidelines" - weight: 40 ---- - -## Recommended instance types - -As a general guideline, we recommend: - -- Processor Type: ARM-based CPU - -- Sizing: - - - If spill-to-disk is not enabled: 1:8 ratio of vCPU to GiB memory - - - If spill-to-disk is enabled (*Recommended*): 1:16 ratio of vCPU to GiB local - instance storage - -{{% self-managed/aws-recommended-instances %}} - -## Locally-attached NVMe storage - -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. - -### Swap support - -Starting in v0.6.1 of Materialize on AWS Terraform, -disk support (using swap on NVMe instance storage) may be enabled for -Materialize. With this change, the Terraform: - -- Creates a node group for Materialize. -- Configures NVMe instance store volumes as swap using a daemonset. -- Enables swap at the Kubelet. - -For swap support, the following configuration option is available: - -- [`swap_enabled`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_swap_enabled) - -See [Upgrade Notes](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#v061). - - -## CPU affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - -## TLS - -When running with TLS in production, run with certificates from an official -Certificate Authority (CA) rather than self-signed certificates. - -## See also - -- [Configuration](/installation/configuration/) -- [Installation](/installation/) -- [Troubleshooting](/installation/troubleshooting/) diff --git a/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md b/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md deleted file mode 100644 index b0be5c7180155..0000000000000 --- a/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: "Appendix: Azure deployment guidelines" -description: "Azure environment setup/deployment guidelines" -menu: - main: - parent: "install-on-azure" - identifier: "azure-deployment-guidelines" - weight: 40 ---- - -## Recommended instance types - -As a general guideline, we recommend: - -- Processor Type: ARM-based CPU - -- Sizing: 2:1 disk-to-RAM ratio with spill-to-disk enabled. - -### Recommended Azure VM Types with Local NVMe Disks - -When operating on Azure in production, we recommend [Epdsv6 -sizes](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/memory-optimized/epdsv6-series?tabs=sizebasic#sizes-in-series) -Azure VM Types with Local NVMe Disk: - -| VM Size | vCPUs | Memory | Ephemeral Disk | Disk-to-RAM Ratio | -| ------------------ | ----- | ------- | -------------- | ----------------- | -| Standard_E2pds_v6 | 2 | 16 GiB | 75 GiB | ~4.7:1 | -| Standard_E4pds_v6 | 4 | 32 GiB | 150 GiB | ~4.7:1 | -| Standard_E8pds_v6 | 8 | 64 GiB | 300 GiB | ~4.7:1 | -| Standard_E16pds_v6 | 16 | 128 GiB | 600 GiB | ~4.7:1 | -| Standard_E32pds_v6 | 32 | 256 GiB | 1,200 GiB | ~4.7:1 | - -{{< important >}} - -These VM types provide **ephemeral** local NVMe SSD disks. Data is lost -when the VM is stopped or deleted. - -{{}} - -See also [Locally attached NVMe storage](#locally-attached-nvme-storage). - -## Locally-attached NVMe storage - -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. - -### Swap support - -Starting in v0.6.1 of Materialize on Azure Terraform, -disk support (using swap on NVMe instance storage) may be enabled for -Materialize. With this change, the Terraform: - - - Creates a node group for Materialize. - - Configures NVMe instance store volumes as swap using a daemonset. - - Enables swap at the Kubelet. - -For swap support, the following configuration option is available: - -- [`swap_enabled`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_swap_enabled) - -See [Upgrade Notes](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#v061). - -## Recommended Azure Blob Storage - -Materialize writes **block** blobs on Azure. As a general guideline, we -recommend **Premium block blob** storage accounts. - -## CPU affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - -## TLS - -When running with TLS in production, run with certificates from an official -Certificate Authority (CA) rather than self-signed certificates. - -## See also - -- [Configuration](/installation/configuration/) -- [Installation](/installation/) -- [Troubleshooting](/installation/troubleshooting/) - -[`enable_disk_support`]: https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#disk-support-for-materialize-on-azure - -[`disk_support_config`]: - https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_disk_support_config - -[`disk_setup_image`]: - https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_disk_setup_image diff --git a/doc/user/content/installation/operational-guidelines.md b/doc/user/content/installation/operational-guidelines.md deleted file mode 100644 index de246686733c0..0000000000000 --- a/doc/user/content/installation/operational-guidelines.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: "Operational guidelines" -description: "" -aliases: - - /self-hosted/operational-guidelines/ -menu: - main: - parent: "installation" - weight: 80 - identifier: "sm-operational-guidelines" ---- - -## Recommended instance types - -- ARM-based CPU -- 1:8 ratio of vCPU to GiB memory (if spill-to-disk is not enabled) -- 1:16 ratio of vCPU to GiB local instance storage (if spill-to-disk is enabled) - -See also the specific cloud provider guidance: - -- [AWS Deployment - guidelines](/installation/install-on-aws/appendix-deployment-guidelines/#recommended-instance-types) - -- [GCP Deployment - guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/#recommended-instance-types) - -- [Azure Deployment - guidelines](/installation/install-on-azure/appendix-deployment-guidelines/#recommended-instance-types) - -## CPU affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - -## TLS - -When running with TLS in production, run with certificates from an official -Certificate Authority (CA) rather than self-signed certificates. - -## Locally-attached NVMe storage - -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. - -Refer to the specific cloud provider guidelines: - -- [AWS Deployment - guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) - -- [GCP Deployment - guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/) - -- [Azure Deployment - guidelines](/installation/install-on-azure/appendix-deployment-guidelines/) - -## See also - -- [Configuration](/installation/configuration/) -- [Installation](/installation/) -- [Troubleshooting](/installation/troubleshooting/) diff --git a/doc/user/content/releases/_index.md b/doc/user/content/releases/_index.md index f5bceeb00d510..77aa3b140a06b 100644 --- a/doc/user/content/releases/_index.md +++ b/doc/user/content/releases/_index.md @@ -77,7 +77,7 @@ file="shared-content/self-managed/gcp-terraform-v0.6.1-upgrade-notes.md" >}}. - To upgrade if **not** using a Materialize-provided Terraforms, you must prepare your nodes by adding the required labels. For detailed instructions, see [Prepare for swap and upgrade to -v26.0](/installation/upgrade-to-swap/). +v26.0](/self-managed-deployments/appendix/upgrade-to-swap/). ### SASL/SCRAM-SHA-256 support @@ -103,7 +103,7 @@ Starting in v26.0.0, Self-Managed Materialize requires a license key. For new deployments, you configure your license key in the Kubernetes Secret resource during the installation process. For details, see the [installation -guides](/installation/). For existing deployments, you can configure your +guides](/self-managed-deployments/installation/). For existing deployments, you can configure your license key via: ```bash @@ -146,7 +146,7 @@ use the new setting `rolloutStrategy` to specify either: - `WaitUntilReady` (*Default*) - `ImmediatelyPromoteCausingDowntime` -For more information, see [`rolloutStrategy`](/installation/#rollout-strategies). +For more information, see [`rolloutStrategy`](/self-managed-deployments/upgrading/#rollout-strategies). ### Terraform helpers @@ -195,7 +195,8 @@ See also Upgrade Notes for release specific notes. {{< include-md file="shared-content/self-managed/upgrade-notes/v26.0.md" >}} -See also [General notes for upgrades](/installation/#general-notes-for-upgrades). +See also [Version specific upgrade +notes](/self-managed-deployments/upgrading/#version-specific-upgrade-notes). ## See also diff --git a/doc/user/content/security/self-managed/authentication.md b/doc/user/content/security/self-managed/authentication.md index 5c06c58c16798..9884f134d65fd 100644 --- a/doc/user/content/security/self-managed/authentication.md +++ b/doc/user/content/security/self-managed/authentication.md @@ -162,7 +162,7 @@ spec: forceRollout: # Rollout without requiring a version change ``` -For more information on rollout configuration, view our [installation overview](/installation/#rollout-configuration). +For more information on rollout configuration, view our [upgrade overview](/self-managed-deployments/upgrading/#rollout-configuration). {{< include-md file="shared-content/auth-kind-warning.md" >}} diff --git a/doc/user/content/self-managed-deployments/_index.md b/doc/user/content/self-managed-deployments/_index.md new file mode 100644 index 0000000000000..ac4ec94c52806 --- /dev/null +++ b/doc/user/content/self-managed-deployments/_index.md @@ -0,0 +1,353 @@ +--- +title: "Self-Managed Deployments" +description: "Learn about the key components and architecture of self-managed Materialize deployments." +disable_list: true +aliases: + - /self-hosted/concepts/ + - /self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/ +menu: + main: + identifier: "sm-deployments" + weight: 5 +--- + +## Overview + +Self-Managed Materialize deployments on Kubernetes consist of several layers of +components that work together to provide a fully functional database +environment. Understanding these components and how they interact is essential +for deploying, managing, and troubleshooting your Self-Managed Materialize. + +This page provides an overview of the core architectural components in a Self-Managed deployment, from the infrastructure level (Helm chart) down to the application level (clusters and replicas). + +## Architecture layers + +A Self-Managed Materialize deployment is organized into the following layers: + +Layer | Component | Description +------|-----------|------------ +**Infrastructure** | [Helm Chart](#helm-chart) | Package manager component that bootstraps the Kubernetes deployment +**Orchestration** | [Materialize Operator](#materialize-operator) | Kubernetes operator that manages Materialize instances +**Database** | [Materialize Instance](#materialize-instance) | The Materialize database instance itself +**Compute** | [Clusters and Replicas](#clusters-and-replicas) | Isolated compute resources for workloads + +## Helm chart + +The Helm chart is the entry point for deploying Materialize in a self-managed Kubernetes environment. It serves as a package manager component that defines and deploys the Materialize Operator. + +### Working with the Helm chart + +You interact with the Helm chart through standard Helm commands. For example: + +- To add the Materialize Helm chart repository: + + ```bash + helm repo add materialize https://materializeinc.github.io/materialize + ``` + +- To update the repository index: + + ```bash + helm repo update materialize + ``` + +- To install the Materialize Helm chart and deploy the Materialize Operator and + other resources: + + ```bash + helm install materialize materialize/materialize-operator + ``` + +- To upgrade the the Materialize Helm chart (and the Materialize Operator and + other resources): + + ```bash + helm upgrade materialize materialize/materialize-operator + ``` + +- To uninstall the Helm chart (and the Materialize Operator and other + resources): + + ```bash + helm uninstall materialize + ``` + +### What gets installed + +```bash +helm install materialize materialize/materialize-operator +``` + +When you install the the Materialize Helm Chart, it: + +- Deploys the **Materialize Operator** as a Kubernetes deployment. +- Creates necessary cluster-wide resources (CRDs, RBAC roles, service accounts). +- Configures operator settings and permissions. + +Once installed, the **Materialize Operator** handles the deployment and +management of Materialize instances. + +## Materialize Operator + +The Materialize Operator (implemented as `orchestratord`) is a Kubernetes operator that automates the deployment and lifecycle management of Materialize instances. It implements the [Kubernetes operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) to extend Kubernetes with domain-specific knowledge about Materialize. + +### Managed resources + +The operator watches for Materialize custom resources and creates/manages all the Kubernetes resources required to run a Materialize instance, including: + +- **Namespaces**: Isolated Kubernetes namespaces for each instance +- **Services**: Network services for connecting to Materialize +- **Network Policies**: Network isolation and security rules +- **Certificates**: TLS certificates for secure connections +- **ConfigMaps and Secrets**: Configuration and sensitive data +- **Deployments**: These support the `balancerd` and `console` pod used as the ingress layer for Materialize. +- **StatefulSets**: `environmentd` and `clusterd` which are the database control plane and compute resources respectively. + +### Configuration + +For configuration options for the Materialize Operator, see +the [Materialize Operator Configuration +page](/self-managed-deployments/operator-configuration/). + +## Materialize Instance + +A Materialize instance is the actual database that you connect to and interact with. Each instance is an isolated Materialize deployment with its own data, configuration, and compute resources. + +### Components + +When you create a Materialize instance, the operator deploys three core components as Kubernetes resources: + +- **`environmentd`**: The main database control plane, deployed as a + StatefulSet. + + **`environmentd`** runs as a Kubernetes pod and is the primary component of a + Materialize instance. It houses the control plane and contains: + + - **Adapter**: The SQL interface that handles client connections, query parsing, and planning + - **Storage Controller**: Maintains durable metadata for storage + - **Compute Controller**: Orchestrates compute resources and manages system state + + On startup, `environmentd` will create several built-in clusters. + + When you connect to Materialize with a SQL client (e.g., `psql`), you're connecting to `environmentd`. + +- **balancerd**: A pgwire and http proxy used to connect to environmentd, + deployed as a Deployment. +- **console**: Web-based administration interface, deployed as a Deployment. + +### Instance responsibilities + +A Materialize instance manages: + +- **SQL objects**: Sources, views, materialized views, indexes, sinks +- **Schemas and databases**: Logical organization of objects +- **User connections**: SQL client connections and authentication +- **Catalog metadata**: System information about all objects and configuration +- **Compute orchestration**: Coordination of work across clusters and replicas + + +### Deploying with the operator + +To deploy Materialize instances with the operator, create and apply Materialize +custom resources definitions(CRDs). For a full list of fields available for the +Materialize CR, see [Materialize CRD Field +Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/). + +```yaml +apiVersion: materialize.cloud/v1alpha1 +kind: Materialize +metadata: + name: 12345678-1234-1234-1234-123456789012 + namespace: materialize-environment +spec: + environmentdImageRef: materialize/environmentd:{{< self-managed/versions/get-latest-version >}} +# ... additional fields omitted for brevity +``` + +When you first apply the Materialize custom resource, the operator automatically +creates all required Kubernetes resources. + +### Modifying the custom resource + +To modify a custom resource, update the CRD with your changes, including the +`requestRollout` field with a new UUID value. When you apply the CRD, the +operator will roll out the changes. + +{{< note >}} If you do not specify a new `requestRollout` UUID, the operator +watches for updates but does not roll out the changes. +{{< /note >}} + +For a full list of fields available for the Materialize CR, see [Materialize CRD +Field +Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/). + +See also: + +- [Upgrade Overview](/self-managed-deployments/upgrading/) + +### Connecting to an instance + +Once deployed, you interact with a Materialize instance through the Materialize +Console or standard PostgreSQL-compatible tools and drivers: + +```bash +# Connect with psql +psql "postgres://materialize@:6875/materialize" +``` + +Once connected, you can issue SQL commands to create sources, define views, run queries, and manage the database: + +```sql +-- Create a source +CREATE SOURCE my_source FROM KAFKA ...; + +-- Create a materialized view +CREATE MATERIALIZED VIEW my_view AS + SELECT ... FROM my_source ...; + +-- Query the view +SELECT * FROM my_view; +``` + +## Clusters and Replicas + +Clusters are isolated pools of compute resources that execute workloads in Materialize. They provide resource isolation and fault tolerance for your data processing pipelines. + +For a comprehensive overview of clusters in Materialize, see the [Clusters concept page](/concepts/clusters/). + +### Cluster architecture + +- **Clusters**: Logical groupings of compute resources dedicated to specific workloads (sources, sinks, indexes, materialized views, queries) +- **Replicas**: Physical instantiations of a cluster's compute resources, deployed as Kubernetes StatefulSets + +Each replica contains identical compute resources and processes the same data independently, providing fault tolerance and high availability. + +### Kubernetes resources + +When you create a cluster with one or more replicas in Materialize, the instance coordinates with the operator to create: + +- One or more **StatefulSet** resources (one per replica) +- **Pods** within each StatefulSet that execute the actual compute workload +- **Persistent volumes** (if configured) for scratch disk space + +For example: + +```sql +-- Create a cluster with 2 replicas +CREATE CLUSTER my_cluster SIZE = '100cc', REPLICATION FACTOR = 2; +``` + +This creates two separate StatefulSets in Kubernetes, each running compute processes. + +### Managing clusters + +You interact with clusters primarily through SQL: + +```sql +-- Create a cluster +CREATE CLUSTER ingest_cluster SIZE = '50cc', REPLICATION FACTOR = 1; + +-- Use the previous cluster for a source +CREATE SOURCE my_source + IN CLUSTER ingest_cluster + FROM KAFKA ...; + +-- Create a cluster for materialized views +CREATE CLUSTER compute_cluster SIZE = '100cc', REPLICATION FACTOR = 2; + +-- Use the previous cluster for a materialized view +CREATE MATERIALIZED VIEW my_view + IN CLUSTER compute_cluster AS + SELECT ... FROM my_source ...; + +-- Resize a cluster +ALTER CLUSTER compute_cluster SET (SIZE = '200cc'); + +``` + +Materialize handles the underlying Kubernetes resource creation and management automatically. + +## Workflow + +The following outlines the workflow process, summarizing how the various +components work together: + +1. **Install the Helm chart**: This deploys the Materialize Operator to your + Kubernetes cluster. + +1. **Create a Materialize instance**: Apply a Materialize custom resource. The + operator detects this and creates all necessary Kubernetes resources, + including the `environmentd`, `balancerd`, and `console` pods. + +1. **Connect to the instance**: Use the Materialize Console on port 8080 to + connecto to the `console` service endpoint or SQL client on port 6875 to + connect to the `balancerd` service endpoint. + + If authentication is enabled, you must first connect to the Materialize + Console and set up users. + +1. **Create clusters**: Issue SQL commands to create clusters. Materialize + coordinates with the operator to provision StatefulSets for replicas. + +1. **Run your workloads**: Create sources, materialized views, indexes, and + sinks on your clusters. + +## Terraform Modules + +To help you get started, Materialize provides Terraform modules. + +{{< important >}} +These modules are intended for evaluation/demonstration purposes and for serving +as a template when building your own production deployment. The modules should +not be directly relied upon for production deployments: **future releases of the +modules will contain breaking changes.** Instead, to use as a starting point for +your own production deployment, either: + +- Fork the repo and pin to a specific version; or + +- Use the code as a reference when developing your own deployment. + +{{}} + +{{< tabs >}} +{{< tab "Unified Terraform Modules (New!)" >}} +### Unified Terraform Modules + +Materialize provides a [**unified Terraform +module**](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main?tab=readme-ov-file#materialize-self-managed-terraform-modules), +which provides concrete examples and an opinionated model for deploying Materialize. + +{{< yaml-table data="self_managed/terraform_list" >}} + +{{< /tab >}} +{{< tab "Legacy Terraform Modules" >}} +### Legacy Terraform Modules + +{{< yaml-table data="self_managed/terraform_list_legacy" >}} +{{< /tab >}} +{{< /tabs >}} + +## Relationship to Materialize concepts + +Self-managed deployments implement the same core Materialize concepts as the Cloud offering: + +- [**Clusters**](/concepts/clusters/): Identical behavior, but backed by Kubernetes StatefulSets +- [**Sources**](/concepts/sources/): Same functionality for ingesting data +- [**Views**](/concepts/views/): Same query semantics and incremental maintenance +- [**Indexes**](/concepts/indexes/): Same in-memory query acceleration +- [**Sinks**](/concepts/sinks/): Same data egress capabilities + +The Self-Managed deployment model adds the Kubernetes infrastructure layer (Helm +chart and operator) but does not change how you interact with Materialize at the +SQL level. + +## Related pages + +- [Installation guides](/self-managed-deployments/installation/) +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) +- [Operational guidelines](/self-managed-deployments/deployment-guidelines/) +- [Clusters concept page](/concepts/clusters/) +- [Materialize architecture overview](/concepts/) diff --git a/doc/user/content/self-managed-deployments/appendix/_index.md b/doc/user/content/self-managed-deployments/appendix/_index.md new file mode 100644 index 0000000000000..65419c0b88596 --- /dev/null +++ b/doc/user/content/self-managed-deployments/appendix/_index.md @@ -0,0 +1,15 @@ +--- +title: "Appendix" +description: "" +menu: + main: + parent: "sm-deployments" + identifier: "sm-deployments-appendix" + weight: 99 +disable_list: true +--- + +## Table of contents + +- [Appendix: Cluster sizes](./appendix-cluster-sizes/) +- [Appendix: Prepare for swap and upgrade to v26.0](./upgrade-to-swap/) diff --git a/doc/user/content/installation/appendix-cluster-sizes.md b/doc/user/content/self-managed-deployments/appendix/appendix-cluster-sizes.md similarity index 93% rename from doc/user/content/installation/appendix-cluster-sizes.md rename to doc/user/content/self-managed-deployments/appendix/appendix-cluster-sizes.md index 848609a04cb93..ddf2804e33a10 100644 --- a/doc/user/content/installation/appendix-cluster-sizes.md +++ b/doc/user/content/self-managed-deployments/appendix/appendix-cluster-sizes.md @@ -1,13 +1,14 @@ --- -title: "Appendix: Cluster sizes" +title: "Cluster sizes" description: "Reference page on self-managed cluster sizes" menu: main: - parent: "installation" + parent: "sm-deployments-appendix" identifier: "appendix-cluster-sizes" weight: 96 aliases: - /self-managed/v25.1/sql/appendix-cluster-sizes/ + - /installation/appendix-cluster-sizes/ --- ## Default Cluster Sizes diff --git a/doc/user/content/self-managed-deployments/appendix/legacy/_index.md b/doc/user/content/self-managed-deployments/appendix/legacy/_index.md new file mode 100644 index 0000000000000..3c535a60cb270 --- /dev/null +++ b/doc/user/content/self-managed-deployments/appendix/legacy/_index.md @@ -0,0 +1,11 @@ +--- +title: "Legacy Terraform: Releases and configurations" +description: "" +menu: + main: + parent: "sm-deployments-appendix" + identifier: "sm-deployments-appendix-legacy" + weight: 96 +--- + +## Table of contents diff --git a/doc/user/content/installation/install-on-aws/appendix-aws-configuration.md b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-aws.md similarity index 69% rename from doc/user/content/installation/install-on-aws/appendix-aws-configuration.md rename to doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-aws.md index e9e520e2c9cb8..90007290e71d8 100644 --- a/doc/user/content/installation/install-on-aws/appendix-aws-configuration.md +++ b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-aws.md @@ -1,11 +1,11 @@ --- -title: "Appendix: Required configuration" +title: "Required configuration (Legacy AWS Terraform)" description: "Required configuration for Materialize on AWS Terraform." menu: main: - parent: "install-on-aws" - identifier: "appendix-aws-provider-config" - weight: 50 + parent: "sm-deployments-appendix-legacy" + identifier: "appendix-legacy-aws-provider-config" + weight: 20 aliases: - /installation/install-on-aws/appendix-aws-provider-configuration --- @@ -60,3 +60,19 @@ provider "helm" { } } ``` + +## Swap support + +Starting in v0.6.1 of Materialize on AWS Terraform, +disk support (using swap on NVMe instance storage) may be enabled for +Materialize. With this change, the Terraform: + +- Creates a node group for Materialize. +- Configures NVMe instance store volumes as swap using a daemonset. +- Enables swap at the Kubelet. + +For swap support, the following configuration option is available: + +- [`swap_enabled`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_swap_enabled) + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#v061). diff --git a/doc/user/content/installation/install-on-azure/appendix-azure-configuration.md b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-azure.md similarity index 79% rename from doc/user/content/installation/install-on-azure/appendix-azure-configuration.md rename to doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-azure.md index c2613724d1f97..f4157b0b0da3d 100644 --- a/doc/user/content/installation/install-on-azure/appendix-azure-configuration.md +++ b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-azure.md @@ -1,11 +1,11 @@ --- -title: "Appendix: Required configuration" +title: "Required configuration (Legacy Azure Terraform)" description: "Required configuration for Materialize on Azure Terraform." menu: main: - parent: "install-on-azure" - identifier: "appendix-azure-config" - weight: 50 + parent: "sm-deployments-appendix-legacy" + identifier: "appendix-legacy-azure-config" + weight: 30 aliases: - /installation/install-on-azure/appendix-azure-provider-configuration --- @@ -83,5 +83,21 @@ provider "helm" { } ``` +## Swap support + +Starting in v0.6.1 of Materialize on Azure Terraform, +disk support (using swap on NVMe instance storage) may be enabled for +Materialize. With this change, the Terraform: + +- Creates a node group for Materialize. +- Configures NVMe instance store volumes as swap using a daemonset. +- Enables swap at the Kubelet. + +For swap support, the following configuration option is available: + +- [`swap_enabled`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_swap_enabled) + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#v061). + [^1]: If using the `examples/simple/main.tf`, the example configuration handles them for you. diff --git a/doc/user/content/installation/install-on-gcp/appendix-gcp-configuration.md b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-gcp.md similarity index 52% rename from doc/user/content/installation/install-on-gcp/appendix-gcp-configuration.md rename to doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-gcp.md index 1522237789e44..6cd67c573681d 100644 --- a/doc/user/content/installation/install-on-gcp/appendix-gcp-configuration.md +++ b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-gcp.md @@ -1,13 +1,13 @@ --- -title: "Appendix: Required configuration" -description: "Required configuration for Materialize on GCP Terraform." +title: "Required configuration (Legacy GCP Terraform)" +description: "Required configuration for Materialize on GCP Terraform (legacy)." menu: main: - parent: "install-on-gcp" - identifier: "appendix-gcp-config" - weight: 50 + parent: "sm-deployments-appendix-legacy" + identifier: "appendix-legacy-gcp-configuration" + weight: 40 aliases: - - /installation/install-on-gcp/appendix-gcp-provider-configuration + - /installation/install-on-gcp/appendix-gcp-configuration/ --- ## Required variables @@ -58,3 +58,29 @@ you need to declare: ```hcl data "google_client_config" "current" {} ``` + +## Swap support + +Starting in v0.6.1 of Materialize on Google Cloud Provider (GCP) Terraform, +disk support (using swap on NVMe instance storage) may be enabled for +Materialize. With this change, the Terraform: + +- Creates a node group for Materialize. +- Configures NVMe instance store volumes as swap using a daemonset. +- Enables swap at the Kubelet. + +For swap support, the following configuration options are available: + +- [`swap_enabled`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_swap_enabled) + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#v061). + +## Storage bucket versioning + +Starting in v0.3.1 of Materialize on GCP Terraform, storage bucket versioning is +disabled (i.e., +[`storage_bucket_versioning`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_versioning) +is set to `false` by default) to facilitate cleanup of resources during testing. +When running in production, versioning should be turned on with a sufficient TTL +([`storage_bucket_version_ttl`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_version_ttl)) +to meet any data-recovery requirements. diff --git a/doc/user/content/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases.md b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases.md new file mode 100644 index 0000000000000..08e65a9299eaa --- /dev/null +++ b/doc/user/content/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases.md @@ -0,0 +1,38 @@ +--- +title: "Legacy Terraform Releases" +description: +menu: + main: + parent: "sm-deployments-appendix-legacy" + weight: 10 +aliases: + - /installation/appendix-terraforms/ +--- + + +## Legacy Terraform Modules + +{{< yaml-table data="self_managed/terraform_list_legacy" >}} + +## Materialize on AWS Terraform module (Legacy) {#materialize-on-aws-terraform-module} + +{{< yaml-table data="self_managed/aws_terraform_versions" >}} + +{{% self-managed/aws-terraform-upgrade-notes %}} + +See also [Upgrade Notes]( +https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#upgrade-notes) +for release-specific upgrade notes. + + +## Materialize on Azure Terraform module (Legacy){#materialize-on-azure-terraform-module} + +{{< yaml-table data="self_managed/azure_terraform_versions" >}} + +{{% self-managed/azure-terraform-upgrade-notes %}} + +## Materialize on GCP Terraform module (Legacy) {#materialize-on-gcp-terraform-module} + +{{< yaml-table data="self_managed/gcp_terraform_versions" >}} + +{{% self-managed/gcp-terraform-upgrade-notes %}} diff --git a/doc/user/content/installation/upgrade-to-swap.md b/doc/user/content/self-managed-deployments/appendix/upgrade-to-swap.md similarity index 83% rename from doc/user/content/installation/upgrade-to-swap.md rename to doc/user/content/self-managed-deployments/appendix/upgrade-to-swap.md index e25c788c3b643..0caffaab8e34d 100644 --- a/doc/user/content/installation/upgrade-to-swap.md +++ b/doc/user/content/self-managed-deployments/appendix/upgrade-to-swap.md @@ -1,11 +1,13 @@ --- -title: "Appendix: Prepare for swap and upgrade to v26.0" +title: "Prepare for swap and upgrade to v26.0" description: "Upgrade procedure for v26.0 if not using Materialize Terraform." menu: main: - parent: "installation" + parent: "sm-deployments-appendix" weight: 95 identifier: "helm-upgrade-from-v25.2.12-aws" +aliases: + - /installation/upgrade-to-swap/ --- {{< annotation type="Disambiguation" >}} @@ -27,7 +29,7 @@ file="shared-content/self-managed/gcp-terraform-v0.6.1-upgrade-notes.md" >}}. - {{< include-md file="shared-content/self-managed/azure-terraform-v0.6.1-upgrade-notes.md" >}}. -See also [General notes for upgrades](/installation/#upgrade). +See also [Upgrade Overview](/self-managed-deployments/upgrading/). {{< /annotation >}} diff --git a/doc/user/content/self-managed-deployments/deployment-guidelines/_index.md b/doc/user/content/self-managed-deployments/deployment-guidelines/_index.md new file mode 100644 index 0000000000000..62fba97a2ef90 --- /dev/null +++ b/doc/user/content/self-managed-deployments/deployment-guidelines/_index.md @@ -0,0 +1,25 @@ +--- +title: "Deployment guidelines" +description: "" +disable_list: true +menu: + main: + parent: "sm-deployments" + identifier: "deployment-guidelines" + weight: 60 +aliases: + - /installation/operational-guidelines/ +--- + +{{% self-managed/materialize-components-sentence %}} + +## Available deployment guidelines + +The following provides guidelines for cloud-specific deployments: + +- [AWS Deployment + Guidelines](/self-managed-deployments/deployment-guidelines/aws-deployment-guidelines/) +- [Azure Deployment + Guidelines](/self-managed-deployments/deployment-guidelines/azure-deployment-guidelines/) +- [GCP Deployment + Guidelines](/self-managed-deployments/deployment-guidelines/gcp-deployment-guidelines/) diff --git a/doc/user/content/self-managed-deployments/deployment-guidelines/aws-deployment-guidelines.md b/doc/user/content/self-managed-deployments/deployment-guidelines/aws-deployment-guidelines.md new file mode 100644 index 0000000000000..c8e54a83baed2 --- /dev/null +++ b/doc/user/content/self-managed-deployments/deployment-guidelines/aws-deployment-guidelines.md @@ -0,0 +1,70 @@ +--- +title: "AWS deployment guidelines" +description: "General guidelines when deploying Self-Managed Materialize on AWS." +disable_list: true +menu: + main: + parent: "deployment-guidelines" + identifier: "aws-deployment-guidelines" + weight: 10 +aliases: + - /installation/install-on-aws/appendix-deployment-guidelines/ +--- + +{{% self-managed/materialize-components-sentence %}} + +## Recommended instance types + +As a general guideline, we recommend: + +- ARM-based CPU +- A 1:8 ratio of vCPU to GiB memory. +- A 8:1 ratio of GiB local instance storage to GiB memory when using swap. + +{{% self-managed/aws-recommended-instances %}} + +## Locally-attached NVMe storage + +Configuring swap on nodes to use locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. + +### Swap support + +{{< tabs >}} +{{< tab "New Unified Terraform" >}} + +#### New Unified Terraform + +The unified Materialize [Terraform module](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws/examples/simple) supports configuring swap out of the box. + +{{< /tab >}} +{{< tab "Legacy Terraform" >}} +#### Legacy Terraform + +The Legacy Terraform provider adds preliminary swap support in v0.6.1, via the [`swap_enabled`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_swap_enabled) variable. +With this change, the Terraform: + - Creates a node group for Materialize. + - Configures NVMe instance store volumes as swap using a daemonset. + - Enables swap at the Kubelet. + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#v061). + +{{< note >}} +If deploying `v25.2`, Materialize clusters will not automatically use swap unless they are configured with a `memory_request` less than their `memory_limit`. In `v26`, this will be handled automatically. +{{< /note >}} + +{{< /tab >}} +{{< /tabs >}} + +## TLS + +When running with TLS in production, run with certificates from an official +Certificate Authority (CA) rather than self-signed certificates. + +## Upgrading guideline + +{{< include-md file="shared-content/self-managed/general-rules-for-upgrades.md" +>}} diff --git a/doc/user/content/self-managed-deployments/deployment-guidelines/azure-deployment-guidelines.md b/doc/user/content/self-managed-deployments/deployment-guidelines/azure-deployment-guidelines.md new file mode 100644 index 0000000000000..aee6fa39ec978 --- /dev/null +++ b/doc/user/content/self-managed-deployments/deployment-guidelines/azure-deployment-guidelines.md @@ -0,0 +1,91 @@ +--- +title: "Azure deployment guidelines" +description: "General guidelines when deploying Self-Managed Materialize on Azure." +disable_list: true +menu: + main: + parent: "deployment-guidelines" + identifier: "azure-deployment-guidelines" + weight: 20 +aliases: + - /installation/install-on-azure/appendix-deployment-guidelines/ +--- + +## Recommended instance types + +As a general guideline, we recommend: + +- ARM-based CPU. +- A 1:8 ratio of vCPU to GiB memory. +- An 8:1 ratio of GiB local instance storage to GiB memory when using swap. + +### Recommended Azure VM Types with Local NVMe Disks + +When operating on Azure in production, we recommend [Epdsv6 +sizes](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/memory-optimized/epdsv6-series?tabs=sizebasic#sizes-in-series) +Azure VM Types with Local NVMe Disk: + +| VM Size | vCPUs | Memory | Ephemeral Disk | Disk-to-RAM Ratio | +| ------------------ | ----- | ------- | -------------- | ----------------- | +| Standard_E2pds_v6 | 2 | 16 GiB | 75 GiB | ~4.7:1 | +| Standard_E4pds_v6 | 4 | 32 GiB | 150 GiB | ~4.7:1 | +| Standard_E8pds_v6 | 8 | 64 GiB | 300 GiB | ~4.7:1 | +| Standard_E16pds_v6 | 16 | 128 GiB | 600 GiB | ~4.7:1 | +| Standard_E32pds_v6 | 32 | 256 GiB | 1,200 GiB | ~4.7:1 | + +{{< warning >}} + +These VM types provide **ephemeral** local NVMe SSD disks. Data is +**lost** when the VM is stopped or deleted. + +{{}} + +## Locally-attached NVMe storage + +Configuring swap on nodes to use locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. + +### Swap support + +{{< tabs >}} +{{< tab "New Unified Terraform" >}} +#### New Unified Terraform + +The unified Materialize [Terraform module](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure/examples/simple) supports configuring swap out of the box. + +{{< /tab >}} +{{< tab "Legacy Terraform" >}} +#### Legacy Terraform + +The Legacy Terraform provider, adds preliminary swap support in v0.6.1, via the [`swap_enabled`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_swap_enabled) variable. +With this change, the Terraform: + - Creates a node group for Materialize. + - Configures NVMe instance store volumes as swap using a daemonset. + - Enables swap at the Kubelet. + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#v061). + +{{< note >}} +If deploying `v25.2`, Materialize clusters will not automatically use swap unless they are configured with a `memory_request` less than their `memory_limit`. In `v26`, this will be handled automatically. +{{< /note >}} + +{{< /tab >}} +{{< /tabs >}} + +## Recommended Azure Blob Storage + +Materialize writes **block** blobs on Azure. As a general guideline, we +recommend **Premium block blob** storage accounts. + +## TLS + +When running with TLS in production, run with certificates from an official +Certificate Authority (CA) rather than self-signed certificates. + +## Upgrading guideline + +{{< include-md file="shared-content/self-managed/general-rules-for-upgrades.md" +>}} diff --git a/doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md b/doc/user/content/self-managed-deployments/deployment-guidelines/gcp-deployment-guidelines.md similarity index 54% rename from doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md rename to doc/user/content/self-managed-deployments/deployment-guidelines/gcp-deployment-guidelines.md index e2a051a92772c..9dad9976dd44e 100644 --- a/doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md +++ b/doc/user/content/self-managed-deployments/deployment-guidelines/gcp-deployment-guidelines.md @@ -1,20 +1,23 @@ --- -title: "Appendix: GCP deployment guidelines" -description: "GCP environment setup/deployment guidelines" +title: "GCP deployment guidelines" +description: "General guidelines when deploying Self-Managed Materialize on GCP." +disable_list: true menu: main: - parent: "install-on-gcp" + parent: "deployment-guidelines" identifier: "gcp-deployment-guidelines" - weight: 40 + weight: 30 +aliases: + - /installation/install-on-gcp/appendix-deployment-guidelines/ --- ## Recommended instance types As a general guideline, we recommend: -- Processor Type: ARM-based CPU - -- Sizing: 2:1 disk-to-RAM ratio with spill-to-disk enabled. +- ARM-based CPU. +- A 1:8 ratio of vCPU to GiB memory. +- An 8:1 ratio of GiB local instance storage to GiB memory when using swap. When operating on GCP in production, we recommend the following machine types that support local SSD attachment: @@ -24,20 +27,16 @@ that support local SSD attachment: | [N2 high-memory series] | `n2-highmem-16` or `n2-highmem-32` with local NVMe SSDs | | [N2D high-memory series] | `n2d-highmem-16` or `n2d-highmem-32` with local NVMe SSDs | -To maintain the recommended 2:1 disk-to-RAM ratio for your machine type, see +To maintain the recommended 8:1 disk-to-RAM ratio for your machine type, see [Number of local SSDs](#number-of-local-ssds) to determine the number of local -SSDs -([`disk_support_config.local_ssd_count`](https://github.com/MaterializeInc/terraform-google-materialize/blob/main/README.md#input_disk_support_config)) -to use. +SSDs to use. See also [Locally attached NVMe storage](#locally-attached-nvme-storage). ## Number of local SSDs Each local NVMe SSD in GCP provides 375GB of storage. Use the appropriate number -of local SSDs -([`disk_support_config.local_ssd_count`](https://github.com/MaterializeInc/terraform-google-materialize/blob/main/README.md#input_disk_support_config)) -to ensure your total disk space is at least twice the amount of RAM in your +of local SSDs to ensure your total disk space is at least twice the amount of RAM in your machine type for optimal Materialize performance. {{< note >}} @@ -65,32 +64,42 @@ type](https://cloud.google.com/compute/docs/disks/local-ssd#lssd_disk_options). [N2D high-memory series]: https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machine_types -[enables spill-to-disk]: https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#disk-support-for-materialize-on-gcp ## Locally-attached NVMe storage -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. +Configuring swap on nodes to use locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. ### Swap support -Starting in v0.6.1 of Materialize on Google Cloud PRovider (GCP) Terraform, -disk support (using swap on NVMe instance storage) may be enabled for -Materialize. With this change, the Terraform: +{{< tabs >}} +{{< tab "New Unified Terraform" >}} + +#### New Unified Terraform -- Creates a node group for Materialize. -- Configures NVMe instance store volumes as swap using a daemonset. -- Enables swap at the Kubelet. +The unified Materialize [Terraform module](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp/examples/simple) supports configuring swap out of the box. -For swap support, the following configuration options are available: +{{< /tab >}} +{{< tab "Legacy Terraform" >}} +#### Legacy Terraform -- [`swap_enabled`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_swap_enabled) +The Legacy Terraform provider, adds preliminary swap support in v0.6.1, via the [`swap_enabled`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_swap_enabled) variable. +With this change, the Terraform: + - Creates a node group for Materialize. + - Configures NVMe instance store volumes as swap using a daemonset. + - Enables swap at the Kubelet. See [Upgrade Notes](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#v061). +{{< note >}} +If deploying `v25.2`, Materialize clusters will not automatically use swap unless they are configured with a `memory_request` less than their `memory_limit`. In `v26`, this will be handled automatically. +{{< /note >}} +{{< /tab >}} +{{< /tabs >}} + ## CPU affinity It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). @@ -102,18 +111,7 @@ to substantially improve the performance of compute-bound workloads. When running with TLS in production, run with certificates from an official Certificate Authority (CA) rather than self-signed certificates. -## Storage bucket versioning - -Starting in v0.3.1 of Materialize on GCP Terraform, storage bucket versioning is -disabled (i.e., -[`storage_bucket_versioning`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_versioning) -is set to `false` by default) to facilitate cleanup of resources during testing. -When running in production, versioning should be turned on with a sufficient TTL -([`storage_bucket_version_ttl`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_version_ttl)) -to meet any data-recovery requirements. - -## See also +## Upgrading guideline -- [Configuration](/installation/configuration/) -- [Installation](/installation/) -- [Troubleshooting](/installation/troubleshooting/) +{{< include-md file="shared-content/self-managed/general-rules-for-upgrades.md" +>}} diff --git a/doc/user/content/installation/faq.md b/doc/user/content/self-managed-deployments/faq.md similarity index 72% rename from doc/user/content/installation/faq.md rename to doc/user/content/self-managed-deployments/faq.md index 4107064488008..a355d5fbf547d 100644 --- a/doc/user/content/installation/faq.md +++ b/doc/user/content/self-managed-deployments/faq.md @@ -1,11 +1,12 @@ --- -title: "FAQ: Self-managed installation" -description: "Frequently asked questions about self-managed installations." +title: "FAQ" +description: "Frequently asked questions about self-managed deployments." aliases: - /self-hosted/faq/ + - /installation/faq/ menu: main: - parent: "installation" + parent: "sm-deployments" weight: 92 --- @@ -27,3 +28,8 @@ existing installation, run: ```bash kubectl -n materialize-environment patch secret materialize-backend -p '{"stringData":{"license_key":""}}' --type=merge ``` + +## How can I downgrade Self-Managed Materialize? + +{{< include-from-yaml data="self_managed/upgrades" +name="downgrade-restriction" >}} diff --git a/doc/user/content/self-managed-deployments/installation/_index.md b/doc/user/content/self-managed-deployments/installation/_index.md new file mode 100644 index 0000000000000..0305bdc8ddbf4 --- /dev/null +++ b/doc/user/content/self-managed-deployments/installation/_index.md @@ -0,0 +1,25 @@ +--- +title: "Installation" +description: "Installation guides for Self-Managed Materialize." +disable_list: true +menu: + main: + parent: "sm-deployments" + identifier: "installation" + weight: 5 +aliases: + - /self-managed/v25.1/installation/ + - /installation/ +--- + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-intro" %}} + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-guides-helm" %}} + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-guides-unified" %}} + +{{% include-from-yaml data="self_managed/installation" +name="installation-landing-guides-legacy" %}} diff --git a/doc/user/content/self-managed-deployments/installation/install-on-aws.md b/doc/user/content/self-managed-deployments/installation/install-on-aws.md new file mode 100644 index 0000000000000..9f9cc6275452a --- /dev/null +++ b/doc/user/content/self-managed-deployments/installation/install-on-aws.md @@ -0,0 +1,340 @@ +--- +title: "Install on AWS" +description: "Install Materialize on AWS using the Unified Terraform module." +aliases: + - /self-hosted/install-on-aws/ + - /self-managed/v25.1/installation/install-on-aws/ + - /installation/install-on-aws/ +disable_list: true +menu: + main: + parent: "installation" + identifier: "install-on-aws" + weight: 20 +--- + +Materialize provides a set of modular [Terraform +modules](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) +that can be used to deploy all services required for Materialize to run on AWS. +The module is intended to provide a simple set of examples on how to deploy +Materialize. It can be used as is or modules can be taken from the example and +integrated with existing DevOps tooling. + +{{% self-managed/materialize-components-sentence %}} The example on this page +deploys a complete Materialize environment on AWS using the modular Terraform +setup from this repository. + + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + + +## What Gets Created + +This example provisions the following infrastructure: + +### Networking + +| Resource | Description | +|----------|-------------| +| VPC | 10.0.0.0/16 with DNS hostnames and support enabled | +| Subnets | 3 private subnets (10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24) and 3 public subnets (10.0.101.0/24, 10.0.102.0/24, 10.0.103.0/24) across availability zones us-east-1a, us-east-1b, us-east-1c | +| NAT Gateway | Single NAT Gateway for all private subnets | +| Internet Gateway | For public subnet connectivity | + +### Compute + +| Resource | Description | +|----------|-------------| +| EKS Cluster | Version 1.32 with CloudWatch logging (API, audit) | +| Base Node Group | 2 nodes (t4g.medium) for Karpenter and CoreDNS | +| Karpenter | Auto-scaling controller with two node classes: Generic nodepool (t4g.xlarge instances for general workloads) and Materialize nodepool (r7gd.2xlarge instances with swap enabled and dedicated taints to run materialize instance workloads) | + +### Database + +| Resource | Description | +|----------|-------------| +| RDS PostgreSQL | Version 15, db.t3.large instance | +| Storage | 50GB allocated, autoscaling up to 100GB | +| Deployment | Single-AZ (non-production configuration) | +| Backups | 7-day retention | +| Security | Dedicated security group with access from EKS cluster and nodes | + +### Storage + +| Resource | Description | +|----------|-------------| +| S3 Bucket | Dedicated bucket for Materialize persistence | +| Encryption | Disabled (for testing; enable in production) | +| Versioning | Disabled (for testing; enable in production) | +| IAM Role | IRSA role for Kubernetes service account access | + +### Kubernetes Add-ons + +| Resource | Description | +|----------|-------------| +| AWS Load Balancer Controller | For managing Network Load Balancers | +| cert-manager | Certificate management controller for Kubernetes that automates TLS certificate provisioning and renewal | +| Self-signed ClusterIssuer | Provides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication. | + +### Materialize + +| Resource | Description | +|----------|-------------| +| Operator | Materialize Kubernetes operator in the `materialize` namespace | +| Instance | Single Materialize instance in the `materialize-environment` namespace | +| Network Load Balancer | Dedicated internal NLB for Materialize access {{< yaml-table data="self_managed/default_ports" >}} | + + +## Prerequisites + +### AWS Account Requirements + +An active AWS account with appropriate permissions to create: +- EKS clusters +- RDS instances +- S3 buckets +- VPCs and networking resources +- IAM roles and policies + +### Required Tools + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) +- [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + + +### License Key + +{{< yaml-table data="self_managed/license_key" >}} + +## Getting started: Simple example + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +### Step 1: Set Up the Environment + +1. Open a terminal window. + +1. Clone the Materialize Terraform repository and go to the + `aws/examples/simple` directory. + + ```bash + git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git + cd materialize-terraform-self-managed/aws/examples/simple + ``` + +1. Ensure your AWS CLI is configured with the appropriate profile, substitute + `` with the profile to use: + + ```bash + # Set your AWS profile for the session + export AWS_PROFILE= + ``` + +### Step 2: Configure Terraform Variables + +1. Create a `terraform.tfvars` file with the following variables: + + - `name_prefix`: Prefix for all resource names (e.g., `simple-demo`) + - `aws_region`: AWS region for deployment (e.g., `us-east-1`) + - `aws_profile`: AWS CLI profile to use + - `license_key`: Materialize license key + - `tags`: Map of tags to apply to resources + + ```hcl + name_prefix = "simple-demo" + aws_region = "us-east-1" + aws_profile = "your-aws-profile" + license_key = "your-materialize-license-key" + tags = { + environment = "demo" + } + ``` + +### Step 3: Apply the Terraform + +1. Initialize the Terraform directory to download the required providers + and modules: + + ```bash + terraform init + ``` + +1. Apply the Terraform configuration to create the infrastructure. + + - To deploy with the default **internal NLB** for Materialize access: + + ```bash + terraform apply + ``` + + - To deploy with **public NLB** for Materialize access: + + ```bash + terraform apply -var="internal=false" + ``` + + If you are satisfied with the planned changes, type `yes` when prompted + to proceed. + + +1. From the output, you will need the following fields to connect using the + Materialize Console and PostgreSQL-compatible clients/drivers: + - `nlb_dns_name` + - `external_login_password_mz_system`. + + ```bash + terraform output -raw + ``` + + {{< tip >}} + Your shell may show an ending marker (such as `%`) because the + output did not end with a newline. Do not include the marker when using the value. + {{< /tip >}} + + +1. Configure `kubectl` to connect to your cluster, replacing: + + - `` with the your cluster name; i.e., the + `eks_cluster_name` in the Terraform output. For the + sample example, your cluster name has the form `{prefix_name}-eks`; e.g., + `simple-demo-eks`. + + - `` with the region of your cluster. Your region can be + found in your `terraform.tfvars` file; e.g., `us-east-1`. + + ```bash + # aws eks update-kubeconfig --name --region + aws eks update-kubeconfig --name $(terraform output -raw eks_cluster_name) --region + ``` + +### Step 4. Optional. Verify the deployment. + +1. Check the status of your deployment: + {{% include-from-yaml data="self_managed/installation" + name="installation-verify-status" %}} + +### Step 5: Connect to Materialize + +Using the `dns_name` and `external_login_password_mz_system` from the Terraform +output, you can connect to Materialize via the Materialize Console or +PostgreSQL-compatible tools/drivers using the following ports: + +{{< yaml-table data="self_managed/default_ports" >}} + +{{< note >}} + +If using an **internal Network Load Balancer (NLB)** for your Materialize +access, you can connect from inside the same VPC or from networks that are +privately connected to it. + +{{< /note >}} + +#### Connect to the Materialize Console + +1. To connect to the Materialize Console, open a browser to + `https://:8080`, substituting your ``. + + From the terminal, you can type: + + ```sh + open "https://$(terraform output -raw nlb_dns_name):8080/materialize" + ``` + + {{< tip >}} + + {{% include-from-yaml data="self_managed/installation" + name="install-uses-self-signed-cluster-issuer" %}} + + {{< /tip >}} + +1. Log in as `mz_system`, using `external_login_password_mz_system` as the + password. + +1. Create new users and log out. + + In general, other than the initial login to create new users for new + deployments, avoid using `mz_system` since `mz_system` also used by the + Materialize Operator for upgrades and maintenance tasks. + + For more information on authentication and authorization for Self-Managed + Materialize, see: + + - [Authentication](/security/self-managed/authentication/) + - [Access Control](/security/self-managed/access-control/) + +1. Login as one of the created user. + +#### Connect using `psql` + +1. To connect using `psql`, in the connection string, specify: + + - `mz_system` as the user + - Your `` as the host + - `6875` as the port: + + ```sh + psql "postgres://mz_system@$(terraform output -raw nlb_dns_name):6875/materialize" + ``` + + When prompted for the password, enter the + `external_login_password_mz_system` value. + +1. Create new users and log out. + + In general, other than the initial login to create new users for new + deployments, avoid using `mz_system` since `mz_system` also used by the + Materialize Operator for upgrades and maintenance tasks. + + For more information on authentication and authorization for Self-Managed + Materialize, see: + + - [Authentication](/security/self-managed/authentication/) + - [Access Control](/security/self-managed/access-control/) + +1. Login as one of the created user. + +## Customizing Your Deployment + +{{< tip >}} +To reduce cost in your demo environment, you can tweak subnet CIDRs +and instance types in `main.tf`. +{{< /tip >}} + +You can customize each Terraform module independently. + +- For details on the Terraform modules, see both the [top +level](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) +and [AWS +specific](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws) READMEs. + +- For details on recommended instance sizing and configuration, see the [AWS +deployment +guide](/self-managed-deployments/deployment-guidelines/aws-deployment-guidelines/). + +See also: + +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) + + +## Cleanup + +{{% self-managed/cleanup-cloud %}} + + +## See Also + + +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/content/self-managed-deployments/installation/install-on-azure.md b/doc/user/content/self-managed-deployments/installation/install-on-azure.md new file mode 100644 index 0000000000000..24f44ee809da7 --- /dev/null +++ b/doc/user/content/self-managed-deployments/installation/install-on-azure.md @@ -0,0 +1,318 @@ +--- +title: "Install on Azure" +description: "Install Materialize on Azure using the Unified Terraform module." +menu: + main: + parent: "installation" + identifier: "install-on-azure" + weight: 20 +--- + +Materialize provides a set of modular [Terraform +modules](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) +that can be used to deploy all services required for Materialize to run on Azure. +The module is intended to provide a simple set of examples on how to deploy +Materialize. It can be used as is or modules can be taken from the example and +integrated with existing DevOps tooling. + +{{% self-managed/materialize-components-sentence %}} The example on this page +deploys a complete Materialize environment on Azure using the modular Terraform +setup from this repository. + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +## What Gets Created + +This example provisions the following infrastructure: + +### Resource Group + +| Resource | Description | +|----------|-------------| +| Resource Group | New resource group to contain all resources | + +### Networking + +| Resource | Description | +|----------|-------------| +| Virtual Network | 20.0.0.0/16 address space | +| AKS Subnet | 20.0.0.0/20 with NAT Gateway association and service endpoints for Storage and SQL | +| PostgreSQL Subnet | 20.0.16.0/24 delegated to PostgreSQL Flexible Server | +| NAT Gateway | Standard SKU with static public IP for outbound connectivity | +| Private DNS Zone | For PostgreSQL private endpoint resolution with VNet link | + +### Compute + +| Resource | Description | +|----------|-------------| +| AKS Cluster | Version 1.32 with Cilium networking (network plugin: azure, data plane: cilium, policy: cilium) | +| Default Node Pool | Standard_D4pds_v6 VMs, autoscaling 2-5 nodes, labeled for generic workloads | +| Materialize Node Pool | Standard_E4pds_v6 VMs with 100GB disk, autoscaling 2-5 nodes, swap enabled, dedicated taints for Materialize workloads | +| Managed Identities | AKS cluster identity (used by AKS control plane to provision Azure resources like load balancers and network interfaces) and Workload identity (used by Materialize pods for secure, passwordless authentication to Azure Storage) | + +### Database + +| Resource | Description | +|----------|-------------| +| Azure PostgreSQL Flexible Server | Version 15 | +| SKU | GP_Standard_D2s_v3 (2 vCores, 4GB memory) | +| Storage | 32GB with 7-day backup retention | +| Network Access | Public Network Access is disabled, Private access only (no public endpoint) | +| Database | `materialize` database pre-created | + +### Storage + +| Resource | Description | +|----------|-------------| +| Storage Account | Premium BlockBlobStorage with LRS replication for Materialize persistence | +| Container | `materialize` blob container | +| Access Control | Workload Identity federation for Kubernetes service account (passwordless authentication via OIDC) | +| Network Access | Currently allows **all traffic**(production deployments should restrict to AKS subnet only traffic) | + +### Kubernetes Add-ons + +| Resource | Description | +|----------|-------------| +| cert-manager | Certificate management controller for Kubernetes that automates TLS certificate provisioning and renewal | +| Self-signed ClusterIssuer | Provides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication. | + +### Materialize + +| Resource | Description | +|----------|-------------| +| Operator | Materialize Kubernetes operator in the `materialize` namespace | +| Instance | Single Materialize instance in the `materialize-environment` namespace | +| Load Balancers | Internal Azure Load Balancers for Materialize access {{< yaml-table data="self_managed/default_ports" >}} | + +## Prerequisites + +### Azure Account Requirements + +An active Azure subscription with appropriate permissions to create: +- AKS clusters +- Azure PostgreSQL Flexible Server instances +- Storage accounts +- Virtual networks and networking resources +- Managed identities and role assignments + +### Required Tools + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + +### License Key + +{{< yaml-table data="self_managed/license_key" >}} + +## Getting started: Simple example + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +### Step 1: Set Up the Environment + +1. Open a terminal window. + +1. Clone the Materialize Terraform repository and go to the + `azure/examples/simple` directory. + + ```bash + git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git + cd materialize-terraform-self-managed/azure/examples/simple + ``` + +1. Authenticate with Azure. + + ```bash + az login + ``` + + The command opens a browser window to sign in to Azure. Sign in. + +1. Select the subscription and tenant to use. After you have signed in, back in + the terminal, your tenant and subscription information is displayed. + + ```none + Retrieving tenants and subscriptions for the selection... + + [Tenant and subscription selection] + + No Subscription name Subscription ID Tenant + ----- ------------------- ------------------------------------ ---------------- + [1]* ... ... ... + + The default is marked with an *; the default tenant is '' and + subscription is '' (). + ``` + + Select the subscription and tenant. + +### Step 2: Configure Terraform Variables + +1. Create a `terraform.tfvars` file with the following variables: + + - `subscription_id`: Azure subscription ID + - `resource_group_name`: Name for the resource group to create (e.g. + `mz-demo-rg`) + - `name_prefix`: Prefix for all resource names (e.g., `simple-demo`) + - `location`: Azure region for deployment (e.g., `westus2`) + - `license_key`: Materialize license key + - `tags`: Map of tags to apply to resources + + ```hcl + subscription_id = "your-subscription-id" + resource_group_name = "mz-demo-rg" + name_prefix = "simple-demo" + location = "westus2" + license_key = "your-materialize-license-key" + tags = { + environment = "demo" + } + ``` + +### Step 3: Apply the Terraform + +1. Initialize the Terraform directory to download the required providers + and modules: + + ```bash + terraform init + ``` + +1. Apply the Terraform configuration to create the infrastructure. + + - To deploy with the default **internal NLB** for Materialize access: + + ```bash + terraform apply + ``` + + - To deploy with **public NLB** for Materialize access: + + ```bash + terraform apply -var="internal=false" + ``` + + If you are satisfied with the planned changes, type `yes` when prompted + to proceed. + +1. From the output, you will need the following field(s) to connect: + - `console_load_balancer_ip` for the Materialize Console + - `balancerd_load_balancer_ip` to connect PostgreSQL-compatible clients/drivers. + + ```bash + terraform output -raw + ``` + + {{< tip >}} + Your shell may show an ending marker (such as `%`) because the + output did not end with a newline. Do not include the marker when using the value. + {{< /tip >}} + +1. Configure `kubectl` to connect to your cluster, replacing: + - `` with your resource group name; i.e., the + `resource_group_name` in the Terraform output or in the + `terraform.tfvars` file. + + - `` with your cluster name; i.e., the + `aks_cluster_name` in the Terraform output. For the sample example, + your cluster name has the form `{prefix_name}-aks`; e.g., `simple-demo-aks`. + + ```bash + # az aks get-credentials --resource-group --name + az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -raw aks_cluster_name) + ``` + +### Step 4. Optional. Verify the deployment. + +1. Check the status of your deployment: + {{% include-from-yaml data="self_managed/installation" + name="installation-verify-status" %}} + +### Step 5: Connect to Materialize + +{{< note >}} + +If using an **internal Network Load Balancer (NLB)** for your Materialize +access, you can connect from inside the same VPC or from networks that are +privately connected to it. + +{{< /note >}} + +#### Connect using the Materialize Console + +Using the `console_load_balancer_ip` from the Terraform output, you can connect +to Materialize via the Materialize Console. + +To connect to the Materialize Console, open a browser to +`https://:8080`, substituting your +``. + +From the terminal, you can type: + +```sh +open "https://$(terraform output -raw console_load_balancer_ip):8080/materialize" +``` + +{{< tip >}} + +{{% include-from-yaml data="self_managed/installation" +name="install-uses-self-signed-cluster-issuer" %}} + +{{< /tip >}} + +#### Connect using the `psql` + +Using the `balancerd_load_balancer_ip` value from the Terraform output, you can +connect to Materialize via PostgreSQL-compatible clients/drivers, such as +`psql`: + +```bash +psql "postgres://$(terraform output -raw balancerd_load_balancer_ip):6875/materialize" +``` + + +## Customizing Your Deployment + +{{< tip >}} +To reduce cost in your demo environment, you can tweak VM sizes and database tiers in `main.tf`. +{{< /tip >}} + +You can customize each Terraform module independently. + +- For details on the Terraform modules, see both the [top +level](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) +and [Azure +specific](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure) modules. + +- For details on recommended instance sizing and configuration, see the [Azure +deployment +guide](/self-managed-deployments/deployment-guidelines/azure-deployment-guidelines/). + +{{< note >}} +Autoscaling: Uses Azure's native cluster autoscaler that integrates directly with Azure Virtual Machine Scale Sets for automated node scaling. In future we are planning to enhance this by making use of karpenter-provider-azure. +{{< /note >}} + +See also: +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) + +## Cleanup + +{{% self-managed/cleanup-cloud %}} + +## See Also + +- [Materialize Operator Configuration](/installation/configuration/) +- [Troubleshooting](/installation/troubleshooting/) diff --git a/doc/user/content/self-managed-deployments/installation/install-on-gcp.md b/doc/user/content/self-managed-deployments/installation/install-on-gcp.md new file mode 100644 index 0000000000000..3c5edd297f4c7 --- /dev/null +++ b/doc/user/content/self-managed-deployments/installation/install-on-gcp.md @@ -0,0 +1,334 @@ +--- +title: "Install on GCP" +description: "" +menu: + main: + parent: "installation" + identifier: "install-gcp-terraform" + weight: 30 + +--- + +Materialize provides a set of modular [Terraform +modules](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) +that can be used to deploy all services required for Materialize to run on Google Cloud. +The module is intended to provide a simple set of examples on how to deploy +Materialize. It can be used as is or modules can be taken from the example and +integrated with existing DevOps tooling. + +{{% self-managed/materialize-components-sentence %}} The example on this page +deploys a complete Materialize environment on GCP using the modular Terraform +setup from this repository. + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +## What Gets Created + +This example provisions the following infrastructure: + +### Networking + +| Resource | Description | +|----------|-------------| +| VPC Network | Custom VPC with auto-create subnets disabled | +| Subnet | 192.168.0.0/20 primary range with private Google access enabled | +| Secondary Ranges | Pods: 192.168.64.0/18, Services: 192.168.128.0/20 | +| Cloud Router | For NAT and routing configuration | +| Cloud NAT | For outbound internet access from private nodes | +| VPC Peering | Service networking connection for Cloud SQL private access | + +### Compute + +| Resource | Description | +|----------|-------------| +| GKE Cluster | Regional cluster with Workload Identity enabled | +| Generic Node Pool | e2-standard-8 machines, autoscaling 2-5 nodes, 50GB disk, for general workloads | +| Materialize Node Pool | n2-highmem-8 machines, autoscaling 2-5 nodes, 100GB disk, 1 local SSD, swap enabled, dedicated taints for Materialize workloads | +| Service Account | GKE service account with workload identity binding | + +### Database + +| Resource | Description | +|----------|-------------| +| Cloud SQL PostgreSQL | Private IP only (no public IP) | +| Tier | db-custom-2-4096 (2 vCPUs, 4GB memory) | +| Database | `materialize` database with UTF8 charset | +| User | `materialize` user with auto-generated password | +| Network | Connected via VPC peering for private access | + +### Storage + +| Resource | Description | +|----------|-------------| +| Cloud Storage Bucket | Regional bucket for Materialize persistence | +| Access | HMAC keys for S3-compatible access (Workload Identity service account with storage permissions is configured but not currently used by Materialize for GCS access, in future we will remove HMAC keys and support access to GCS either via Workload Identity Federation or via Kubernetes ServiceAccounts that impersonate IAM service accounts) | +| Versioning | Disabled (for testing; enable in production) | + +### Kubernetes Add-ons + +| Resource | Description | +|----------|-------------| +| cert-manager | Certificate management controller for Kubernetes that automates TLS certificate provisioning and renewal | +| Self-signed ClusterIssuer | Provides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication. | + +### Materialize + +| Resource | Description | +|----------|-------------| +| Operator | Materialize Kubernetes operator in the `materialize` namespace | +| Instance | Single Materialize instance in the `materialize-environment` namespace | +| Load Balancers | GCP Load Balancers for Materialize access {{< yaml-table data="self_managed/default_ports" >}} | + +## Prerequisites + +### GCP Account Requirements + +A Google account with permission to: +- Enable Google Cloud APIs/services on for your project. +- Create: + - GKE clusters + - Cloud SQL instances + - Cloud Storage buckets + - VPC networks and networking resources + - Service accounts and IAM bindings + +### Required Tools + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [gcloud CLI](https://cloud.google.com/sdk/docs/install) +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) +- [kubectl gke plugin](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin) + +### License Key + +{{< yaml-table data="self_managed/license_key" >}} + +## Getting started: Simple example + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +### Step 1: Set Up the Environment + +1. Open a terminal window. + +1. Clone the Materialize Terraform repository and go to the + `gcp/examples/simple` directory. + + ```bash + git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git + cd materialize-terraform-self-managed/gcp/examples/simple + ``` + +1. Authenticate to GCP with your user account. + + ```bash + gcloud auth login + ``` + +1. Find the list of GCP projects: + + ```bash + gcloud projects list + ``` + +1. Set your active GCP project, substitute with your ``. + + ```bash + gcloud config set project + ``` + +1. Enable the following APIs for your project: + + ```bash + gcloud services enable container.googleapis.com # For creating Kubernetes clusters + gcloud services enable compute.googleapis.com # For creating GKE nodes and other compute resources + gcloud services enable sqladmin.googleapis.com # For creating databases + gcloud services enable cloudresourcemanager.googleapis.com # For managing GCP resources + gcloud services enable servicenetworking.googleapis.com # For private network connections + gcloud services enable iamcredentials.googleapis.com # For security and authentication + gcloud services enable iam.googleapis.com # For managing IAM service accounts and policies + gcloud services enable storage.googleapis.com # For Cloud Storage buckets + ``` + +1. Authenticate application default credentials for Terraform + + ```bash + gcloud auth application-default login + ``` + +### Step 2: Configure Terraform Variables + +1. Create a `terraform.tfvars` file and specify the following variables: + + | Variable | Description | + | ----------- | ----------------------------| + | `project_id` | Set to your GCP project ID. | + | `name_prefix` | Set a prefix for all resource names (e.g., `simple-demo`) as well as your release name for the Operator | + | `region` | Set the GCP region for the deployment (e.g., `us-central1`). | + | `license_key` | Set to your Materialize license key. | + | `labels` | Set to the labels to apply to resources. | + + ```bash + project_id = "my-gcp-project" + name_prefix = "simple-demo" + region = "us-central1" + license_key = "your-materialize-license-key" + labels = { + environment = "demo" + created_by = "terraform" + } + ``` + +### Step 3: Apply the Terraform + +1. Initialize the Terraform directory to download the required providers + and modules: + + ```bash + terraform init + ``` + +1. Apply the Terraform configuration to create the infrastructure. + + - To deploy with the default **internal NLB** for Materialize access: + + ```bash + terraform apply + ``` + + - To deploy with **public NLB** for Materialize access: + + ```bash + terraform apply -var="internal=false" + ``` + + If you are satisfied with the planned changes, type `yes` when prompted + to proceed. + +1. From the output, you will need the following field(s) to connect: + - `console_load_balancer_ip` for the Materialize Console + - `balancerd_load_balancer_ip` to connect PostgreSQL-compatible clients/drivers. + + ```bash + terraform output -raw + ``` + + {{< tip >}} + Your shell may show an ending marker (such as `%`) because the + output did not end with a newline. Do not include the marker when using the value. + {{< /tip >}} + +1. Configure `kubectl` to connect to your GKE cluster, replacing: + + - `` with your cluster name; i.e., the + `gke_cluster_name` in the Terraform output. For the sample example, your + cluster name has the form `-gke`; e.g., `simple-demo-gke` + + - `` with your cluster location; i.e., the + `gke_cluster_location` in the Terraform output. Your + region can also be found in your `terraform.tfvars` file. + + - `` with your GCP project ID. + + ```bash + # gcloud container clusters get-credentials --region --project + gcloud container clusters get-credentials $(terraform output -raw gke_cluster_name) \ + --region $(terraform output -raw gke_cluster_location) \ + --project + ``` + +### Step 4. Optional. Verify the status of your deployment + +1. Check the status of your deployment: + {{% include-from-yaml data="self_managed/installation" + name="installation-verify-status" %}} + +### Step 5: Connect to Materialize + +{{< note >}} + +If using an **internal Network Load Balancer (NLB)** for your Materialize +access, you can connect from inside the same VPC or from networks that are +privately connected to it. + +{{< /note >}} + +#### Connect using the Materialize Console + +Using the `console_load_balancer_ip` from the Terraform output, you can connect +to Materialize via the Materialize Console. + +To connect to the Materialize Console, open a browser to +`https://:8080`, substituting your +``. + +From the terminal, you can type: + +```sh +open "https://$(terraform output -raw console_load_balancer_ip):8080/materialize" +``` + +{{< tip >}} + +{{% include-from-yaml data="self_managed/installation" +name="install-uses-self-signed-cluster-issuer" %}} + +{{< /tip >}} + +#### Connect using the `psql` + +Using the `balancerd_load_balancer_ip` value from the Terraform output, you can +connect to Materialize via PostgreSQL-compatible clients/drivers, such as +`psql`: + +```bash +psql "postgres://$(terraform output -raw balancerd_load_balancer_ip):6875/materialize" +``` + +## Customizing Your Deployment + +{{< tip >}} +To reduce cost in your demo environment, you can tweak machine types and database tiers in `main.tf`. +{{< /tip >}} + +You can customize each module independently. + +- For details on the Terraform modules, see both the [top +level](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) +and [GCP +specific](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp) +modules. + +- For details on recommended instance sizing and configuration, see the [GCP +deployment +guide](/self-managed-deployments/deployment-guidelines/gcp-deployment-guidelines/). + +{{< note >}} +**GCP Storage Authentication Limitation:** Materialize currently only supports HMAC key authentication for GCS access (S3-compatible API). +Current State: The modules configure both HMAC keys and Workload Identity, but Materialize uses HMAC keys for actual storage access. +Future: Native GCS access via Workload Identity Federation or Kubernetes service account impersonation will be supported in a future release, eliminating the need for static credentials. +{{< /note >}} + +See also: +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) + +## Cleanup + +{{% self-managed/cleanup-cloud %}} + +## See Also + +- [Materialize Operator Configuration](/installation/configuration/) +- [Troubleshooting](/installation/troubleshooting/) diff --git a/doc/user/content/installation/install-on-local-kind/_index.md b/doc/user/content/self-managed-deployments/installation/install-on-local-kind.md similarity index 99% rename from doc/user/content/installation/install-on-local-kind/_index.md rename to doc/user/content/self-managed-deployments/installation/install-on-local-kind.md index bb7bfb41c5af3..88f380e3ef98b 100644 --- a/doc/user/content/installation/install-on-local-kind/_index.md +++ b/doc/user/content/self-managed-deployments/installation/install-on-local-kind.md @@ -1,17 +1,17 @@ --- -title: "Install locally on kind (via Helm)" +title: "Install locally on kind" description: "Deploy Self-managed Materialize to a local kind cluster." aliases: - /self-hosted/install-on-local-kind/ - /installation/install-on-local-minikube/ - /self-managed/v25.1/installation/install-on-local-kind/ - /self-managed/v25.1/installation/install-on-local-minikube/ + - /installation/install-on-local-kind/ menu: main: parent: "installation" identifier: "install-on-local-kind" weight: 10 -disable_list: true --- {{% self-managed/materialize-components-sentence %}} diff --git a/doc/user/content/self-managed-deployments/installation/legacy/_index.md b/doc/user/content/self-managed-deployments/installation/legacy/_index.md new file mode 100644 index 0000000000000..d64b5b7c6fb01 --- /dev/null +++ b/doc/user/content/self-managed-deployments/installation/legacy/_index.md @@ -0,0 +1,12 @@ +--- +title: "Install Guides (Legacy)" +description: "Install Self-Managed Materialize using legacy Terraform modules" +menu: + main: + parent: "installation" + weight: 50 + identifier: "installation-legacy" +--- + +For deployments that are using the legacy Terraform modules, the following +upgrade guides are available: diff --git a/doc/user/content/installation/install-on-aws/_index.md b/doc/user/content/self-managed-deployments/installation/legacy/install-on-aws-legacy.md similarity index 92% rename from doc/user/content/installation/install-on-aws/_index.md rename to doc/user/content/self-managed-deployments/installation/legacy/install-on-aws-legacy.md index 4c16e703b7763..a06b61d473ef0 100644 --- a/doc/user/content/installation/install-on-aws/_index.md +++ b/doc/user/content/self-managed-deployments/installation/legacy/install-on-aws-legacy.md @@ -1,29 +1,28 @@ --- -title: "Install on AWS (via Terraform)" -description: "" -aliases: - - /self-hosted/install-on-aws/ - - /self-managed/v25.1/installation/install-on-aws/ +title: "Install on AWS(Legacy Terraform)" +description: "Install on AWS using the legacy Terraform module." disable_list: true +disable_toc: true menu: main: - parent: "installation" - identifier: "install-on-aws" - weight: 20 + parent: "installation-legacy" + identifier: "install-aws-terraform-legacy" + weight: 5 --- + {{% self-managed/materialize-components-sentence %}} The tutorial deploys Materialize to AWS Elastic Kubernetes Service (EKS) with a PostgreSQL RDS database as the metadata database and AWS S3 for blob storage. -The tutorial uses [Materialize on AWS Terraform +The tutorial uses the [Legacy Materialize on AWS Terraform module](https://github.com/MaterializeInc/terraform-aws-materialize) to: - Set up the AWS Kubernetes environment. - Call - [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) - module to deploy Materialize Operator and Materialize instances to that EKS - cluster. + [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) + module to deploy Materialize Operator and Materialize instances to that EKS + cluster. {{< warning >}} @@ -33,8 +32,7 @@ module](https://github.com/MaterializeInc/terraform-aws-materialize) to: {{% self-managed/aws-recommended-instances %}} -See [Appendix: AWS Deployment -guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) for +See [AWS Deployment guidelines](/self-managed-deployments/deployment-guidelines/aws-deployment-guidelines/) for more information. ## Prerequisites @@ -62,9 +60,7 @@ documentation](https://helm.sh/docs/intro/install/). ### License key -Starting in v26.0, Self-Managed Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} +{{< include-md file="shared-content/license-key-required.md" >}} ## Set up AWS Kubernetes environment and install Materialize @@ -272,7 +268,7 @@ components: `mz_instances.tfvars` file with the [Materialize instance configuration](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances). - For example, the following specifies the configuration for a `demo`: + For example, the following specifies the configuration for a `demo` instance. ```bash cat < mz_instances.tfvars @@ -285,16 +281,12 @@ components: cpu_request = "1" memory_request = "2Gi" memory_limit = "2Gi" - license_key = "" + license_key = "" } ] EOF ``` - - **Starting in v26.0**, Self-Managed Materialize requires a license key. To - get your license key: - {{% yaml-table data="self_managed/license_key" %}} - - **Starting in v0.3.0**, the Materialize on AWS Terraform module also deploys, by default, Network Load Balancers (NLBs) for each Materialize instance (i.e., the @@ -316,7 +308,7 @@ components: {{< tip >}} {{% self-managed/aws-terraform-upgrade-notes %}} - See [Materialize on AWS releases](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) for notable changes. + See [Materialize on AWS releases](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-aws-terraform-module) for notable changes. {{}} 1. Run `terraform plan` with both `.tfvars` files and review the changes to be @@ -417,7 +409,7 @@ components: ``` If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting/). + [Troubleshooting](/self-managed-deployments/troubleshooting/). 1. Open the Materialize Console in your browser: @@ -482,8 +474,6 @@ components: ## See also -- [Materialize Operator Configuration](/installation/configuration/) -- [Troubleshooting](/installation/troubleshooting/) -- [Appendix: AWS Deployment -guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) -- [Installation](/installation/) +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/content/installation/install-on-azure/_index.md b/doc/user/content/self-managed-deployments/installation/legacy/install-on-azure-legacy.md similarity index 96% rename from doc/user/content/installation/install-on-azure/_index.md rename to doc/user/content/self-managed-deployments/installation/legacy/install-on-azure-legacy.md index d61eba7238635..d31c46f28857d 100644 --- a/doc/user/content/installation/install-on-azure/_index.md +++ b/doc/user/content/self-managed-deployments/installation/legacy/install-on-azure-legacy.md @@ -1,16 +1,14 @@ --- -title: "Install on Azure (via Terraform)" +title: "Install on Azure (Legacy Terraform)" description: "Install Materialize on Azure Kubernetes Service (AKS) using Terraform" -disable_list: true menu: main: - parent: "installation" - identifier: "install-on-azure" - weight: 30 -aliases: - - /self-managed/v25.1/installation/install-on-azure/ + parent: "installation-legacy" + identifier: "install-azure-terraform-legacy" + weight: 20 --- + {{% self-managed/materialize-components-sentence blobstorage="blob storage; specifically **block** blob storage on Azure" %}} The tutorial deploys Materialize to Azure Kubernetes Service (AKS) with a @@ -69,9 +67,7 @@ If you want to use `jq` and do not have `jq` installed, install. ### License key -Starting in v26.0, Self-Managed Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} +{{< include-md file="shared-content/license-key-required.md" >}} ## A. Authenticate with Azure @@ -339,16 +335,12 @@ deploys a sample infrastructure on Azure with the following components: cpu_request = "1" memory_request = "2Gi" memory_limit = "2Gi" - license_key = "" + license_key = "" } ] EOF ``` - - **Starting in v26.0**, Self-Managed Materialize requires a license key. To - get your license key: - {{% yaml-table data="self_managed/license_key" %}} - - **Starting in v0.3.0**, the Materialize on Azure Terraform module also deploys, by default, a self-signed `ClusterIssuer`. The `ClusterIssuer` is deployed after the `cert-manager` is deployed and running. @@ -369,7 +361,7 @@ deploys a sample infrastructure on Azure with the following components: {{< tip >}} {{% self-managed/azure-terraform-upgrade-notes %}} - See [Materialize on Azure releases](/installation/appendix-terraforms/#materialize-on-azure-terraform-module) for notable changes. + See [Materialize on Azure releases](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-azure-terraform-module) for notable changes. {{}} 1. Run `terraform plan` with both `.tfvars` files and review the changes to be diff --git a/doc/user/content/installation/install-on-gcp/_index.md b/doc/user/content/self-managed-deployments/installation/legacy/install-on-gcp-legacy.md similarity index 96% rename from doc/user/content/installation/install-on-gcp/_index.md rename to doc/user/content/self-managed-deployments/installation/legacy/install-on-gcp-legacy.md index b6c9117b1f6e8..b91fdb8f99eef 100644 --- a/doc/user/content/installation/install-on-gcp/_index.md +++ b/doc/user/content/self-managed-deployments/installation/legacy/install-on-gcp-legacy.md @@ -1,15 +1,13 @@ --- -title: "Install on GCP (via Terraform)" +title: "Install on GCP (Legacy Terraform)" description: "" aliases: - /self-hosted/install-on-gcp/ - - /self-managed/v25.1/installation/install-on-gcp/ -disable_list: true menu: main: - parent: "installation" - identifier: "install-on-gcp" - weight: 40 + parent: "installation-legacy" + identifier: "install-gcp-terraform-legacy" + weight: 30 --- {{% self-managed/materialize-components-sentence %}} @@ -94,9 +92,7 @@ If you want to use `jq` and do not have `jq` installed, install. ### License key -Starting in v26.0, Self-Managed Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} +{{< include-md file="shared-content/license-key-required.md" >}} ## A. Configure GCP project and service account @@ -407,16 +403,12 @@ components: cpu_request = "1" memory_request = "2Gi" memory_limit = "2Gi" - license_key = "" + license_key = "" } ] EOF ``` - - **Starting in v26.0**, Self-Managed Materialize requires a license key. To - get your license key: - {{% yaml-table data="self_managed/license_key" %}} - - **Starting in v0.3.0**, the Materialize on GCP Terraform module also deploys, by default: @@ -431,7 +423,7 @@ components: {{< tip >}} {{% self-managed/gcp-terraform-upgrade-notes %}} - See [Materialize on GCP releases](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) for notable changes. + See [Materialize on GCP releases](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-gcp-terraform-module) for notable changes. {{}} 1. Run `terraform plan` with both `.tfvars` files and review the changes to be @@ -584,7 +576,5 @@ components: ## See also -- [Troubleshooting](/installation/troubleshooting/) +- [Troubleshooting](/self-managed-deployments/troubleshooting/) - [Materialize Operator Configuration](/installation/configuration/) -- [Appendix: Google deployment guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/) -- [Installation](/installation/) diff --git a/doc/user/content/self-managed-deployments/materialize-crd-field-descriptions.md b/doc/user/content/self-managed-deployments/materialize-crd-field-descriptions.md new file mode 100644 index 0000000000000..bca28ab7a50d5 --- /dev/null +++ b/doc/user/content/self-managed-deployments/materialize-crd-field-descriptions.md @@ -0,0 +1,13 @@ +--- +title: "Materialize CRD Field Descriptions" +description: "Reference page on Materialize CRD Fields" +menu: + main: + parent: "sm-deployments" + identifier: "materialize-crd-field-descriptions" + weight: 66 +aliases: + - /installation/appendix-materialize-crd-field-descriptions/ +--- + +{{% self-managed/materialize-crd-descriptions %}} diff --git a/doc/user/content/installation/configuration.md b/doc/user/content/self-managed-deployments/operator-configuration.md similarity index 91% rename from doc/user/content/installation/configuration.md rename to doc/user/content/self-managed-deployments/operator-configuration.md index 181e3ee7ebfc1..8ce9313942a04 100644 --- a/doc/user/content/installation/configuration.md +++ b/doc/user/content/self-managed-deployments/operator-configuration.md @@ -4,10 +4,12 @@ description: "Configuration reference for the Materialize Operator Helm chart" aliases: - /self-hosted/configuration/ - /self-managed/v25.1/installation/configuration/ + - /self-managed/nstallation/configuration/ + - /installation/configuration/ menu: main: - parent: "installation" - weight: 70 + parent: "sm-deployments" + weight: 65 --- ## Configure the Materialize operator diff --git a/doc/user/content/installation/release-versions.md b/doc/user/content/self-managed-deployments/release-versions.md similarity index 65% rename from doc/user/content/installation/release-versions.md rename to doc/user/content/self-managed-deployments/release-versions.md index bd4ef9565b2e9..a200882295102 100644 --- a/doc/user/content/installation/release-versions.md +++ b/doc/user/content/self-managed-deployments/release-versions.md @@ -3,8 +3,10 @@ title: "Self-managed release versions" description: "" menu: main: - parent: "installation" - weight: 85 + parent: "sm-deployments" + weight: 95 +aliases: + - /installation/release-versions/ --- ## V26 releases diff --git a/doc/user/content/installation/troubleshooting.md b/doc/user/content/self-managed-deployments/troubleshooting.md similarity index 95% rename from doc/user/content/installation/troubleshooting.md rename to doc/user/content/self-managed-deployments/troubleshooting.md index a7fdd20674421..5be696e1d8d7b 100644 --- a/doc/user/content/installation/troubleshooting.md +++ b/doc/user/content/self-managed-deployments/troubleshooting.md @@ -4,16 +4,15 @@ description: "" aliases: - /self-hosted/troubleshooting/ - /self-managed/v25.1/installation/troubleshooting/ + - /installation/troubleshooting/ menu: main: - parent: "installation" + parent: "sm-deployments" weight: 90 --- ## Troubleshooting Kubernetes -### Materialize operator - To check the status of the Materialize operator: ```shell @@ -127,8 +126,3 @@ To increase the cluster's size, you can follow the following steps: quickstart | r1 (25cc) | (6 rows) ``` - -## See also - -- [Configuration](/installation/configuration/) -- [Installation](/installation/) diff --git a/doc/user/content/self-managed-deployments/upgrading/_index.md b/doc/user/content/self-managed-deployments/upgrading/_index.md new file mode 100644 index 0000000000000..1f8cde017e551 --- /dev/null +++ b/doc/user/content/self-managed-deployments/upgrading/_index.md @@ -0,0 +1,271 @@ +--- +title: "Upgrading" +description: "Upgrading Self-Managed Materialize." +disable_list: true +menu: + main: + parent: "sm-deployments" + weight: 30 + identifier: "upgrading" +--- + +Materialize releases new Self-Managed versions per the schedule outlined in [Release schedule](/releases/schedule/#self-managed-release-schedule). + +## General rules for upgrading + +{{< include-from-yaml data="self_managed/upgrades" +name="upgrades-general-rules" >}} + +{{< note >}} + +{{< include-from-yaml data="self_managed/upgrades" +name="upgrade-major-version-restriction" >}} + +{{< /note >}} + +## Upgrading the Helm Chart and Materialize Operator + +{{< important >}} + +When upgrading Materialize, always upgrade the Helm Chart and Materialize +Operator first. + +{{}} + +### Update the Helm Chart repository + +To update your Materialize Helm Chart repository: + +```shell +helm repo update materialize +``` + +View the available chart versions: + +```shell +helm search repo materialize/materialize-operator --versions +``` + +### Upgrade your Materialize Operator + +The Materialize Kubernetes Operator is deployed via Helm and can be updated +through standard `helm upgrade` command: + +{{% include-syntax file="self_managed/upgrades" +example="syntax-helm-upgrade-operator" %}} + +You can use `helm list` to find your release name. For example, if your Operator +is running in the namespace `materialize`, run `helm list`: + +```shell +helm list -n materialize +``` + +Retrieve the name associated with the `materialize-operator` **CHART**; for +example, `my-demo` in the following helm list: + +```none +NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION +my-demo materialize 1 2025-12-08 11:39:50.185976 -0500 EST deployed materialize-operator-v26.1.0 v26.1.0 +``` + +Then, to upgrade: + +```shell +helm upgrade -n materialize my-demo materialize/operator \ + -f my-values.yaml \ + --version {{< self-managed/versions/get-latest-version >}} +``` + +## Upgrading Materialize Instances + +**After** you have upgraded your Materialize Operator, upgrade your Materialize +instance(s) to the **APP Version** of the Operator. To find the version of your +currently deployed Materialize Operator: + +```shell +helm list -n materialize +``` + +You will use the returned **App Version** for the updated `environmentdImageRef` +value. Specifically, for your Materialize instance(s), set +`environmentdImageRef` value to use the new version: + +``` +spec: + environmentdImageRef: docker.io/materialize/environmentd: +``` + +To minimize unexpected downtime and avoid connection drops at critical +periods for your application, the upgrade process involves two steps: + +- First, stage the changes (update the `environmentdImageRef` with the new + version) to the Materialize custom resource. The Operator watches for changes + but does not automatically roll out the changes. + +- Second, roll out the changes by specifying a new UUID for `requestRollout`. + +### Stage the Materialize instance version change + +To stage the Materialize instances version upgrade, update the +`environmentdImageRef` field in the Materialize custom resource spec to the +compatible version of your currently deployed Materialize Operator. + +To stage, but **not** rollout, the Materialize instance version upgrade, you can +use the `kubectl patch` command; for example, if the **App Version** is {{< self-managed/versions/get-latest-version >}}: + +```shell +kubectl patch materialize \ + -n \ + --type='merge' \ + -p "{\"spec\": {\"environmentdImageRef\": \"docker.io/materialize/environmentd:{{< self-managed/versions/get-latest-version >}}\"}}" +``` + +{{< note >}} +Until you specify a new `requestRollout`, the Operator watches for updates but +does not roll out the changes. +{{< /note >}} + + +### Applying the changes via `requestRollout` + +To apply chang Materialize instance upgrade, you must update the `requestRollout` field in the Materialize custom resource spec to a new UUID. +Be sure to consult the [Rollout Configurations](#rollout-configuration) to ensure you've selected the correct rollout behavior. +```shell +# Then trigger the rollout with a new UUID +kubectl patch materialize \ + -n \ + --type='merge' \ + -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\"}}" +``` + +### Staging and applying in a single command + +Although separating the staging and rollout of the changes into two steps can +minimize unexpected downtime and avoid connection drops at critical periods, you +can, if preferred, combine both operations in a single command + +```shell +kubectl patch materialize \ + -n materialize-environment \ + --type='merge' \ + -p "{\"spec\": {\"environmentdImageRef\": \"docker.io/materialize/environmentd:{{< self-managed/versions/get-latest-version >}}\", \"requestRollout\": \"$(uuidgen)\"}}" +``` + +#### Using YAML Definition + +Alternatively, you can update your Materialize custom resource definition directly: + +```yaml +apiVersion: materialize.cloud/v1alpha1 +kind: Materialize +metadata: + name: 12345678-1234-1234-1234-123456789012 + namespace: materialize-environment +spec: + environmentdImageRef: materialize/environmentd:{{< self-managed/versions/get-latest-version >}} # Update version as needed + requestRollout: 22222222-2222-2222-2222-222222222222 # Use a new UUID + forceRollout: 33333333-3333-3333-3333-333333333333 # Optional: for forced rollouts + inPlaceRollout: false # In Place rollout is deprecated and ignored. Please use rolloutStrategy + rolloutStrategy: WaitUntilReady # The mechanism to use when rolling out the new version. Can be WaitUntilReady or ImmediatelyPromoteCausingDowntime + backendSecretName: materialize-backend +``` + +Apply the updated definition: + +```shell +kubectl apply -f materialize.yaml +``` + +## Rollout Configuration + +### `requestRollout` + +Specify a new `UUID` value for the `requestRollout` to roll out the changes to +the Materialize instance. + +{{< note >}} + +`requestRollout` without the `forcedRollout` field only rolls out if changes +exist to the Materialize instance. To roll out even if there are no changes to +the instance, use with `forcedRollouts`. + +{{< /note >}} + +```shell +# Only rolls out if there are changes +kubectl patch materialize \ + -n \ + --type='merge' \ + -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\"}}" +``` +### `requestRollout` with `forcedRollouts` + +Specify a new `UUID` value for `forcedRollout` to roll out even when there are +no changes to the instance. Use `forcedRollout` with `requestRollout`. + +```shell +kubectl patch materialize \ + -n materialize-environment \ + --type='merge' \ + -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\", \"forceRollout\": \"$(uuidgen)\"}}" +``` + +### Rollout strategies + +The behavior of the new version rollout follows your `rolloutStrategy` setting: + +| `rolloutStrategy` | Description | +| ----------------- | -----------------------------------| +| `WaitUntilReady` | *Default*. New instances are created and all dataflows are determined to be ready before cutover and terminating the old version, temporarily requiring twice the resources during the transition. | +| `ImmediatelyPromoteCausingDowntime`| Tears down the prior version before creating and promoting the new version. This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources. | +| `inPlaceRollout`| *Deprecated*. The setting is ignored. | + +## Verifying the Upgrade + +After initiating the rollout, you can monitor the status field of the Materialize custom resource to check on the upgrade. + +```shell +# Watch the status of your Materialize environment +kubectl get materialize -n materialize-environment -w + +# Check the logs of the operator +kubectl logs -l app.kubernetes.io/name=materialize-operator -n materialize +``` + +## Upgrade guides + +The following upgrade guides are available: + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-landing-guides-helm" %}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-landing-guides-unified" %}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-landing-guides-legacy" %}} + +## Version Specific Upgrade Notes + +### Upgrading to `v26.1` and later versions + +{{< include-md file="shared-content/self-managed/upgrade-notes/v26.1.md" >}} + +### Upgrading to `v26.0` + +{{< include-md file="shared-content/self-managed/upgrade-notes/v26.0.md" >}} + +### Upgrading between minor versions less than `v26` + - Prior to `v26`, you must upgrade at most one minor version at a time. For + example, upgrading from `v25.1.5` to `v25.2.16` is permitted. + +## See also + +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) + +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) + +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/content/self-managed-deployments/upgrading/legacy/_index.md b/doc/user/content/self-managed-deployments/upgrading/legacy/_index.md new file mode 100644 index 0000000000000..f576539376766 --- /dev/null +++ b/doc/user/content/self-managed-deployments/upgrading/legacy/_index.md @@ -0,0 +1,12 @@ +--- +title: "Upgrade Guides (Legacy)" +description: "Upgrading Self-Managed Materialize using legacy Terraform modules" +menu: + main: + parent: "upgrading" + weight: 60 + identifier: "upgrading-legacy" +--- + +For deployments that are using the legacy Terraform modules, the following +upgrade guides are available: diff --git a/doc/user/content/installation/install-on-aws/upgrade-on-aws.md b/doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-aws-legacy.md similarity index 93% rename from doc/user/content/installation/install-on-aws/upgrade-on-aws.md rename to doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-aws-legacy.md index 6deb0e7ee509c..aadbcace8f3f4 100644 --- a/doc/user/content/installation/install-on-aws/upgrade-on-aws.md +++ b/doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-aws-legacy.md @@ -1,10 +1,10 @@ --- -title: "Upgrade on AWS (Terraform)" +title: "Upgrade on AWS (Legacy Terraform)" description: "Procedure to upgrade your Materialize operator and instances running on AWS" menu: main: - parent: "install-on-aws" - identifier: "upgrade-on-aws" + parent: "upgrading-legacy" + identifier: "upgrade-on-aws-legacy" weight: 10 --- @@ -102,7 +102,7 @@ documentation](https://helm.sh/docs/intro/install/). {{< tip >}} {{% self-managed/aws-terraform-upgrade-notes %}} - See [Materialize on AWS releases](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) for notable changes. + See [Materialize on AWS releases](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-aws-terraform-module) for notable changes. {{}} diff --git a/doc/user/content/installation/install-on-azure/upgrade-on-azure.md b/doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-azure-legacy.md similarity index 95% rename from doc/user/content/installation/install-on-azure/upgrade-on-azure.md rename to doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-azure-legacy.md index 763d5004888cc..8ce8412780c05 100644 --- a/doc/user/content/installation/install-on-azure/upgrade-on-azure.md +++ b/doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-azure-legacy.md @@ -1,11 +1,14 @@ --- -title: "Upgrade on Azure (Terraform)" +title: "Upgrade on Azure (Legacy Terraform)" description: "Procedure to upgrade your Materialize operator and instances running on Azure" +disable_list: true +disable_toc: true menu: main: - parent: "install-on-azure" - identifier: "upgrade-on-azure" - weight: 10 + parent: "upgrading-legacy" + identifier: "upgrade-on-azure-legacy" + weight: 30 + --- {{< annotation type="Disambiguation" >}} @@ -25,7 +28,7 @@ Materialize deployment running on Azure Kubernetes Service (AKS). The tutorial assumes you have installed Materialize on Azure Kubernetes Service (AKS) using the instructions on [Install on -Azure](/installation/install-on-azure/) (either from the examples/simple +Azure](/self-managed-deployments/installation/legacy/install-on-azure-legacy/) (either from the examples/simple directory or the root). ## Version compatibility diff --git a/doc/user/content/installation/install-on-gcp/upgrade-on-gcp.md b/doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-gcp-legacy.md similarity index 93% rename from doc/user/content/installation/install-on-gcp/upgrade-on-gcp.md rename to doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-gcp-legacy.md index eab4ab0ebe91b..4694f4d1f75f5 100644 --- a/doc/user/content/installation/install-on-gcp/upgrade-on-gcp.md +++ b/doc/user/content/self-managed-deployments/upgrading/legacy/upgrade-on-gcp-legacy.md @@ -1,11 +1,13 @@ --- -title: "Upgrade on GCP (Terraform)" -description: "Procedure to upgrade your Materialize operator and instances running on GCP" +title: "Upgrade on GCP (Legacy Terraform)" +description: "" +disable_list: true +disable_toc: true menu: main: - parent: "install-on-gcp" - identifier: "upgrade-on-gcp" - weight: 10 + parent: "upgrading-legacy" + identifier: "upgrade-on-gcp-legacy-terraform-module" + weight: 40 --- {{< annotation type="Disambiguation" >}} @@ -17,7 +19,7 @@ file="shared-content/self-managed/gcp-terraform-v0.6.1-upgrade-notes.md" >}}. - To upgrade to `v26.0` if **not** using a Materialize-provided Terraforms, you must prepare your nodes by adding the required labels. For detailed instructions, see [Prepare for swap and upgrade to -v26.0](/installation/upgrade-to-swap/). +v26.0](/self-managed-deployments/appendix/upgrade-to-swap/). {{< /annotation >}} @@ -26,7 +28,7 @@ Materialize deployment running on GCP Google Kubernetes Engine (GKE). The tutorial assumes you have installed Materialize on GCP Google Kubernetes Engine (GKE) using the instructions on [Install on -GCP](/installation/install-on-gcp/) (either from the examples/simple directory +GCP](/self-managed-deployments/installation/legacy/install-on-gcp-legacy/) (either from the examples/simple directory or the root). ## Version compatibility @@ -200,7 +202,7 @@ If you want to use `jq` and do not have `jq` installed, install. {{< tip >}} {{% self-managed/gcp-terraform-upgrade-notes %}} - See [Materialize on GCP releases](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) for notable changes. + See [Materialize on GCP releases](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-gcp-terraform-module) for notable changes. {{}} diff --git a/doc/user/content/self-managed-deployments/upgrading/upgrade-on-aws.md b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-aws.md new file mode 100644 index 0000000000000..46cc6b5b360c8 --- /dev/null +++ b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-aws.md @@ -0,0 +1,131 @@ +--- +title: "Upgrade on AWS" +description: "Upgrade Materialize on AWS using the Unified Terraform module." +menu: + main: + parent: "upgrading" + weight: 20 +--- + +The following tutorial upgrades your Materialize deployment running on AWS +Elastic Kubernetes Service (EKS). The tutorial assumes you have installed the +example on [Install on +AWS](/self-managed-deployments/installation/install-on-aws/). + +## Upgrade guidelines + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrades-general-rules" %}} + +{{< note >}} +{{< include-from-yaml data="self_managed/upgrades" +name="upgrade-major-version-restriction" >}} +{{< /note >}} + +{{< note >}} +{{< include-from-yaml data="self_managed/upgrades" +name="downgrade-restriction" >}} +{{< /note >}} + +## Prerequisites + +### Required Tools + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) +- [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + +## Upgrade process + +{{< important >}} + +The following procedure performs a rolling upgrade, where both the old and new Materialize instances are running before the old instances are removed. When performing a rolling upgrade, ensure you have enough resources to support having both the old and new Materialize instances running. + +{{}} + +### Step 1: Set up + +1. Open a Terminal window. + +1. Configure AWS CLI with your AWS credentials. For details, see the [AWS + documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). + +1. Go to the Terraform directory for your Materialize deployment. For example, + if you deployed from the `aws/examples/simple` directory: + + ```bash + cd materialize-terraform-self-managed/aws/examples/simple + ``` + +1. Ensure your AWS CLI is configured with the appropriate profile, substitute + `` with the profile to use: + + ```bash + # Set your AWS profile for the session + export AWS_PROFILE= + ``` + +1. Configure `kubectl` to connect to your EKS cluster, replacing: + + - `` with the your cluster name; i.e., the + `eks_cluster_name` in the Terraform output. For the + sample example, your cluster name has the form `{prefix_name}-eks`; e.g., + `simple-demo-eks`. + + - `` with the region of your cluster. Your region can be + found in your `terraform.tfvars` file; e.g., `us-east-1`. + + ```bash + # aws eks update-kubeconfig --name --region + aws eks update-kubeconfig --name $(terraform output -raw eks_cluster_name) --region + ``` + + To verify that you have configured correctly, run the following command: + + ```bash + kubectl get nodes + ``` + + For help with `kubectl` commands, see [kubectl Quick reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). + +### Step 2: Update the Helm Chart + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-update-helm-chart" %}} + +### Step 3: Upgrade the Materialize Operator + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-materialize-operator" %}} + +### Step 4: Upgrading Materialize Instances + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-materialize-instance" %}} + +## See also + +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/content/self-managed-deployments/upgrading/upgrade-on-azure.md b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-azure.md new file mode 100644 index 0000000000000..e74e064e5dc70 --- /dev/null +++ b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-azure.md @@ -0,0 +1,123 @@ +--- +title: "Upgrade on Azure" +description: "Upgrade Materialize on Azure using the Unified Terraform module." +menu: + main: + parent: "upgrading" + weight: 30 +--- + +The following tutorial upgrades your Materialize deployment running on Azure +Kubernetes Service (AKS). The tutorial assumes you have installed the +example on [Install on +Azure](/self-managed-deployments/installation/install-on-azure/). + +## Upgrade guidelines + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrades-general-rules" %}} + +{{< note >}} +{{< include-from-yaml data="self_managed/upgrades" +name="upgrade-major-version-restriction" >}} +{{< /note >}} + +{{< note >}} +{{< include-from-yaml data="self_managed/upgrades" +name="downgrade-restriction" >}} +{{< /note >}} + +## Prerequisites + +### Required Tools + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) +- [kubectl](https://kubernetes.io/docs/tasks/tools/) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + +## Upgrade process + +{{< important >}} + +The following procedure performs a rolling upgrade, where both the old and new Materialize instances are running before the old instances are removed. When performing a rolling upgrade, ensure you have enough resources to support having both the old and new Materialize instances running. + +{{}} + +### Step 1: Set up + +1. Open a Terminal window. + +1. Configure Azure CLI with your Azure credentials. For details, see the [Azure + documentation](https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli). + +1. Go to the Terraform directory for your Materialize deployment. For example, + if you deployed from the `azure/examples/simple` directory: + + ```bash + cd materialize-terraform-self-managed/azure/examples/simple + ``` + +1. Configure `kubectl` to connect to your AKS cluster, replacing: + + - `` with your resource group name; i.e., the + `resource_group_name` in the Terraform output or in the + `terraform.tfvars` file. + + - `` with your cluster name; i.e., the + `aks_cluster_name` in the Terraform output. For the sample example, + your cluster name has the form `{prefix_name}-aks`; e.g., simple-demo-aks`. + + ```bash + # az aks get-credentials --resource-group --name + az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -raw aks_cluster_name) + ``` + + To verify that you have configured correctly, run the following command: + + ```bash + kubectl get nodes + ``` + + For help with `kubectl` commands, see [kubectl Quick reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). + +### Step 2: Update the Helm Chart + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-update-helm-chart" %}} + +### Step 3: Upgrade the Materialize Operator + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-materialize-operator" %}} + +### Step 4: Upgrading Materialize Instances + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-materialize-instance" %}} + +## See also + +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/content/self-managed-deployments/upgrading/upgrade-on-gcp.md b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-gcp.md new file mode 100644 index 0000000000000..84b8fab7099f4 --- /dev/null +++ b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-gcp.md @@ -0,0 +1,127 @@ +--- +title: "Upgrade on GCP" +description: "Upgrade Materialize on GCP using the Unified Terraform module." +menu: + main: + parent: "upgrading" + weight: 30 +--- + +The following tutorial upgrades your Materialize deployment running on Google +Kubernetes Engine (GKE). The tutorial assumes you have installed the +example on [Install on +GCP](/self-managed-deployments/installation/install-on-gcp/). + +## Upgrade guidelines + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrades-general-rules" %}} + +{{< note >}} +{{< include-from-yaml data="self_managed/upgrades" +name="upgrade-major-version-restriction" >}} +{{< /note >}} + +{{< note >}} +{{< include-from-yaml data="self_managed/upgrades" +name="downgrade-restriction" >}} +{{< /note >}} + +## Prerequisites + +### Required Tools + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) +- [kubectl](https://kubernetes.io/docs/tasks/tools/) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + +## Upgrade process + +{{< important >}} + +The following procedure performs a rolling upgrade, where both the old and new Materialize instances are running before the old instances are removed. When performing a rolling upgrade, ensure you have enough resources to support having both the old and new Materialize instances running. + +{{}} + +### Step 1: Set up + +1. Open a Terminal window. + +1. Configure Google Cloud CLI with your GCP credentials. For details, see the [Google Cloud + documentation](https://cloud.google.com/sdk/docs/initializing). + +1. Go to the Terraform directory for your Materialize deployment. For example, + if you deployed from the `gcp/examples/simple` directory: + + ```bash + cd materialize-terraform-self-managed/gcp/examples/simple + ``` + +1. Configure `kubectl` to connect to your GKE cluster, replacing: + + - `` with your cluster name; i.e., the + `gke_cluster_name` in the Terraform output. For the sample example, your + cluster name has the form `-gke`; e.g., `simple-demo-gke` + + - `` with your cluster location; i.e., the + `gke_cluster_location` in the Terraform output. Your + region can also be found in your `terraform.tfvars` file. + + - `` with your GCP project ID. + + ```bash + # gcloud container clusters get-credentials --region --project + gcloud container clusters get-credentials $(terraform output -raw gke_cluster_name) \ + --region $(terraform output -raw gke_cluster_location) \ + --project + ``` + + To verify that you have configured correctly, run the following command: + + ```bash + kubectl get nodes + ``` + + For help with `kubectl` commands, see [kubectl Quick reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). + +### Step 2: Update the Helm Chart + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-update-helm-chart" %}} + +### Step 3: Upgrade the Materialize Operator + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-materialize-operator" %}} + +### Step 4: Upgrading Materialize Instances + +{{< important >}} + +{{% include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" %}} + +{{}} + +{{% include-from-yaml data="self_managed/upgrades" +name="upgrade-materialize-instance" %}} + +## See also + +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Materialize CRD Field + Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/) +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/content/installation/install-on-local-kind/upgrade-on-local-kind.md b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-kind.md similarity index 80% rename from doc/user/content/installation/install-on-local-kind/upgrade-on-local-kind.md rename to doc/user/content/self-managed-deployments/upgrading/upgrade-on-kind.md index 286b9b970f65a..866e8a933b2b6 100644 --- a/doc/user/content/installation/install-on-local-kind/upgrade-on-local-kind.md +++ b/doc/user/content/self-managed-deployments/upgrading/upgrade-on-kind.md @@ -3,19 +3,19 @@ title: "Upgrade on kind" description: "Upgrade Materialize running locally on a kind cluster." menu: main: - parent: "install-on-local-kind" - identifier: "upgrade-on-local-kind" -weight: 10 + parent: "upgrading" + weight: 10 aliases: - - /installation/install-on-local-minikube/upgrade-on-local-minikube/ - /self-managed/v25.1/installation/install-on-local-kind/upgrade-on-local-kind/ + - /installation/install-on-local-kind/upgrade-on-local-kind/ --- To upgrade your Materialize instances, first choose a new operator version and upgrade the Materialize operator. Then, upgrade your Materialize instances to the same version. The following tutorial upgrades your Materialize deployment running locally on a [`kind`](https://kind.sigs.k8s.io/) cluster. The tutorial assumes you have installed Materialize on `kind` using the -instructions on [Install locally on kind](/installation/install-on-local-kind/). +instructions on [Install locally on +kind](/self-managed-deployments/installation/install-on-local-kind/). {{< include-md file="shared-content/self-managed/version-compatibility-upgrade-banner.md" >}} @@ -53,6 +53,6 @@ having both the old and new Materialize instances running. ## See also -- [Materialize Operator Configuration](/installation/configuration/) -- [Troubleshooting](/installation/troubleshooting/) -- [Installation](/installation/) +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) +- [Troubleshooting](/self-managed-deployments/troubleshooting/) diff --git a/doc/user/data/self_managed/aws_terraform_deployed_components.yml b/doc/user/data/self_managed/aws_terraform_deployed_components.yml index aa25f746aa432..b8ce5adc576fb 100644 --- a/doc/user/data/self_managed/aws_terraform_deployed_components.yml +++ b/doc/user/data/self_managed/aws_terraform_deployed_components.yml @@ -20,14 +20,14 @@ rows: AWS Load Balancer Controller and Network Load Balancers for each Materialize instance Version: | - [v0.3.0+](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) + [v0.3.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-aws-terraform-module) - Component: | OpenEBS and NVMe instance storage to enable spill-to-disk Version: | - [v0.3.1+](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) + [v0.3.1+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-aws-terraform-module) - Component: | `cert-manager` and a self-signed `ClusterIssuer`. `ClusterIssuer` is deployed on subsequent runs after the `cert-manager` is running. Version: | - [v0.4.0+](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) + [v0.4.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-aws-terraform-module) diff --git a/doc/user/data/self_managed/azure_terraform_deployed_components.yml b/doc/user/data/self_managed/azure_terraform_deployed_components.yml index b293fe8fe0a30..f30a2382efc4d 100644 --- a/doc/user/data/self_managed/azure_terraform_deployed_components.yml +++ b/doc/user/data/self_managed/azure_terraform_deployed_components.yml @@ -23,12 +23,12 @@ rows: `ClusterIssuer` is deployed on subsequent runs after the `cert-manager` is running. Version: | - [v0.3.0+](/installation/appendix-terraforms/#materialize-on-azure-terraform-module) + [v0.3.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-azure-terraform-module) - Component: | Load balancers for each Materialize instance Version: | - [v0.3.1+](/installation/appendix-terraforms/#materialize-on-azure-terraform-module) + [v0.3.1+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-azure-terraform-module) - Component: | OpenEBS and NVMe instance storage to enable spill-to-disk Version: | - [v0.4.0+](/installation/appendix-terraforms/#materialize-on-azure-terraform-module) + [v0.4.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-azure-terraform-module) diff --git a/doc/user/data/self_managed/default_ports.yml b/doc/user/data/self_managed/default_ports.yml new file mode 100644 index 0000000000000..92b7ed134446e --- /dev/null +++ b/doc/user/data/self_managed/default_ports.yml @@ -0,0 +1,11 @@ +columns: + - column: "Port" + - column: "Description" + +rows: + - "Port": "6875" + "Description": "For SQL connections to the database" + - "Port": "6876" + "Description": "For HTTP(S) connections to the database" + - "Port": "8080" + "Description": "For HTTP(S) connections to Materialize Console" diff --git a/doc/user/data/self_managed/gcp_terraform_deployed_components.yml b/doc/user/data/self_managed/gcp_terraform_deployed_components.yml index a094584f31f06..1cacbd46ce7bb 100644 --- a/doc/user/data/self_managed/gcp_terraform_deployed_components.yml +++ b/doc/user/data/self_managed/gcp_terraform_deployed_components.yml @@ -22,16 +22,16 @@ rows: - Component: | Load balancers for each Materialize instance Version: | - [v0.3.0+](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) + [v0.3.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-gcp-terraform-module) - Component: | `cert-manager` and a self-signed `ClusterIssuer`. `ClusterIssuer` is deployed on subsequent runs after the `cert-manager` is running. Version: | - [v0.3.0+](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) + [v0.3.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-gcp-terraform-module) - Component: | OpenEBS and NVMe instance storage to enable spill-to-disk Version: | - [v0.4.0+](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) + [v0.4.0+](/self-managed-deployments/appendix/legacy/appendix-legacy-terraform-releases/#materialize-on-gcp-terraform-module) diff --git a/doc/user/data/self_managed/installation.yml b/doc/user/data/self_managed/installation.yml new file mode 100644 index 0000000000000..e440ee31146c8 --- /dev/null +++ b/doc/user/data/self_managed/installation.yml @@ -0,0 +1,76 @@ +- name: installation-landing-intro + content: | + You can install Self-Managed Materialize on a Kubernetes cluster running + locally or on a cloud provider. Self-Managed Materialize requires: + + {{% self-managed/materialize-components-list %}} + + ## License key + + Starting in v26.0, Materialize requires a license key. + + {{< yaml-table data="self_managed/license_key" >}} + + ## Installation guides + + The following installation guides are available to help you get started: + +- name: installation-landing-guides-helm + content: | + + ### Install using Helm Commands + + | Guide | Description | + | ------------- | -------| + | [Install locally on Kind](/self-managed-deployments/installation/install-on-local-kind/) | Uses standard Helm commands to deploy Materialize to a Kind cluster in Docker. + + +- name: installation-landing-guides-unified + content: | + ### Install using Unified Terraform Modules + + + | Guide | Description | + | ------------- | -------| + | [Install on AWS](/self-managed-deployments/installation/install-on-aws/) |Uses Terraform module to deploy Materialize to AWS Elastic Kubernetes Service (EKS). + | [Install on Azure](/self-managed-deployments/installation/install-on-azure/) | Uses Terraform module to deploy Materialize to Azure Kubernetes Service (AKS). + | [Install on GCP](/self-managed-deployments/installation/install-on-gcp/) | Uses Terraform module to deploy Materialize to Google Kubernetes Engine (GKE). + + +- name: installation-landing-guides-legacy + content: | + + ### Install using Legacy Terraform Modules + + | Guide | Description | + | ------------- | ----------- | + | [Install on AWS (Legacy Terraform)](/self-managed-deployments/installation/legacy/install-on-aws-legacy/) | Uses legacy Terraform module to deploy Materialize to AWS Elastic Kubernetes Service (EKS). + | [Install on Azure (Legacy Terraform)](/self-managed-deployments/installation/legacy/install-on-azure-legacy/) | Uses legacy Terraform module to deploy Materialize to Azure Kubernetes Service (AKS). + | [Install on GCP (Legacy Terraform)](/self-managed-deployments/installation/legacy/install-on-gcp-legacy/) | Uses legacy Terraform module to deploy Materialize to Google Kubernetes Engine (GKE). + +- name: install-uses-self-signed-cluster-issuer + content: | + The example uses a self-signed ClusterIssuer. As such, you may encounter a + warning with regards to the certificate. In production, run with + certificates from an official Certificate Authority (CA) rather than + self-signed certificates. + +- name: installation-verify-status + content: | + {{< tabs >}} + {{< tab "Operator" >}} + To check the status of the Materialize operator, which runs in the `materialize` namespace: + ```bash + kubectl -n materialize get all + ``` + {{< /tabs >}} + {{< tab "Materialize instance" >}} + To check the status of the Materialize instance, which runs in the `materialize-environment` namespace: + ```bash + kubectl -n materialize-environment get all + ``` + {{< /tab >}} + {{< /tabs >}} + + If you run into an error during deployment, refer to the + [Troubleshooting](/self-managed-deployments/troubleshooting/). diff --git a/doc/user/data/self_managed/terraform_list.yml b/doc/user/data/self_managed/terraform_list.yml index 8170eb218d06f..94f6f99e7fcf3 100644 --- a/doc/user/data/self_managed/terraform_list.yml +++ b/doc/user/data/self_managed/terraform_list.yml @@ -1,29 +1,22 @@ columns: - - column: "Sample Module" + - column: "Module" - column: "Description" rows: -- "Sample Module": | - [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) +- "Module": | + [Amazon Web Services (AWS)](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws) "Description": | - A sample Terraform module for installing the Materialize Helm chart into a Kubernetes cluster. + An example Terraform module for deploying Materialize on AWS. + See [Install on AWS](/self-managed-deployments/installation/install-on-aws/) for detailed instructions usage. -- "Sample Module": | - [Materialize on AWS](https://github.com/MaterializeInc/terraform-aws-materialize) +- "Module": | + [Azure](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure) "Description": | - A sample Terraform module for deploying Materialize on AWS Cloud Platform - with all required infrastructure components. - See [Install on AWS](/installation/install-on-aws/) for an example usage. + An example Terraform module for deploying Materialize on Azure. + See [Install on Azure](/self-managed-deployments/installation/install-on-azure/) for detailed instructions usage. -- "Sample Module": | - [Materialize on Azure](https://github.com/MaterializeInc/terraform-azurerm-materialize) +- "Module": | + [Google Cloud Platform (GCP)](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp) "Description": | - A sample Terraform module for deploying Materialize on Azure with all - required infrastructure components. See [Install on - Azure](/installation/install-on-azure/) for an example usage. - -- "Sample Module": | - [Materialize on Google Cloud Platform (GCP)](https://github.com/MaterializeInc/terraform-google-materialize) - "Description": | - A sample Terraform module for deploying Materialize on Google Cloud Platform - (GCP) with all required infrastructure components. See [Install on GCP](/installation/install-on-gcp/) for an example usage. + An example Terraform module for deploying Materialize on GCP. + See [Install on GCP](/self-managed-deployments/installation/install-on-gcp/) for detailed instructions usage. diff --git a/doc/user/data/self_managed/terraform_list_legacy.yml b/doc/user/data/self_managed/terraform_list_legacy.yml new file mode 100644 index 0000000000000..d430f5c42e867 --- /dev/null +++ b/doc/user/data/self_managed/terraform_list_legacy.yml @@ -0,0 +1,29 @@ +columns: + - column: "Sample Module" + - column: "Description" + +rows: +- "Sample Module": | + [terraform-helm-materialize (Legacy)](https://github.com/MaterializeInc/terraform-helm-materialize) + "Description": | + A sample Terraform module for installing the Materialize Helm chart into a Kubernetes cluster. + +- "Sample Module": | + [Materialize on AWS (Legacy)](https://github.com/MaterializeInc/terraform-aws-materialize) + "Description": | + A sample Terraform module for deploying Materialize on AWS Cloud Platform + with all required infrastructure components. + See [Install on AWS (Legacy)](/self-managed-deployments/installation/legacy/install-on-aws-legacy/) for an example usage. + +- "Sample Module": | + [Materialize on Azure (Legacy)](https://github.com/MaterializeInc/terraform-azurerm-materialize) + "Description": | + A sample Terraform module for deploying Materialize on Azure with all + required infrastructure components. See [Install on + Azure](/self-managed-deployments/installation/legacy/install-on-azure-legacy/) for an example usage. + +- "Sample Module": | + [Materialize on GCP (Legacy)](https://github.com/MaterializeInc/terraform-google-materialize) + "Description": | + A sample Terraform module for deploying Materialize on Google Cloud Platform + (GCP) with all required infrastructure components. See [Install on GCP](/self-managed-deployments/installation/legacy/install-on-gcp-legacy/) for an example usage. diff --git a/doc/user/data/self_managed/upgrades.yml b/doc/user/data/self_managed/upgrades.yml new file mode 100644 index 0000000000000..86cf90363eeb8 --- /dev/null +++ b/doc/user/data/self_managed/upgrades.yml @@ -0,0 +1,195 @@ +- name: upgrades-general-rules + content: | + When upgrading: + + - **Always** check the [version specific upgrade + notes](/self-managed-deployments/upgrading/#version-specific-upgrade-notes). + + - {{< include-from-yaml data="self_managed/upgrades" name="upgrade-order-rule" >}} + +- name: upgrade-order-rule + content: | + **Always** upgrade the Materialize Operator **before** + upgrading the Materialize instances. + +- name: syntax-helm-upgrade-operator + code: | + helm upgrade -n materialize/materialize-operator \ + --version \ + -f + syntax_elements: + - name: "``" + description: | + The namespace where the Operator is running. (e.g., `materialize`) + - name: "``" + description: | + The release name. You can use `helm list -n ` to find your release name. + - name: "``" + description: | + The upgrade version. + - name: "``" + description: | + The name of your customization file, if using. If you are configuring using `\-\-set key=value` options, include them as well. + +- name: upgrade-landing-guides-helm + content: | + + ### Upgrade using Helm Commands + + | Guide | Description | + | ------------- | -------| + | [Upgrade on Kind](/self-managed-deployments/upgrading/upgrade-on-kind/) | Uses standard Helm commands to upgrade Materialize on a Kind cluster in Docker. + + +- name: upgrade-landing-guides-unified + content: | + ### Upgrade using Unified Terraform Modules + + | Guide | Description | + | ------------- | ----------- | + | [Upgrade on AWS (Unified Terraform)](/self-managed-deployments/upgrading/upgrade-on-aws/) | Uses Unified Terraform module to deploy Materialize to AWS Elastic Kubernetes Service (EKS). + | [Upgrade on Azure (Unified Terraform)](/self-managed-deployments/upgrading/upgrade-on-azure/) | Uses Unified Terraform module to deploy Materialize to Azure Kubernetes Service (AKS). + | [Upgrade on GCP (Unified Terraform)](/self-managed-deployments/upgrading/upgrade-on-gcp/) | Uses Unified Terraform module to deploy Materialize to Google Kubernetes Engine (GKE). + +- name: upgrade-landing-guides-legacy + content: | + + ### Upgrade using Legacy Terraform Modules + + | Guide | Description | + | ------------- | ----------- | + | [Upgrade on AWS (Legacy Terraform)](/self-managed-deployments/upgrading/legacy/upgrade-on-aws-legacy/) | Uses legacy Terraform module to deploy Materialize to AWS Elastic Kubernetes Service (EKS). + | [Upgrade on Azure (Legacy Terraform)](/self-managed-deployments/upgrading/legacy/upgrade-on-azure-legacy/) | Uses legacy Terraform module to deploy Materialize to Azure Kubernetes Service (AKS). + | [Upgrade on GCP (Legacy Terraform)](/self-managed-deployments/upgrading/legacy/upgrade-on-gcp-legacy/) | Uses legacy Terraform module to deploy Materialize to Google Kubernetes Engine (GKE). + +- name: downgrade-restriction + content: | + Downgrading is not supported. + +- name: upgrade-major-version-restriction + content: | + For major version upgrades, you can **only** upgrade **one** major version + at a time. For example, upgrades from **v26**.1.0 to **v27**.3.0 is + permitted but **v26**.1.0 to **v28**.0.0 is not. + +- name: upgrade-update-helm-chart + content: | + To update your Materialize Helm Chart repository: + + 1. Update the Helm repo: + + ```shell + helm repo update materialize + ``` + + 1. View the available chart versions: + + ```shell + helm search repo materialize/materialize-operator --versions + ``` +- name: upgrade-materialize-operator + content: | + 1. Use `helm list` to find the release name. The sample example + deployment using the unified Terraform module deploys the Operator in the + `materialize` namespace. + + ```shell + helm list -n materialize + ``` + + Retrieve the release name (corresponds to the `name_prefix` variable + specified in your `terraform.tfvars`) associated with the + `materialize-operator` **CHART**; for example, `simple-demo` in the following output: + + ```none + NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION + simple-demo materialize 1 2025-12-08 11:39:50.185976 -0500 EST deployed materialize-operator-v26.1.0 v26.1.0 + ``` + + 1. Export your current values to a file to preserve any custom settings: + + ```shell + helm get values -n materialize simple-demo -o yaml > my-values.yaml + ``` + + + 1. Upgrade your Operator. For example, the following upgrades the Operator + to {{< self-managed/versions/get-latest-version >}}: + + {{< note >}} + {{% include-from-yaml data="self_managed/upgrades" + name="upgrade-major-version-restriction" %}} + {{< /note >}} + + ```shell + helm upgrade -n materialize simple-demo materialize/materialize-operator \ + -f my-values.yaml \ + --version {{< self-managed/versions/get-latest-version >}} + ``` + + 1. Verify that the Operator is running: + + ```bash + kubectl -n materialize get all + ``` + + 1. Get the **APP VERSION** of the Operator. + + ```shell + helm list -n materialize + ``` + + The **APP VERSION** will be the value that you will use for upgrading + Materialize instances. + +- name: upgrade-materialize-instance + content: | + + **After** you have upgraded your Materialize Operator, upgrade your + Materialize instance(s) to the **APP Version** of the Operator. When + upgrading Materialize Instances versions, changes are not automatically + rolled out by the Operator in order to minimize unexpected downtime and + avoid connection drops at critical periods. Instead, the upgrade process + involves two steps: + + - First, staging the version change to the Materialize custom resource. + - Second, rolling out the changes via a `requestRollout` flag. + + 1. Find the name of the Materialize instance to upgrade. The sample example + deployment using the unified Terraform module deploys the Materialie + instance in the`materialize-environment` namespace. + + ```shell + kubectl get materialize -n materialize-environment + ``` + + In the example deployment, the name of the instance is `main`. + + ```none + NAME + main + ``` + + 1. Stage, but not rollout, the Materialize instance version upgrade. + + ```shell + kubectl patch materialize main\ + -n materialize-environment \ + --type='merge' \ + -p "{\"spec\": {\"environmentdImageRef\": \"docker.io/materialize/environmentd:{{< self-managed/versions/get-latest-version >}}\"}}" + ``` + + 1. Rollout the Materialize instance version change. + + ```shell + kubectl patch materialize main \ + -n materialize-environment \ + --type='merge' \ + -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\"}}" + ``` + + 1. Verify the upgrade by checking the `environmentd` events: + + ```bash + kubectl -n materialize-environment describe pod -l app=environmentd + ``` diff --git a/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html b/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html index 82e7dc968bf68..811158c414722 100644 --- a/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html +++ b/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html @@ -1,9 +1,9 @@ The tutorial uses the `main.tf` found in the `examples/simple/` directory, which -requires minimal user input. For details on the `examples/simple/` +requires minimal user input. For details on the `examples/simple/` infrastructure configuration (such as the node instance type, etc.), see the [examples/simple/main.tf](https://github.com/MaterializeInc/terraform-aws-materialize/blob/main/examples/simple/main.tf). For more configuration options, you can use the `main.tf` file at the [root of the repository](https://github.com/MaterializeInc/terraform-aws-materialize/) instead. When running with the root `main.tf`, see [AWS required -configuration](/installation/install-on-aws/appendix-aws-configuration/). +configuration](/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-aws/). diff --git a/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html b/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html index f87f789efa914..cd042d70bfe90 100644 --- a/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html +++ b/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html @@ -7,4 +7,4 @@ the repository](https://github.com/MaterializeInc/terraform-azurerm-materialize/) instead. When running with the root `main.tf`, see [Azure required -configuration](/installation/install-on-azure/appendix-azure-configuration/). +configuration](/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-azure/). diff --git a/doc/user/layouts/shortcodes/self-managed/gcp-terraform-configs.html b/doc/user/layouts/shortcodes/self-managed/gcp-terraform-configs.html index c60a45ed552e1..54cb717e27239 100644 --- a/doc/user/layouts/shortcodes/self-managed/gcp-terraform-configs.html +++ b/doc/user/layouts/shortcodes/self-managed/gcp-terraform-configs.html @@ -6,4 +6,4 @@ For more configuration options, you can use the `main.tf` file at the [root of the repository](https://github.com/MaterializeInc/terraform-google-materialize/) instead. When running with the root `main.tf`, see [GCP required -configuration](/installation/install-on-gcp/appendix-gcp-configuration/). +configuration](/self-managed-deployments/appendix/legacy/appendix-configuration-legacy-gcp/). diff --git a/doc/user/layouts/shortcodes/self-managed/next-steps.html b/doc/user/layouts/shortcodes/self-managed/next-steps.html index 11312eb350be8..73e25e0ca7a49 100644 --- a/doc/user/layouts/shortcodes/self-managed/next-steps.html +++ b/doc/user/layouts/shortcodes/self-managed/next-steps.html @@ -3,4 +3,4 @@ [Quickstart](/get-started/quickstart/). - To start ingesting your own data from an external system like Kafka, MySQL or - PostgreSQL, check the documentation for [sources](/sql/create-source/). + PostgreSQL, see [Ingest data](/ingest-data/). diff --git a/doc/user/layouts/shortcodes/self-managed/troubleshoot-console-mz_catalog_server_blurb.md b/doc/user/layouts/shortcodes/self-managed/troubleshoot-console-mz_catalog_server_blurb.md index ebd012ee58fdf..6e3c8431e46e8 100644 --- a/doc/user/layouts/shortcodes/self-managed/troubleshoot-console-mz_catalog_server_blurb.md +++ b/doc/user/layouts/shortcodes/self-managed/troubleshoot-console-mz_catalog_server_blurb.md @@ -1,5 +1,5 @@ If you experience long loading screens or unresponsiveness in the Materialize Console, we recommend increasing the size of the `mz_catalog_server` cluster. Refer to the [Troubleshooting Console -Unresponsiveness](/installation/troubleshooting/#troubleshooting-console-unresponsiveness) +Unresponsiveness](/self-managed-deployments/troubleshooting/#troubleshooting-console-unresponsiveness) guide. diff --git a/doc/user/layouts/shortcodes/self-managed/versions/get-latest-version.html b/doc/user/layouts/shortcodes/self-managed/versions/get-latest-version.html new file mode 100644 index 0000000000000..b8cdd679e4bde --- /dev/null +++ b/doc/user/layouts/shortcodes/self-managed/versions/get-latest-version.html @@ -0,0 +1,3 @@ +{{ $environmentd_version := +site.Data.self_managed.latest_versions.environmentd_version }} +{{- $environmentd_version -}} diff --git a/doc/user/layouts/shortcodes/self-managed/versions/step-install-helm-version-local-minikube-install.html b/doc/user/layouts/shortcodes/self-managed/versions/step-install-helm-version-local-minikube-install.html deleted file mode 100644 index ff81308fcdebe..0000000000000 --- a/doc/user/layouts/shortcodes/self-managed/versions/step-install-helm-version-local-minikube-install.html +++ /dev/null @@ -1,13 +0,0 @@ -{{ $operator_version := site.Data.self_managed.latest_versions.operator_helm_chart_version }} - -1. Install the Materialize Operator. The operator will be installed in the -`materialize` namespace. - - ```shell - helm install my-materialize-operator materialize/materialize-operator \ - --namespace=materialize --create-namespace \ - --version {{ $operator_version }} \ - --set operator.cloudProvider.region=minikube \ - --set observability.podMetrics.enabled=true \ - -f sample-values.yaml - ``` diff --git a/doc/user/layouts/shortcodes/self-managed/versions/upgrade/upgrade-steps-local-kind.html b/doc/user/layouts/shortcodes/self-managed/versions/upgrade/upgrade-steps-local-kind.html index 83507aebdfcaa..94ee296c4aac3 100644 --- a/doc/user/layouts/shortcodes/self-managed/versions/upgrade/upgrade-steps-local-kind.html +++ b/doc/user/layouts/shortcodes/self-managed/versions/upgrade/upgrade-steps-local-kind.html @@ -61,13 +61,13 @@ kubectl -n materialize-environment patch secret materialize-backend -p '{"stringData":{"license_key":""}}' --type=merge ``` -1. Create a new `upgrade-materialize.yaml` file with the following content: +1. Create a new `upgrade-materialize.yaml` file, updating the following fields: | Field | Description | |-------|-------------| | `environmentdImageRef` | Update the version to the new version. This should be the same as the operator version: `{{ $operator_version }}`. | | `requestRollout` or `forceRollout`| Enter a new UUID. Can be generated with `uuidgen`.
  • `requestRollout` triggers a rollout only if changes exist.
  • `forceRollout` triggers a rollout even if no changes exist.
| - | `inPlaceRollout` | Set to `false` to perform a rolling upgrade. For rolling upgrades, ensure you have enough resources to support having both the old and new Materialize instances running during the upgrade. | + ```yaml apiVersion: materialize.cloud/v1alpha1 @@ -78,13 +78,10 @@ spec: environmentdImageRef: materialize/environmentd:{{ $environmentd_version }} # Update version requestRollout: 22222222-2222-2222-2222-222222222222 # Enter a new UUID - # forceRollout: 33333333-3333-3333-3333-333333333333 # For forced rollouts - inPlaceRollout: false # When false, performs a rolling upgrade rather than in-place + # forceRollout: 33333333-3333-3333-3333-333333333333 # For forced rollouts + rolloutStrategy: WaitUntilReady # The mechanism to use when rolling out the new version. backendSecretName: materialize-backend ``` -
- WARNING! Please consult the Materialize team before setting inPlaceRollout to true and performing an in-place rollout. In almost all cases a rolling upgrade is preferred. -
1. Apply the upgrade-materialize.yaml file to your Materialize instance: diff --git a/doc/user/shared-content/rbac-sm/enable-rbac.md b/doc/user/shared-content/rbac-sm/enable-rbac.md index 44afc438b1930..b9160b76575f9 100644 --- a/doc/user/shared-content/rbac-sm/enable-rbac.md +++ b/doc/user/shared-content/rbac-sm/enable-rbac.md @@ -8,12 +8,13 @@ enable RBAC, set the system parameter `enable_rbac_checks` to `'on'` or `True`. You can enable the parameter in one of the following ways: - For [local installations using - Kind/Minikube](/installation/#installation-guides), set `spec.enableRbac: + Kind/Minikube](/self-managed-deployments/installation/#installation-guides), set `spec.enableRbac: true` option when instantiating the Materialize object. - For [Cloud deployments using Materialize's - Terraforms](/installation/#installation-guides), set `enable_rbac_checks` in - the environment CR via the `environmentdExtraArgs` flag option. + Terraforms](/self-managed-deployments/installation/#installation-guides), set + `enable_rbac_checks` in the environment CR via the `environmentdExtraArgs` + flag option. - After the Materialize instance is running, run the following command as `mz_system` user: diff --git a/doc/user/shared-content/self-managed/general-rules-for-upgrades.md b/doc/user/shared-content/self-managed/general-rules-for-upgrades.md index bf9b29bbdb4d0..86f661ddcde23 100644 --- a/doc/user/shared-content/self-managed/general-rules-for-upgrades.md +++ b/doc/user/shared-content/self-managed/general-rules-for-upgrades.md @@ -1,10 +1,10 @@ Whe upgrading: -- **Always** upgrade the operator first and ensure version compatibility between - the operator and the Materialize instance you are upgrading to. +- **Always** check the [version specific upgrade + notes](/self-managed-deployments/upgrading/#version-specific-upgrade-notes). -- **Always** upgrade your Materialize instances after upgrading the operator to - ensure compatibility. +- **Always** upgrade the operator **first** and ensure version compatibility + between the operator and the Materialize instance you are upgrading to. -- **Always check** the [version specific upgrade - notes](/installation/#version-specific-upgrade-notes). +- **Always** upgrade your Materialize instances **after** upgrading the operator + to ensure compatibility. diff --git a/doc/user/shared-content/self-managed/install-landing-page.md b/doc/user/shared-content/self-managed/install-landing-page.md deleted file mode 100644 index decb48f54fd2d..0000000000000 --- a/doc/user/shared-content/self-managed/install-landing-page.md +++ /dev/null @@ -1,31 +0,0 @@ -You can install Self-Managed Materialize on a Kubernetes cluster running locally -or on a cloud provider. Self-Managed Materialize requires: - -{{% self-managed/materialize-components-list %}} - -## License key - -Starting in v26.0, Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} - -## Install - -### Installation guides - -The following installation guides are available: - -| | Notes | -| ------------- | -------| -| [Install locally on kind](/installation/install-on-local-kind/) | -| [Deploy Materialize to AWS Elastic Kubernetes Service (EKS)](/installation/install-on-aws/) | Uses Materialize provided Terraform | -| [Deploy Materialize to Azure Kubernetes Service (AKS)](/installation/install-on-azure/) | Uses Materialize provided Terraform | -| [Deploy Materialize to Google Kubernetes Engine (GKE)](/installation/install-on-gcp/) | Uses Materialize provided Terraform | - -See also: -- [AWS Deployment - guidelines](/installation/install-on-aws/appendix-deployment-guidelines/#recommended-instance-types) -- [GCP Deployment - guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/#recommended-instance-types) -- [Azure Deployment - guidelines](/installation/install-on-azure/appendix-deployment-guidelines/#recommended-instance-types) diff --git a/doc/user/shared-content/self-managed/prepare-nodes-and-upgrade.md b/doc/user/shared-content/self-managed/prepare-nodes-and-upgrade.md index 555c283f3a6a5..b51a9ec66812f 100644 --- a/doc/user/shared-content/self-managed/prepare-nodes-and-upgrade.md +++ b/doc/user/shared-content/self-managed/prepare-nodes-and-upgrade.md @@ -15,7 +15,7 @@ Materialize-provided Terraforms. {{< include-md file="shared-content/self-managed/general-rules-for-upgrades.md" >}} -See also [General notes for upgrades](/installation/#general-notes-for-upgrades) +See also [General notes for upgrades](/self-managed-deployments/upgrading/) {{< /tip >}} 1. Label existing scratchfs/lgalloc node groups. diff --git a/doc/user/shared-content/self-managed/upgrade-notes/v26.0.md b/doc/user/shared-content/self-managed/upgrade-notes/v26.0.md index 3d6dafa40feb1..a59c43a810a08 100644 --- a/doc/user/shared-content/self-managed/upgrade-notes/v26.0.md +++ b/doc/user/shared-content/self-managed/upgrade-notes/v26.0.md @@ -8,7 +8,7 @@ - `ImmediatelyPromoteCausingDowntime` For more information, see - [`rolloutStrategy`](/installation/#rollout-strategies). + [`rolloutStrategy`](/self-managed-deployments/upgrading/#rollout-strategies). - New requirements were introduced for [license keys](/releases/#license-key). To upgrade, you will first need to add a license key to the `backendSecret` @@ -26,4 +26,4 @@ `v0.6.1` of the Terraform. - If you are **not** using a Materialize-provided Terraform, refer - to [Prepare for swap and upgrade to v26.0](/installation/upgrade-to-swap/). + to [Prepare for swap and upgrade to v26.0](/self-managed-deployments/appendix/upgrade-to-swap/). diff --git a/misc/helm-charts/operator/README.md b/misc/helm-charts/operator/README.md index 3dd939312b913..995140fe62e00 100644 --- a/misc/helm-charts/operator/README.md +++ b/misc/helm-charts/operator/README.md @@ -406,12 +406,6 @@ Materialize has been vetted to work on instances with the following properties: When operating in AWS, we recommend using the `r7gd` and `r6gd` families of instances (and `r8gd` once available) when running with local disk, and the `r8g`, `r7g`, and `r6g` families when running without local disk. -## CPU Affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - ## Learn More - [Materialize Documentation](https://materialize.com/docs) diff --git a/misc/helm-charts/operator/README.md.gotmpl b/misc/helm-charts/operator/README.md.gotmpl index a69d25b3ab480..7b5af4ad70f89 100644 --- a/misc/helm-charts/operator/README.md.gotmpl +++ b/misc/helm-charts/operator/README.md.gotmpl @@ -347,12 +347,6 @@ Materialize has been vetted to work on instances with the following properties: When operating in AWS, we recommend using the `r7gd` and `r6gd` families of instances (and `r8gd` once available) when running with local disk, and the `r8g`, `r7g`, and `r6g` families when running without local disk. -## CPU Affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - ## Learn More - [Materialize Documentation](https://materialize.com/docs)