MINOR: Merge pull request #14 from Datatamer/DEV-14999-update-dataproc-config

DEV-14999 remove initialization script from dataproc workflow template

Author: daniosim

CHANGELOG.md

@@ -1,17 +1,23 @@
-# Tamr Terraform Template Repo - v0.3.1 - August 4nd 2021
+# GCP Tamr VM module
+
+## v0.4.1 - August 31st 2021
+* Updates dataproc workflow template to remove use of public initialization script
+* Bumps dataproc version to 1.4
+
+## v0.3.1 - August 4nd 2021
 * Correct content of shutdown-script resource
 
-# Tamr Terraform Template Repo - v0.3.0 - August 2nd 2020
+## v0.3.0 - August 2nd 2020
 * Added support for external IPs
 * Added support for deletion protection
 
-# Tamr Terraform Template Repo - v0.2.1 - August 6th 2020
+## v0.2.1 - August 6th 2020
 * Add zone/region flags into dataproc workflow template
 
-# Tamr Terraform Template Repo - v0.2.0 - August 4th 2020
+## v0.2.0 - August 4th 2020
 * Upgrade to spark 2.4
 * Pin GCS connector version
 * Use default spark event log directory
 
-# Tamr Terraform Template Repo - v0.1.0 - July 28th 2020
+## v0.1.0 - July 28th 2020
 * Initing project

README.md

@@ -89,7 +89,7 @@ This modules creates:
 | tamr\_dataproc\_cluster\_master\_instance\_type | Instance type to use as dataproc master This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"n1-highmem-4"` | no |
 | tamr\_dataproc\_cluster\_service\_account | Service account to attach to dataproc workers. If not set will use the tamr\_instance\_service\_account as the default value. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `""` | no |
 | tamr\_dataproc\_cluster\_subnetwork\_uri | Subnetwork URI for dataproc to use. If not set will use the tamr\_instance\_subnet as the default value. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `""` | no |
-| tamr\_dataproc\_cluster\_version | Version of dataproc to use. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"1.3"` | no |
+| tamr\_dataproc\_cluster\_version | Version of dataproc to use. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"1.4"` | no |
 | tamr\_dataproc\_cluster\_worker\_machine\_type | machine type of default worker pool. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"n1-standard-16"` | no |
 | tamr\_dataproc\_cluster\_worker\_num\_instances | Number of default workers to use. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `4` | no |
 | tamr\_dataproc\_cluster\_worker\_num\_local\_ssds | Number of localssds to attach to each worker node. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `2` | no |

VERSION

@@ -1 +1 @@
-0.1.0
+0.4.1

examples/basic/main.tf

@@ -25,7 +25,7 @@ module "sample" {
   tamr_cloud_sql_location = local.default_region
   tamr_cloud_sql_name     = "tamr-db"
   tamr_sql_user           = "tamr"
-  tamr_sql_password       = "super_secure_password"
+  tamr_sql_password       = "super_secure_password" # tfsec:ignore:GEN003
   # filesystem
   tamr_filesystem_bucket = "tamr_application_home"
 }

templates/dataproc.yaml.tmpl

@@ -5,11 +5,7 @@ gceClusterConfig:
   internalIpOnly: true
   serviceAccount: ${service_account}
   metadata:
-    gcs-connector-version: 2.1.4
-initializationActions:
-  # see https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/connectors
-  - executableFile: gs://goog-dataproc-initialization-actions-${region}/connectors/connectors.sh
-    executionTimeout: 500s
+    GCS_CONNECTOR_VERSION: 2.2.0
 softwareConfig:
   imageVersion: 1.4
   properties:

variables.tf

@@ -173,7 +173,7 @@ variable "tamr_dataproc_cluster_enable_stackdriver_logging" {
 }
 
 variable "tamr_dataproc_cluster_version" {
-  default     = "1.3"
+  default     = "1.4"
   type        = string
   description = "Version of dataproc to use. This only used if using the built in tamr_dataproc_cluster_config configuration"
 }