MAJOR: DEV-15132 integrate with config module (#19)

Author: Tamr-Frank

CHANGELOG.md

@@ -1,5 +1,8 @@
 # GCP Tamr VM module
 
+## v2.0.0 - July 18th 2022
+* Remove overlap with config module
+
 ## v1.0.0 - June 1st 2022
 * Set minimum terraform to 1.0.0 and minimum google provider to 4.6.0
 

README.md

@@ -64,77 +64,33 @@ This modules creates:
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| tamr\_bigtable\_cluster\_id | Bigtable cluster ID | `string` | n/a | yes |
-| tamr\_bigtable\_instance\_id | Bigtable instance ID | `string` | n/a | yes |
-| tamr\_bigtable\_max\_nodes | Max number of nodes to scale up to | `string` | n/a | yes |
-| tamr\_bigtable\_min\_nodes | Min number of nodes to scale down to | `string` | n/a | yes |
-| tamr\_cloud\_sql\_location | location for cloud sql instance. NOTE: this is either a region or a zone. | `string` | n/a | yes |
-| tamr\_cloud\_sql\_name | name of cloud sql instance | `string` | n/a | yes |
-| tamr\_dataproc\_bucket | GCS bucket to use for the tamr dataproc cluster | `string` | n/a | yes |
-| tamr\_dataproc\_region | Region the dataproc uses. | `string` | n/a | yes |
+| tamr\_config\_file | full tamr config file | `string` | n/a | yes |
 | tamr\_filesystem\_bucket | GCS bucket to use for the tamr default file system | `string` | n/a | yes |
 | tamr\_instance\_image | Image to use for boot disk | `string` | n/a | yes |
 | tamr\_instance\_project | The project to launch the tamr VM instance in. | `string` | n/a | yes |
 | tamr\_instance\_service\_account | email of service account to attach to the tamr instance | `string` | n/a | yes |
 | tamr\_instance\_subnet | subnetwork to attach instance too | `string` | n/a | yes |
 | tamr\_instance\_zone | zone to deploy tamr vm | `string` | n/a | yes |
-| tamr\_sql\_password | password for the cloud sql user | `string` | n/a | yes |
 | tamr\_zip\_uri | gcs location to download tamr zip from | `string` | n/a | yes |
 | labels | labels to attach to created resources | `map(string)` | `{}` | no |
-| tamr\_bigtable\_project\_id | The google project that the bigtable instance lives in. If not set will use the tamr\_instance\_project as the default value. | `string` | `""` | no |
-| tamr\_cloud\_sql\_project | project containing cloudsql instance. If not set will use the tamr\_instance\_project as the default value. | `string` | `""` | no |
-| tamr\_config | Override generated tamr configuration. The tamr configuration is specified using a yaml file, in the format that is documented (https://docs.tamr.com/previous/docs/configuration-configuring-unify#section-setting-configuration-variables) for configuring “many variables” at once. | `string` | `""` | no |
-| tamr\_dataproc\_cluster\_config | If you do not want to use the default dataproc configuration template, pass in a complete dataproc configuration file to variable.<br>If you are passing in a dataproc configure it should not be left padded, we will handle that inside of our template. It is expected to<br>a yaml document of a dataproc cluster config<br>Refrence spec is https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig | `string` | `""` | no |
-| tamr\_dataproc\_cluster\_enable\_stackdriver\_logging | Enabled stackdriver logging on dataproc clusters. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `bool` | `true` | no |
-| tamr\_dataproc\_cluster\_master\_disk\_size | Size of disk to use on dataproc master disk This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `1000` | no |
-| tamr\_dataproc\_cluster\_master\_instance\_type | Instance type to use as dataproc master This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"n1-highmem-4"` | no |
-| tamr\_dataproc\_cluster\_service\_account | Service account to attach to dataproc workers. If not set will use the tamr\_instance\_service\_account as the default value. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `""` | no |
-| tamr\_dataproc\_cluster\_subnetwork\_uri | Subnetwork URI for dataproc to use. If not set will use the tamr\_instance\_subnet as the default value. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `""` | no |
-| tamr\_dataproc\_cluster\_version | Version of dataproc to use. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"1.4"` | no |
-| tamr\_dataproc\_cluster\_worker\_machine\_type | machine type of default worker pool. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"n1-standard-16"` | no |
-| tamr\_dataproc\_cluster\_worker\_num\_instances | Number of default workers to use. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `4` | no |
-| tamr\_dataproc\_cluster\_worker\_num\_local\_ssds | Number of localssds to attach to each worker node. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `2` | no |
-| tamr\_dataproc\_cluster\_worker\_preemptible\_machine\_type | machine type of preemptible worker pool. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `"n1-standard-16"` | no |
-| tamr\_dataproc\_cluster\_worker\_preemptible\_num\_instances | Number of preemptible workers to use. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `0` | no |
-| tamr\_dataproc\_cluster\_worker\_preemptible\_num\_local\_ssds | Number of localssds to attach to each preemptible worker node. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `number` | `2` | no |
-| tamr\_dataproc\_cluster\_zone | Zone to launch dataproc cluster into. If not set will use the tamr\_instance\_zone as the default value. This only used if using the built in tamr\_dataproc\_cluster\_config configuration | `string` | `""` | no |
-| tamr\_dataproc\_project\_id | Project for the dataproc cluster. If not set will use the tamr\_instance\_project as the default value. | `string` | `""` | no |
-| tamr\_es\_apihost | The hostname and port of the REST API endpoint of the Elasticsearch cluster to use. If unset will use < ip of vm>:9200 | `string` | `""` | no |
-| tamr\_es\_enabled | Whether Tamr will index user data in Elasticsearch or not.  Elasticsearch is used to power Tamr's interactive data UI, so when this is set to false Tamr will run 'headless,' that is, without its core UI capabilities.  It can be useful to disable Elasticsearch in production settings where the models are trained on a separate instance and the goal is to maximize pipeline throughput. | `bool` | `true` | no |
-| tamr\_es\_number\_of\_shards | The number of shards to set when creating the Tamr index in Elasticsearch.  Default value is the number of cores on the local host machine, so this should be overridden when using a remote Elasticsearch cluster.  Note: this value is only applied when the index is created. | `number` | `1` | no |
-| tamr\_es\_password | Password to use to authenticate to Elasticsearch, using basic authentication.  Not required unless the Elasticsearch cluster you're using has security and authentication enabled.  The value passed in may be encrypted. | `string` | `""` | no |
-| tamr\_es\_socket\_timeout | Defines the socket timeout for Elasticsearch clients, in milliseconds. This is the timeout for waiting for data or, put differently, a maximum period of inactivity between two consecutive data packets. A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default).  The default value is 900000, i.e., fifteen minutes. | `number` | `900000` | no |
-| tamr\_es\_ssl\_enabled | Whether to connect to Elasticsearch over https or not.  Default is false (http). | `bool` | `false` | no |
-| tamr\_es\_user | Username to use to authenticate to Elasticsearch.  Not required unless the Elasticsearch cluster you're using has security and authentication enabled. | `string` | `""` | no |
 | tamr\_external\_ip | Create and attach an external ip to tamr VM | `bool` | `false` | no |
-| tamr\_hbase\_namespace | HBase namespace to user, for bigtable this will be the table prefix. | `string` | `"ns0"` | no |
 | tamr\_instance\_deletion\_protection | Enabled deletion protection for the tamr VM | `bool` | `true` | no |
 | tamr\_instance\_disk\_size | size of the boot disk | `number` | `100` | no |
 | tamr\_instance\_disk\_type | boot disk type | `string` | `"pd-ssd"` | no |
 | tamr\_instance\_install\_directory | directory to install tamr into | `string` | `"/data/tamr"` | no |
 | tamr\_instance\_machine\_type | machine type to use for tamr vm | `string` | `"n1-highmem-8"` | no |
 | tamr\_instance\_name | Name of the VM running tamr | `string` | `"tamr"` | no |
 | tamr\_instance\_tags | list of network tags to attach to instance | `list(string)` | `[]` | no |
-| tamr\_json\_logging | Toggle json formatting for tamr logs. | `bool` | `false` | no |
-| tamr\_license\_key | Set a tamr license key | `string` | `""` | no |
-| tamr\_spark\_driver\_memory | Amount of memory spark should allocate to spark driver | `string` | `"12G"` | no |
-| tamr\_spark\_executor\_cores | Amount of cores spark should allocate to each spark executor | `number` | `5` | no |
-| tamr\_spark\_executor\_instances | number of spark executor instances | `number` | `12` | no |
-| tamr\_spark\_executor\_memory | Amount of memory spark should allocate to each spark executor | `string` | `"13G"` | no |
-| tamr\_spark\_properties\_override | json blob of spark properties to override, if not set will use a default set of properties that should work for most use cases | `string` | `""` | no |
-| tamr\_sql\_user | username for the cloud sql user | `string` | `"tamr"` | no |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
-| tamr\_config\_file | full tamr config file |
 | tamr\_instance\_internal\_ip | internal ip of tamr vm |
 | tamr\_instance\_name | name of the tamr vm |
 | tamr\_instance\_self\_link | full self link of created tamr vm |
 | tamr\_instance\_zone | zone of the tamr vm |
-| tmpl\_dataproc\_config | dataproc config |
-| tmpl\_statup\_script | rendered metadata startup script |
+| tmpl\_startup\_script | rendered metadata startup script |
 
 <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 

VERSION

@@ -1 +1 @@
-0.4.1
+2.0.0

examples/basic/config.yaml

@@ -0,0 +1,2 @@
+# currently single node
+TAMR_LICENSE_KEY: placeholder

examples/basic/main.tf

@@ -1,35 +1,23 @@
-locals {
-  gcp_project    = "tamr-deployment"
-  default_region = "us-east1"
-  default_zone   = "us-east1-b"
+data "google_compute_subnetwork" "project_subnet" {
+  name    = var.subnet_name
+  region  = var.region_id
+  project = "tamr-networking"
 }
 
 module "sample" {
   source = "../../"
-  # bigtable config
-  tamr_bigtable_project_id  = local.gcp_project
-  tamr_bigtable_instance_id = "tamr-bigtable-instance"
-  tamr_bigtable_cluster_id  = "TAMR_BIGTABLE_CLUSTER_ID"
-  tamr_bigtable_min_nodes   = 1
-  tamr_bigtable_max_nodes   = 10
-  # dataproc
-  tamr_dataproc_project_id = local.gcp_project
-  tamr_dataproc_bucket     = "tamr_dataproc_home"
-  tamr_dataproc_region     = local.default_region
-  # dataproc_cluster_config
-  tamr_dataproc_cluster_subnetwork_uri  = "projects/${local.gcp_project}/regions/${local.default_region}/subnetworks/default"
-  tamr_dataproc_cluster_service_account = "tamr-instance@${local.gcp_project}.iam.gserviceaccount.com"
-  tamr_dataproc_cluster_zone            = local.default_zone
-  # cloud sql
-  tamr_cloud_sql_project  = local.gcp_project
-  tamr_cloud_sql_location = local.default_region
-  tamr_cloud_sql_name     = "tamr-db"
-  tamr_sql_user           = "tamr"
-  tamr_sql_password       = "super_secure_password" # tfsec:ignore:GEN003
+
+  tamr_instance_project         = var.project_id
+  tamr_instance_name            = var.instance_id
+  tamr_instance_zone            = var.zone_id
+  tamr_instance_image           = var.instance_image
+  tamr_instance_subnet          = replace(data.google_compute_subnetwork.project_subnet.self_link, "https://www.googleapis.com/compute/v1/", "")
+  tamr_zip_uri                  = var.zip_url
+  tamr_instance_disk_size       = 600
+  tamr_instance_service_account = var.service_account
   # filesystem
-  tamr_filesystem_bucket = "tamr_application_home"
-}
+  tamr_filesystem_bucket = var.filesystem_bucket
 
-output "tamr_config" {
-  value = module.sample.tamr_config_file
+  tamr_config_file = file("config.yaml")
 }
+

examples/basic/variables.tf

@@ -0,0 +1,41 @@
+variable "project_id" {
+  type = string
+}
+
+variable "instance_id" {
+  default = "tamr-vm-example"
+  type    = string
+}
+
+variable "region_id" {
+  default = "us-east1"
+  type    = string
+}
+
+variable "zone_id" {
+  default = "us-east1-b"
+  type    = string
+}
+
+variable "service_account" {
+  default = ""
+  type    = string
+}
+
+variable "filesystem_bucket" {
+  type = string
+}
+
+variable "instance_image" {
+  default = "tamr-private-images/bionic-base-1644877703"
+  type    = string
+}
+
+variable "zip_url" {
+  default = "gs://tamr-releases/v2022.005.0/unify.zip"
+  type    = string
+}
+
+variable "subnet_name" {
+  type = string
+}

main.tf

@@ -1,89 +1,9 @@
 locals {
-  tamr_dataproc_cluster_zone            = var.tamr_dataproc_cluster_zone == "" ? var.tamr_instance_zone : var.tamr_dataproc_cluster_zone
-  tamr_dataproc_cluster_subnetwork_uri  = var.tamr_dataproc_cluster_subnetwork_uri == "" ? var.tamr_instance_subnet : var.tamr_dataproc_cluster_subnetwork_uri
-  tamr_dataproc_cluster_service_account = var.tamr_dataproc_cluster_service_account == "" ? var.tamr_instance_service_account : var.tamr_dataproc_cluster_service_account
-
-  tamr_es_apihost = var.tamr_es_apihost == "" ? "${google_compute_address.tamr_ip.address}:9200" : var.tamr_es_apihost
-  remote_es       = var.tamr_es_apihost == "" ? false : true
-
-  tamr_bigtable_project_id = var.tamr_bigtable_project_id == "" ? var.tamr_instance_project : var.tamr_bigtable_project_id
-  tamr_cloud_sql_project   = var.tamr_cloud_sql_project == "" ? var.tamr_instance_project : var.tamr_cloud_sql_project
-  tamr_dataproc_project_id = var.tamr_dataproc_project_id == "" ? var.tamr_instance_project : var.tamr_dataproc_project_id
-
-  dataproc_config  = var.tamr_dataproc_cluster_config == "" ? local.default_dataproc : var.tamr_dataproc_cluster_config
-  tamr_config      = var.tamr_config == "" ? local.default_tamr_config : var.tamr_config
-  external_ip      = var.tamr_external_ip == true ? 1 : 0
-  spark_properties = var.tamr_spark_properties_override == "" ? file("${path.module}/files/spark_properties.json") : var.tamr_spark_properties_override
-
-  default_dataproc = templatefile("${path.module}/templates/dataproc.yaml.tmpl", {
-    subnetwork_uri       = local.tamr_dataproc_cluster_subnetwork_uri
-    service_account      = local.tamr_dataproc_cluster_service_account
-    zone                 = local.tamr_dataproc_cluster_zone
-    region               = var.tamr_dataproc_region
-    stackdriver_logging  = var.tamr_dataproc_cluster_enable_stackdriver_logging
-    version              = var.tamr_dataproc_cluster_version
-    tamr_dataproc_bucket = var.tamr_dataproc_bucket
-
-    master_instance_type = var.tamr_dataproc_cluster_master_instance_type
-    master_disk_size     = var.tamr_dataproc_cluster_master_disk_size
-
-    worker_machine_type   = var.tamr_dataproc_cluster_worker_machine_type
-    worker_num_instances  = var.tamr_dataproc_cluster_worker_num_instances
-    worker_num_local_ssds = var.tamr_dataproc_cluster_worker_num_local_ssds
-
-    worker_preemptible_machine_type   = var.tamr_dataproc_cluster_worker_preemptible_machine_type
-    worker_preemptible_num_instances  = var.tamr_dataproc_cluster_worker_preemptible_num_instances
-    worker_preemptible_num_local_ssds = var.tamr_dataproc_cluster_worker_preemptible_num_local_ssds
-  })
-
-
-  default_tamr_config = templatefile("${path.module}/templates/tamr_config.yaml.tmpl", {
-    # Bigtable
-    tamr_hbase_namespace      = var.tamr_hbase_namespace
-    tamr_bigtable_project_id  = local.tamr_bigtable_project_id
-    tamr_bigtable_instance_id = var.tamr_bigtable_instance_id
-    tamr_bigtable_cluster_id  = var.tamr_bigtable_cluster_id
-    tamr_bigtable_min_nodes   = var.tamr_bigtable_min_nodes
-    tamr_bigtable_max_nodes   = var.tamr_bigtable_max_nodes
-    # dataproc
-    tamr_dataproc_project_id = local.tamr_dataproc_project_id
-    tamr_dataproc_region     = var.tamr_dataproc_region
-    # NOTE: indent does not indent the first line of a variable, so we prefix it
-    # with a new file
-    tamr_dataproc_cluster_config = indent(2, "\n${local.dataproc_config}")
-    tamr_dataproc_bucket         = var.tamr_dataproc_bucket
-    # spark
-    tamr_spark_driver_memory      = var.tamr_spark_driver_memory
-    tamr_spark_executor_memory    = var.tamr_spark_executor_memory
-    tamr_spark_executor_cores     = var.tamr_spark_executor_cores
-    tamr_spark_executor_instances = var.tamr_spark_executor_instances
-    # ditto, comment about indent() above
-    tamr_spark_properties_override = indent(4, "\n${local.spark_properties}")
-    # sql
-    tamr_cloud_sql_project  = local.tamr_cloud_sql_project
-    tamr_cloud_sql_location = var.tamr_cloud_sql_location
-    tamr_cloud_sql_name     = var.tamr_cloud_sql_name
-    tamr_sql_user           = var.tamr_sql_user
-    tamr_sql_password       = var.tamr_sql_password
-    # elastic
-    remote_es                = local.remote_es
-    tamr_es_enabled          = var.tamr_es_enabled
-    tamr_es_apihost          = local.tamr_es_apihost
-    tamr_es_user             = var.tamr_es_user
-    tamr_es_password         = var.tamr_es_password
-    tamr_es_ssl_enabled      = var.tamr_es_ssl_enabled
-    tamr_es_number_of_shards = var.tamr_es_number_of_shards
-    tamr_es_socket_timeout   = var.tamr_es_socket_timeout
-    # file system
-    tamr_filesystem_bucket = var.tamr_filesystem_bucket
-    # miscellaneous
-    tamr_license_key  = var.tamr_license_key
-    tamr_json_logging = var.tamr_json_logging
-  })
+  external_ip = var.tamr_external_ip == true ? 1 : 0
 
   startup_script = templatefile("${path.module}/templates/startup_script.sh.tmpl", {
     tamr_zip_uri        = var.tamr_zip_uri
-    tamr_config         = local.tamr_config
+    tamr_config         = var.tamr_config_file
     tamr_home_directory = var.tamr_instance_install_directory
   })
 

outputs.tf

@@ -5,7 +5,7 @@ output "tamr_instance_self_link" {
 }
 
 output "tamr_instance_internal_ip" {
-  value       = google_compute_instance.tamr.network_interface.0.network_ip
+  value       = google_compute_address.tamr_ip.address
   description = "internal ip of tamr vm"
 }
 
@@ -19,19 +19,7 @@ output "tamr_instance_zone" {
   description = "zone of the tamr vm"
 }
 
-# config files
-# NOTE: these are very useful for debugging
-output "tamr_config_file" {
-  value       = local.tamr_config
-  description = "full tamr config file"
-}
-
-output "tmpl_dataproc_config" {
-  value       = local.default_dataproc
-  description = "dataproc config"
-}
-
-output "tmpl_statup_script" {
+output "tmpl_startup_script" {
   value       = local.startup_script
   description = "rendered metadata startup script"
 }