PostgreSQL 11 is EOL and out of standard support for RDS causing additional cost

[hexa00]

The PostgreSQL version used in the datastore module is currently 11 as seen in :

terraform-aws-metaflow/modules/datastore/variables.tf

Line 14 in ee7093c

default = "11"

However this version is now at EOL and costs extra to be supported by AWS (0.10$/h/VCPU) see:

https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
https://aws.amazon.com/rds/postgresql/pricing/

Could this be upgraded ?

[savingoyal]

sure! would you like to submit a PR?

[hexa00]

I could I'm just not sure what we'd need to watch out for for CI / testing of this... ?

If it's all automatic , I'm fine with a 1-3 liner for it that will change to the latest 11 (11.22) to 12.17
see: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion

You can't just do 11 -> 12 .. needs to be 12.17 and we need to assume that people are on the latest minor for the upgrade to work without issue on aws's side (which is a resonable assumption I think given it upgrades minors automatically)

You also need to add allow_major_version_upgrade = true
And potentially apply_immediately = true

Is that ok ?

[hexa00]

Actually I'm going to keep allow_major & apply to false and add that as a var for the user

[hexa00]

See: #97

[mszepieniec]

I'm hitting this issue as well, in fact the terraform apply no longer works:

│ Error: creating RDS DB Instance (metaflowmetaflowkrut0myf): operation error RDS: CreateDBInstance, https response error StatusCode: 400, RequestID: 469e6a44-04c0-49ea-8b0a-d12a8b323ad1, api error InvalidParameterCombination: RDS does not support creating a DB instance with the following combination: DBInstanceClass=db.t2.small, Engine=postgres, EngineVersion=11.22, LicenseModel=postgresql-license. For supported combinations of instance class and database engine version, see the documentation.
│
│   with module.metaflow-datastore.aws_db_instance.this[0],
│   on .terraform/modules/metaflow-datastore/modules/datastore/rds.tf line 98, in resource "aws_db_instance" "this":
│   98: resource "aws_db_instance" "this" {
│
╵
╷

Is this repo still the recommended way to set up metaflow?

[davified]

Hi @mszepieniec , I encountered the same issue when using the terraform-aws-metaflow terraform module, and fixed it by overriding db_engine_version and db_instance_type to select an engine-instance_type combination that's valid/supported as of the time of writing (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.Support.html)

# main.tf
module "metaflow" {
  source  = "outerbounds/metaflow/aws"
  version = "0.12.0"

  # ... other parameters
  db_engine_version                      = var.db_engine_version
  db_instance_type                       = var.db_instance_type
}

# variables.tf

variable "db_engine_version" {
  type        = string
  default     = "16"
  description = "RDS DB Engine Version"
}

variable "db_instance_type" {
  type        = string
  default     = "db.t3.small"
  description = "RDS instance class"
}

Haven't found the time to make a PR to https://github.com/outerbounds/metaflow-tools/tree/master/aws/terraform (which has been really helpful btw)