Error: checking for presence of existing Secret "openai-api-key"

[fpittelo]

│ Error: checking for presence of existing Secret "openai-api-key" (Key Vault "https://dev-alpinebot-vault.vault.azure.net/"): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 20.81.158.144\r\nCaller: appid=***;oid=09c14ecb-663a-4efb-92a0-5e2182605824;iss=https://sts.windows.net/***/\r\nVault: dev-alpinebot-vault;location=switzerlandnorth" InnerError={"code":"ForbiddenByFirewall"}
│
│ with azurerm_key_vault_secret.openai_key,
│ on main.tf line 45, in resource "azurerm_key_vault_secret" "openai_key":
│ 45: resource "azurerm_key_vault_secret" "openai_key" {
│
╵
Releasing state lock. This may take a few moments...
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

[fpittelo]

│ Error: checking for presence of existing Secret "openai-api-key" (Key Vault "https://dev-alpinebot-vault.vault.azure.net/"): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 52.234.42.34\r\nCaller: appid=***;oid=09c14ecb-663a-4efb-92a0-5e2182605824;iss=https://sts.windows.net/***/\r\nVault: dev-alpinebot-vault;location=switzerlandnorth" InnerError={"code":"ForbiddenByFirewall"}
│
│ with azurerm_key_vault_secret.openai_key,
│ on main.tf line 47, in resource "azurerm_key_vault_secret" "openai_key":
│ 47: resource "azurerm_key_vault_secret" "openai_key" {
│
╵
Releasing state lock. This may take a few moments...
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

[fpittelo]

module.linux_web_app.azurerm_linux_web_app.wap_app: Modifications complete after 33s [id=/subscriptions//resourceGroups/dev-alpinebot/providers/Microsoft.Web/sites/dev-alpinebot-as]
╷
│ Error: checking for presence of existing Secret "openai-api-key" (Key Vault "https://dev-alpinebot-vault.vault.azure.net/"): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 172.215.231.114\r\nCaller: appid=;oid=09c14ecb-663a-4efb-92a0-5e2182605824;iss=https://sts.windows.net/***/\r\nVault: dev-alpinebot-vault;location=switzerlandnorth" InnerError={"code":"ForbiddenByFirewall"}
│
│ with azurerm_key_vault_secret.openai_key,
│ on main.tf line 61, in resource "azurerm_key_vault_secret" "openai_key":
│ 61: resource "azurerm_key_vault_secret" "openai_key" {
│
╵
Releasing state lock. This may take a few moments...
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

[fpittelo]

Error persist:

module.linux_web_app.azurerm_linux_web_app.wap_app: Modifications complete after 35s [id=/subscriptions//resourceGroups/dev-alpinebot/providers/Microsoft.Web/sites/dev-alpinebot-as]
╷
│ Error: checking for presence of existing Secret "openai-api-key" (Key Vault "https://dev-alpinebot-vault.vault.azure.net/"): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 57.151.136.178\r\nCaller: appid=;oid=09c14ecb-663a-4efb-92a0-5e2182605824;iss=https://sts.windows.net/***/\r\nVault: dev-alpinebot-vault;location=switzerlandnorth" InnerError={"code":"ForbiddenByFirewall"}
│
│ with azurerm_key_vault_secret.openai_key,
│ on main.tf line 61, in resource "azurerm_key_vault_secret" "openai_key":
│ 61: resource "azurerm_key_vault_secret" "openai_key" {
│
╵
Releasing state lock. This may take a few moments...
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

Proposed solution: Dynamic IP Injection
To solve this dynamically (allowing for full destroy/redeploy cycles on ephemeral build agents without manual intervention), you need to instruct Terraform to fetch the current runner's public IP and inject it into the Key Vault's firewall rules during the run.
You need to use the hashicorp/http provider to fetch the execution environment's IP address and pass it to the network_acls block of your Key Vault

1. Add the HTTP Provider
2. Retrieve the Runner's IP
3. Configure the Key Vault Firewall

Critical Step for Stability: Handling Propagation Latency
Azure Key Vault firewall rule updates are not instant. Even if Terraform says the Key Vault is updated, it may take 10-60 seconds for the firewall to actually allow the connection. This often causes the subsequent azurerm_key_vault_secret resource to fail with the same 403 error because it runs too fast.

To fix this, introduce a time_sleep resource to force a pause after the Key Vault is created/updated but before the Secret is created.

[fpittelo]

We still have an Terraform Apply error.

Plan: 3 to add, 2 to change, 1 to destroy.
╷
│ Error: making Read request on Azure KeyVault Secret openai-api-key: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 52.176.35.114\r\nCaller: appid=***;oid=09c14ecb-663a-4efb-92a0-5e2182605824;iss=https://sts.windows.net/***/\r\nVault: dev-alpinebot-vault;location=switzerlandnorth" InnerError={"code":"ForbiddenByFirewall"}
│
│ with azurerm_key_vault_secret.openai_key,
│ on main.tf line 86, in resource "azurerm_key_vault_secret" "openai_key":
│ 86: resource "azurerm_key_vault_secret" "openai_key" {
│
╵
Releasing state lock. This may take a few moments...
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

Recommended Debugging Step: The "Open Door" Test
We want to confirm if the rest of the infrastructure works, the fastest solution is to temporarily open the firewall. This confirms that the issue is only network blocking and not a permission (RBAC) issue.

1. Modify modules/key_vault/main.tf Change the default_action from "Deny" to "Allow".
2. We will Run the Deploy Infrastructure workflow.

If this succeeds, we have confirmed the Terraform logic and RBAC assignments are correct. We can then revert to "Deny" and focus purely on fixing the dynamic IP script.

If this fails with a 403, you have a permission (Role Assignment) issue, not just a firewall issue.

[fpittelo]

I will implement the proposed diagnostic step to confirm if the 403 error is solely due to the firewall.

Proposed Correction:
Temporarily modify modules/key_vault/main.tf to allow all network traffic.

  network_acls {
    default_action             = "Allow" # Temporarily changed from "Deny"
    bypass                     = "AzureServices"
    ip_rules                   = var.key_vault_ip_rules
    virtual_network_subnet_ids = var.key_vault_subnet_ids
  }

I will apply this change now. Please verify the deploy-infra workflow run. If it succeeds, we will revert this to "Deny" and focus on the dynamic IP whitelisting.

[fpittelo]

After applying fix Terraform Apply error:

Plan: 3 to add, 2 to change, 1 to destroy.
╷
│ Error: making Read request on Azure KeyVault Secret openai-api-key: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 20.236.100.1\r\nCaller: appid=***;oid=09c14ecb-663a-4efb-92a0-5e2182605824;iss=https://sts.windows.net/***/\r\nVault: dev-alpinebot-vault;location=switzerlandnorth" InnerError={"code":"ForbiddenByFirewall"}
│
│ with azurerm_key_vault_secret.openai_key,
│ on main.tf line 86, in resource "azurerm_key_vault_secret" "openai_key":
│ 86: resource "azurerm_key_vault_secret" "openai_key" {
│
╵
Releasing state lock. This may take a few moments...
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

[fpittelo]

Analysis:
The error ForbiddenByFirewall persists, which indicates that the Key Vault firewall is still set to "Deny" in Azure.
Since the code in modules/key_vault/main.tf is definitely set to default_action = "Allow", this means the change has not been successfully applied to the Azure resource yet.

Possible Reasons:

1. Commit/Push: The change to modules/key_vault/main.tf might not have been committed or pushed to the repository before the pipeline ran.
2. Apply Order: Terraform might be attempting to read/create the secret before it has finished updating the Key Vault firewall rules.

Proposed Correction:

1. Verify Commit: Please ensure that the change (default_action = "Allow") is committed and pushed to the correct branch.
2. Check Plan: Look at the "Terraform Plan" step in your CI/CD logs. You should see a modification for module.key_vault.azurerm_key_vault.alpinebot_kv changing network_acls.default_action from "Deny" to "Allow".
3. Targeted Apply (Optional): If the plan shows the change but it still fails, the Key Vault might need a moment to propagate the firewall change. You could try re-running the job.

I will wait for your confirmation that the plan shows the expected change.