v0.24.0 Server Panics with Policy Checks

[chriskuchin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Policy check when combined with prometheus metrics seems to cause a panic in the server.

Reproduction Steps

My theory is it just requires metrics to be enabled with a policy check based on the error/panic

Logs

{"level":"error","ts":"2023-05-23T17:50:02.818Z","caller":"events/command_runner.go:429","msg":"PANIC: descriptor Desc{fqName: \"atlantis_project_policy check_execution_time\", help: \"atlantis_project_policy check_execution_time summary\", constLabels: {}, variableLabels: [pr_number project project_path base_repo terraform_version workspace]} is invalid: \"atlantis_project_policy check_execution_time\" is not a valid metric name\ngithub.com/uber-go/tally@v3.5.3+incompatible/prometheus/reporter.go:306 (0x63867b)\ngithub.com/uber-go/tally@v3.5.3+incompatible/prometheus/reporter.go:554 (0x63813b)\ngithub.com/uber-go/tally@v3.5.3+incompatible/scope.go:363 (0x53264f)\ngithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:66 (0xa67177)\ngithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42 (0xa66bff)\ngithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48 (0xa77fbb)\ngithub.com/runatlantis/atlantis/server/events/policy_check_command_runner.go:65 (0xa6e6a7)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:282 (0xa6d98f)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xa6db7b)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:298 (0xa56ddb)\nruntime/asm_arm64.s:1172 (0x78623)\n","json":{"repo":"route/platform/sre/infrastructure/atlantis","pull":"58"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:429\nruntime.gopanic\n\truntime/panic.go:890\ngithub.com/uber-go/tally/prometheus.NewReporter.func1\n\tgithub.com/uber-go/tally@v3.5.3+incompatible/prometheus/reporter.go:306\ngithub.com/uber-go/tally/prometheus.(*reporter).AllocateTimer\n\tgithub.com/uber-go/tally@v3.5.3+incompatible/prometheus/reporter.go:554\ngithub.com/uber-go/tally.(*scope).Timer\n\tgithub.com/uber-go/tally@v3.5.3+incompatible/scope.go:363\ngithub.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:66\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*PolicyCheckCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/policy_check_command_runner.go:65\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:282\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:298"}

[rkstrickland]

I'm seeing incompatible scope in that log. Could it be related to this change? #3416
cc: @nitrocode @albertorm95

[nitrocode]

Do you mind checking the latest build prior to 0.24.1 release?

  ghcr.io/runatlantis/atlantis:dev-alpine
  ghcr.io/runatlantis/atlantis:dev-alpine-4a7fc33
  ghcr.io/runatlantis/atlantis:dev

[albertorm95]

This was fix in:

• fixbug: Update commandName in RunAndEmitStats #3437

In order to release the fix, first this one needs to be resolve:

• v0.24.0 conftest regression: invalid character 't' after top-level value #3438

Updated:
I can confirm the metrics are not generating the error in policy check from:

• v0.24.0 conftest regression: invalid character 't' after top-level value #3438

The bug was introduced in:

• feat(policies): Add granular policy_sets #3086

[albertorm95]

This can be mark as solve :)

[nitrocode]

Thanks for confirming @albertorm95