Adding Functionality For Custom Metrics Testing #463
Conversation
| # Initialize OTEL metrics for span metrics | ||
| meter = metrics.get_meter(__name__) | ||
| request_counter = meter.create_counter("custom_requests_total", description="Total requests") | ||
| response_time_histogram = meter.create_histogram("custom_response_time", description="Response time") | ||
|
|
There was a problem hiding this comment.
IMO we should create two meters - one using metrics.get_meter(__name__) (Agent-based export) like you've done here, and another using Custom export pipeline from public docs
There was a problem hiding this comment.
2nd meter has been created.
| # Initialize OTEL metrics for span metrics | ||
| meter = metrics.get_meter(__name__) | ||
| request_counter = meter.create_counter("custom_requests_total", description="Total requests") | ||
| response_time_histogram = meter.create_histogram("custom_response_time", description="Response time") |
There was a problem hiding this comment.
response_time_histogram unused?
There was a problem hiding this comment.
histogram code commented out for easy implementation after initial test validation.
| # Setup Span Attributes And Initialize Counter To Recieve Custom Metrics | ||
| span = trace.get_current_span() | ||
| span.set_attribute("operation.type", "aws_sdk_call") | ||
| request_counter.add(1, {"operation.type": "aws_sdk_call"}) | ||
|
|
There was a problem hiding this comment.
TBH I don't think we need custom metrics on every API. Probably one is fine.
There was a problem hiding this comment.
Removed from all API's except sdk call.
There was a problem hiding this comment.
Are we setting OTEL_RESOURCE_ATTRIBUTES="service.name=$YOUR_SVC_NAME,deployment.environment.name=$YOUR_ENV_NAME" anywhere?
There was a problem hiding this comment.
yes it is being set in 'main.tf'
| variable "custom_metrics_config" { | ||
| description = "JSON configuration for custom metrics" | ||
| type = string | ||
| default = "{}" |
There was a problem hiding this comment.
amazon-cloudwatch-custom-agent.json?
There was a problem hiding this comment.
It replaces the empty curly brackets as the default now.
| name: Service | ||
| value: {{serviceName}} |
There was a problem hiding this comment.
Should also see deployment.environment.name
There was a problem hiding this comment.
It has been added.
There was a problem hiding this comment.
I don't think you will see anything in remote service metrics
There was a problem hiding this comment.
It has been removed.
2 participants
This pr is adds mustache files and a new CWA for custom metrics e2e testing, along with an update to variables (Terraform) to recognize those changes.
Changes made:
Added CWA with otlp specification for custom metrics.
Updated python sample app to collect span/Otel custom metrics for example:
( span = trace.get_current_span()
span.set_attribute("operation.type", "downstream_service")
request_counter.add(1, {"operation.type": "downstream_service"})
Updated variables file to include "custom_metrics_enabled" and "custom_metrics_config"
Added 5 mustache files to validate against new metrics collected from sample app.
References:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AppSignals-CustomMetrics.html#AppSignals-CustomMetrics-OpenTelemetry
https://opentelemetry.io/docs/specs/otel/trace/sdk/
https://opentelemetry.io/docs/specs/otel/glossary/#instrumented-library
https://opentelemetry.io/docs/specs/otel/trace/api/#tracerprovider
Should this be merged and fail a git revert to SHA b58dd61 will be enacted to rollback all changes made.
Ensure you've run the following tests on your changes and include the link below:
To do so, create a
test.ymlfile withname: Testand workflow description to test your changes, then remove the file for your PR. Link your test run in your PR description. This process is a short term solution while we work on creating a staging environment for testing.NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL. See the needs keyword to run tests in succession.
e2e-playgroundin us-east-1 and eu-central-2e2e-playgroundin us-east-1 and eu-central-2e2e-playgroundin us-east-1 and eu-central-2By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.