unable to reproduce cloud run usability

[jsteinberg-rbi]

I copied the python cloud run example line-for-line. By now I've tried a million variations and additions and am simply unable to get traces shipped to my datadog instance.

# app.py

# I added the /config route
# because I was getting no
# useful cloud run logs
# as to why no traces were being
# collected
import os
from flask import Flask, Response

app = Flask(__name__)

@app.route("/")
def home():
    return Response("{\"msg\":\"Hello world!\"}", status=200, mimetype='application/json')

@app.route("/config")
def ddtrace_config():
    return Response(os.system('ddtrace-run --info'), status=200, mimetype='application/json')

app.run(host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

# Dockerfile

FROM python:3.9-slim
COPY --from=datadog/serverless-init:beta4 /datadog-init /app/datadog-init
ENV PYTHONUNBUFFERED True
WORKDIR /app
COPY requirements.txt app.py /app/
RUN pip install --no-cache-dir -r requirements.txt

ENV DD_SERVICE=REDACTED
# created only for test purposes
# will be destroyed after tests pass
ENV DD_API_KEY=REDACTED
ENV DD_HOST=https://us5.datadoghq.com
ENV DD_TRACE_ENABLED=true
ENV DD_ENV=REDACTED
ENV DD_VERSION=0.1
# added after not seeing a remote-config option
# in my org settings? have tried it with and without.
ENV DD_REMOTE_CONFIGURATION_ENABLED=false

ENTRYPOINT ["/app/datadog-init"]
CMD ["ddtrace-run", "python", "app.py"]

# requirements.txt

flask
ddtrace

# works just fine, service comes up and can respond without issues
gcloud run deploy REDACTED --image $DOCKER_IMAGE --region us-east1 --platform=managed --project=REDACTED --memory=8G --cpu=2 --vpc-connector=projects/REDACTED/locations/us-east1/connectors/dev-connector --allow-unauthenticated

# interesting DD log output showing results of `ddtrace-run --info`

X: telemetry.Proxy: http: proxy error: use of closed network connection

# from when I hit /config and ddtrace-run --info runs
Tracer Tags: None
Partial flushing enabled: True
Debug logging: False
Writing traces to: http://localhost:8126
Log injection enabled: False
Agent error: None
Partial flush minimum number of spans: 500
Health metrics enabled: False
Tracer enabled: True
DD Version: 0.1
Priority sampling enabled: True
Global Tags: None
DD Env: REDACTED
DD Service: REDACTED
App Analytics enabled(deprecated): False

• the cloud run service comes up
• I can hit it just fine and get hello world, etc
• cloud run logs are flowing to datadog from it no issue
• cloud run metrics are flowing to datadog no issue
• DD_API_KEY env var is present
• DD_HOST env var is present
• not using a proxy of any kind
• no weird networking of any kind
• other services (that I don't know anything about) are able to send traces though I don't think any use datadog-init

I'm utterly baffled as to why I'm not seeing traces populate my instance. I've scoured numerous logs and APM angles and nothing. And to make matters worse the logging isn't offering anything helpful. My understanding with ddtrace is that if you want to just "trace everything" it's enough to just have the Dockerfile CMD wrap app.py with ddtrace-run -- nothing else needed -- no decorators, no imports -- nothing. Any insight you could offer would be greatly, greatly appreciated. I'm sure its something completely stupid on my end!

UPDATE 1:

# added tracer debug and:

DEBUG:ddtrace.internal.telemetry.writer:failed to send telemetry to the Datadog Agent at http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry. response: 403

UPDATE 2:

• I've tried EXPOSE 8126 which makes no sense
• DD_APM_RECEIVER_PORT=80 and then setting cloud run deploy with port=80 which also makes no sense

UPDATE 3:

• now I've written a simple api that runs linux commands inside the container as a cheap hack for debugging it although there's probably a much better way to loudly debug a simple cloud run container: anyway
• ps aux returns: root 1 7.1 0.8 1329572 67580 ? Ssl 23:49 0:03 /app/datadog-init ddtrace-run python app.py which to me looks normal
• curl localhost:8126 returns 404 which is weird since if it's not refusing connections from curl why would it be refusing connections from datadog-init on that port -- I have done nothing whatsoever to the permissions of the package and actually I did check them and they x on every bit possible.
• I've also turned on DD_LOG_LEVEL=debug but this hasn't done anything helpful.

UPDATE 4:

• lullllz
• I opened both egress and ingress for 0.0.0.0/0 on the firewall for 8126 (which makes absolutely no sense why a process running in a container would be 403'd on a port) but this didn't do anything either

UPDATE 5:

• I created an APM service. Not sure if that is required. Documentation doesn't say anywhere that it is. Didn't change anything.

UPDATE 6:

• it appears at least that the 403 denied was something to do with my code maybe? Anyway -- I've resolved that I think, but still no joy on my end. I assume that with the datadog-init agent listening and a functional service actually running that when I curl various endpoints in my cloud run service that is enough to generate and ship a trace?

[jsteinberg-rbi]

still an issue, have written a ton more "api calls" to run linux commands inside the container, but still no joy.

[rynmccrmck]

@jsteinberg-rbi I recently hit the same issues as you and your post was helpful. I haven't figured out why , but using the pinned version of ddtrace (==1.1.2) from the requirements file in this repo has allowed me to get apm working with our python application on cloud run.