-
Notifications
You must be signed in to change notification settings - Fork 48
Open
Description
Frequently the Glean Dictionary deployment is failing because the EXPERIMENT_DATA_URL returns an error/invalid JSON.
e.g. https://app.netlify.com/projects/glean-dictionary-prod/deploys/696c5f5651f8740008620264
<snip>
5:20:10 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2 (https://ntl.fyi/exit-code-2)
<snip>
5:20:10 AM: File "/opt/build/repo/etl/__main__.py", line 24, in build_metadata
5:20:10 AM: write_glean_metadata(
5:20:10 AM: File "/opt/build/repo/etl/glean_etl.py", line 242, in write_glean_metadata
5:20:10 AM: metrics_sampling_info = _get_metric_sample_data(requests.get(EXPERIMENT_DATA_URL).json())
5:20:10 AM: File "/opt/buildhome/.local/share/mise/installs/python/3.9.23/lib/python3.9/site-packages/requests/models.py", line 975, in json
5:20:10 AM: raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
5:20:10 AM: requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
A deployment sometime later usually just works then (rarely it needs 2-3 times to work).
We should change the code to simply try that URL a couple of times (with a small backoff each time). That most likely will make the deployments slightly more robust.
Code is here:
glean-dictionary/etl/glean_etl.py
Line 242 in 4dcb98b
| metrics_sampling_info = _get_metric_sample_data(requests.get(EXPERIMENT_DATA_URL).json()) |
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels