Databricks: Unable to Access Files Created by CloudFetch #114324
Description
What happened?
This originated from this escalation where the customer noticed that they were not getting any results or an error when trying to run large Databricks queries.
The Databricks SDK automatically uses a feature called CloudFetch when returning a large amount of data. Basically they write results to a customer's cloud provider's bucket (in this case Azure) and return pre-signed URLs that we would read the results from. Looking through the logs I noticed this line which indicates that we don't have permission to read the files:
2025-11-19 17:58:36.020debuglogger=plugin.grafana-databricks-datasource t=2025-11-19T17:58:36.020581111Z level=debug msg="{"level":"debug","time":"2025-11-19T17:58:36Z","message":"CloudFetch: failed to download link at offset 0 row count 200, reason: Get "[https://dbstorage5rsakvq33xot2.blob.core.windows.net/jobs/8419334039469530/sql/2025-11-19/17/results_2025-11-19T17:58:35Z_93ca647f-c8e0-4e57-b474-5f352936a2f0?sig=FlXfiK77I6WcRbbJSGX7yR0c05otLRzN9aJ0JObAYGQ%3D&se=2025-11-19T18%3A13%3A35Z&sv=2019-02-02&spr=https&sp=r&sr=b\](https://dbstorage5rsakvq33xot2.blob.core.windows.net/jobs/8419334039469530/sql/2025-11-19/17/results_2025-11-19T17:58:35Z_93ca647f-c8e0-4e57-b474-5f352936a2f0?sig=FlXfiK77I6WcRbbJSGX7yR0c05otLRzN9aJ0JObAYGQ%3D&se=2025-11-19T18%3A13%3A35Z&sv=2019-02-02&spr=https&sp=r&sr=b%5C%5C%5C)": dial tcp: lookup dbstorage5rsakvq33xot2.blob.core.windows.net on 10.113.128.10:53: no such host"}"
This customer is also using PDC so this may be related to that. We should figure out if this is a wide spread problem or just isolated to this customer. In addition to resolving the CloudFetch issue, we should also make sure that an appropriate error message is returned.
What did you expect to happen?
We should be able to read the files returned by Databricks, or an error should be displayed to the user if we have trouble reading them.
Did this work before?
No, at least not for this customer
How do we reproduce it?
This may be difficult to reproduce since it requires a large amount of data in Databricks, and there is no way to force CloudFetch (as far as I can tell). We can only force CloudFetch to not be used, which is what we did to resolve the escalation.
- Run a Databricks query that returns a large amount of data so CloudFetch will be used
- If the feature toggle
disableCloudFetchis not set to true, then you will not get any data back or see an error message - You can then turn the feature toggle on if you want to see data returned not using CloudFetch
Is the bug inside a dashboard panel?
No response
Environment (with versions)?
Grafana:
OS:
Browser:
Grafana platform?
None
Datasource(s)?
No response