Nondeterministic "Cannot fetch logs" error in Nomad UI

[nh2]

Nomad version

v1.11.3

Operating system and Environment details

NixOS Linux

Issue

Often, when trying to view logs in the Nomad web UI, I get this error:

Cannot fetch logs
The logs for this task are inaccessible. Check the condition of the node the allocation is on.

• I usually proxy the Nomad UI through nginx, having followed the docs for proper reverse proxying with nginx: https://developer.hashicorp.com/nomad/docs/deploy/clusters/reverse-proxy-ui#review-the-complete-nginx-configuration
• However, the same error also occurs when SSH-forwarding the port for the Nomad UI, and accessing http://localhost:4646/, so nginx proxy settings should be irrelevant to this bug.
• Thus, I'm using this simpler non-nginx setup for the rest of the report.

The issue is nondeterministic:

• Within the same browser (e.g. Firefox): With sufficient amount of refreshing, I can toggle between the error being shown, and logs actually loading. Sometimes that can take a long time though.
• Across different browsers: Right now as I'm typing this sentence, I have a situation that when refreshing in Firefox, the error is shown, while in Chromium (refreshing simultaneously), the logs are shown successfully. On the same the URL.
  • This is after logs were shown successfully on that same Firefox 10 minutes earlier.

Observations:

• In the browser devtools, I can see that the browser first fails to load http://10.0.5.10:4646/v1/client/fs/logs/7eee1a15-...?follow=true... which is the IP of the allocation's client, which cannot be connected to from my browser. This is apparently expected as explained in Nomad UI - Why are logs piped directly from agents?  #6409 (comment)
  • The log warnings slightly differ (not sure that's relevant):
    • In a log-showing Chromium: LOG FETCH: Couldn't connect to //10.0.5.10:4646/v1/client/fs/logs/7eee1a15...
    • In a failing Firefox: LOG FETCH: Couldn't connect to /v1/client/fs/logs/7eee1a15
• I notice that the underlying errors that prevent connecting to 10.0.5.10 are slightly different:
  • In a log-showing Chromium: GET http://10.0.5.10:4646/v1/client/fs/logs/7eee1a15-70c5-9fe5-b9b3-7fce76cc0ae0?follow=true&offset=50000&origin=end&task=script-task&type=stderr net::ERR_CONNECTION_REFUSED
  • In a failing Firefox: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://10.0.5.10:4646/v1/client/fs/logs/7eee1a15-70c5-9fe5-b9b3-7fce76cc0ae0?follow=true&offset=50000&origin=end&task=script-task&type=stderr. (Reason: CORS request did not succeed). Status code: (null).
    That said, in Firefox's Network panel it also says NS_ERROR_CONNECTION_REFUSED for that request in the Transferred column.

Failing screenshot:

Succeeding screenshot:

Suffice to say, the actual log file on the Nomad client is available at all times in /var/lib/nomad/alloc/....

Reproduction steps

• Unclear

Expected Result

• Logs load reliably.

Actual Result

• Logs load unreliably as described above.

Possibly related issues

The same error message appears in:

• Add client header for UI to fix proxy buffering issues when viewing logs #7744
• Nomad UI gives errors when watching logs #5404

However, these issues were unhelpfully closed and locked by the github-actions stalebot, thus making it impossible to ask detail questions there.

[tehut]

Thanks for the bug report @nh2 , I've added this to our backlog for prioritization and will spend some time with it tomorrow as well.

[bfqrst]

Thanks for raising this @nh2 and thanks for looking into it @tehut ! I thought I was imagining this. We're operating behind Traefik FWIW. I too was aware of the needed settings when operating behind a reverse proxy, but wasn't quite sure if maybe Traefik was the culprit.

[tehut]

@nh2 & @bfqrst, I can absolutely confirm that neither of you are imagining this. I was able to trigger this behavior pretty consistently when reproducing via tunneling but I did find that I couldn't really force it to happen on a well behaved job when I followed the proxy configuration documentation.

I was able to reproduce intermittent failures against a job that failed quickly which makes it smell to me like there may be a race somewhere.

Does that match either of your experience? Are you seeing this behavior more frequently with failing jobs ( when logs matter most 😓 ) or does it appear to be more arbitrary.

[nh2]

Are you seeing this behavior more frequently with failing jobs

@tehut I don't have any good info on whether it happens more for some jobs than for others. I'm running exclusively batch jobs (no services); for me those jobs fail (which is indeed why I look at the logs) but in Nomad they are Complete (in our setup, our processing job exiting with non-0 exit code is expected for some jobs, so that also counts as "done" for us).

I also have the suspicion that it happened a lot more after upgrading from Nomad 1.6.8 to 1.10.5 (it hadn't really occurred to me before that upgrade, I believe), but haven't tested that hypothesis.

[nh2]

I also want to point out that this doesn't happen "sometimes" for me. It happens on ~90% of my jobs, making the UI virtually useless.

[Haydnspass]

I also experience the issue. For me it happens 50% of the time. More frequently when jobs start, older running jobs seem to work more often.

I experience it in the small window when clicking on View Logs more often

than when I click on the allocation, then the task and then on logs. Here it works 99% of the time.