Fluentbit doesn't work in version 3.21

[fermion2020]

Summary

The Fluent Bit service can't send logs to a Loki server. This error started appearing in version 3.21, while the previous version worked fine.

Steps to reproduce

Setup fluentbit environments according to documentation, run a container and check the fluentbit log.

What is the expected correct behavior?

All works like in alpine:3.20 version.

Relevant logs and/or screenshots

Here is a log:

[2025/03/11 13:05:01] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2025/03/11 13:05:01] [error] [output:loki:loki.1] no upstream connections available

Even if you set

     - FLUENTBIT_OUTPUT_LOKI_TLS=FALSE
     - FLUENTBIT_OUTPUT_LOKI_TLS_VERIFY=FALSE

You will get this error:

[2025/03/11 13:24:17] [ warn] [engine] failed to flush chunk '940-1741724656.883442233.flb', retry in 8 seconds: task_id=4, input=tail.4 > output=loki.1 (out_id=1)
[2025/03/11 13:24:18] [error] [output:loki:loki.1] <loki-server>:443, HTTP status=404
404 page not found

Environment

• Image version / tag: 3.21
• Host OS: Arch Linux / NixOS

Any logs | docker-compose.yml

Possible fixes

[tiredofit]

For 3.20 and 3.21 can I get the output of the following:

• image_version
• env | grep 'FLUENTBIT_'

It sounds as if there could be a problem in upstream Alpine 3.21 LibSSL as all images should theoretically use the same fluent-bit version. This may take some digging. I'm with limited internet for the forseeable future, so I'm not sure if using 3.21 for Fluent-Bit forwarding is entirely necessary, as 3.20 still (should) be running the same version, and is also still receiving distribution updates.

[fermion2020]

Of course. Here are:

For 3.20:

image_version - Image: /assets/.changelogs/tiredofit_docker-alpine.md 7.10.12
environment:

FLUENTBIT_MODE=PROXY
FLUENTBIT_OUTPUT=LOKI
FLUENTBIT_OUTPUT_LOKI_HOST=ship.<censored>
FLUENTBIT_OUTPUT_LOKI_PASS=<censored>
FLUENTBIT_OUTPUT_LOKI_PORT=443
FLUENTBIT_OUTPUT_LOKI_TLS=TRUE
FLUENTBIT_OUTPUT_LOKI_TLS_VERIFY=TRUE
FLUENTBIT_OUTPUT_LOKI_USER=<censored>
FLUENTBIT_VERSION=3.1.7

For 3.21:

image_version - Image: /assets/.changelogs/tiredofit_docker-alpine.md 7.10.28
environment:

FLUENTBIT_MODE=PROXY
FLUENTBIT_OUTPUT=LOKI
FLUENTBIT_OUTPUT_LOKI_HOST=ship..<censored>
FLUENTBIT_OUTPUT_LOKI_PASS=.<censored>
FLUENTBIT_OUTPUT_LOKI_PORT=443
FLUENTBIT_OUTPUT_LOKI_TLS=TRUE
FLUENTBIT_OUTPUT_LOKI_TLS_VERIFY=TRUE
FLUENTBIT_OUTPUT_LOKI_USER=.<censored>
FLUENTBIT_VERSION=3.1.10

[tiredofit]

Lots of differences between both image version and Fluent-bit. You may try tiredofit/alpine:3.20-7.10.29 and also tiredofit/alpine:3.21-7.10.29 and see if they have similar issues.

[fermion2020]

Sorry for the delay.

I tested tiredofit/alpine:3.20-7.10.28 with FLUENTBIT_VERSION=3.1.10 and tiredofit/alpine:3.21-7.10.28 with the same fluentbit version. The second image produces TLS and other errors, as previously mentioned. It seems likely that the issue stems from library changes.

[tiredofit]

Interesting. 3.21 alpine brought significant breakage with newer versions of libraries. Will try a different way of dealing with TLS shortly.

[tiredofit]

Did you want to try if the latest tiredofit/alpine:3.21 resolves the issue? I haven't committed the changes, only built locally and pushed to registry so far.

[fermion2020]

I tried the latest version, but unfortunately, I got the same errors:

 [error] [tls] error: unexpected EOF with reason: certificate verify failed
[error] [output:loki:loki.1] no upstream connections available

[tiredofit]

OK, Have you checked any of the environment variables in the image to see if they are working or have been changed at all? Specifically FLUENTBIT_*_TLS_*.

Also, may not be a bad idea to try to see from the configuration files and this reference to see if something has changed.

[fermion2020]

Oh, this is exactly what I was looking for. I set:

- FLUENTBIT_OUTPUT_LOKI_TLS=TRUE
- FLUENTBIT_OUTPUT_LOKI_TLS_VERIFY=FALSE

and it’s working for me now. Thank you!