DF doesn't surface the kafka connection issue

[vnvo]

What happened?

If kafka has an issue and DF can not send a batch, there are logs from librdkafka but df itself doesn't surface the problem at all. The pipeline behavior overall is correct, there is some delay in the send but it reconnects and recovers but it potentially could stay hidden if librdkafka is suppressed in production.

What was expected?

deltaforge should log the issue and possible update some metrics

Steps to reproduce

• run the sink-outage chaos test: cargo run -p chaos -- --scenario sink-outage
• see the logs from deltaforge

Environment

• DeltaForge version: 0.1.0-beta.8
• Source/Sink/Processor: mysql, kafka with no processor
• OS: ubuntu

Logs/Error/Details

as you can see, there are no mention of the issue from deltaforge itself.

deltaforge-1  | 2026-03-14T18:32:23.615809Z  INFO dispatch_event:handle_table_map{source_id=chaos-mysql}: table mapped table_id=90 db=orders table=order_events
deltaforge-1  | 2026-03-14T18:32:26.043367Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Disconnected (after 10573ms in state UP)
deltaforge-1  | 2026-03-14T18:32:26.043381Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Disconnected (after 10573ms in state UP)
deltaforge-1  | 2026-03-14T18:32:26.043385Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
deltaforge-1  | 2026-03-14T18:32:26.043386Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
deltaforge-1  | 2026-03-14T18:32:26.966174Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/bootstrap]: toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:26.966192Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:26.966198Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:26.966203Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 2/2 brokers are down
deltaforge-1  | 2026-03-14T18:32:26.966206Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 2/2 brokers are down
deltaforge-1  | 2026-03-14T18:32:26.981925Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/1]: toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:26.981940Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:26.981946Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:27.218253Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/1]: toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:27.218268Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:27.218274Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:27.967010Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/bootstrap]: toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:27.967024Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:27.967031Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:55.724301Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Disconnected (after 7749ms in state UP)
deltaforge-1  | 2026-03-14T18:32:55.724318Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Disconnected (after 7749ms in state UP)
deltaforge-1  | 2026-03-14T18:32:55.724344Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Disconnected (after 7019ms in state UP)
deltaforge-1  | 2026-03-14T18:32:55.724352Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Disconnected (after 7019ms in state UP)
deltaforge-1  | 2026-03-14T18:32:55.724356Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 2/2 brokers are down
deltaforge-1  | 2026-03-14T18:32:55.724358Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 2/2 brokers are down
deltaforge-1  | 2026-03-14T18:32:55.724854Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/bootstrap]: toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:55.724860Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:55.724863Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:57.977177Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/bootstrap]: toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 5 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:57.977188Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 5 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:57.977193Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/bootstrap: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 5 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:32:58.703849Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/1]: toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:58.703896Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:32:58.703900Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT)
deltaforge-1  | 2026-03-14T18:33:08.703992Z ERROR librdkafka: FAIL [thrd:toxiproxy:5102/1]: toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:33:08.704005Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:33:08.704009Z ERROR librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): toxiproxy:5102/1: Connect to ipv4#172.18.0.7:5102 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
deltaforge-1  | 2026-03-14T18:33:17.984703Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 2/2 brokers are down
deltaforge-1  | 2026-03-14T18:33:17.984727Z ERROR librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 2/2 brokers are down