Skip to content

Shutdown has 2s delay when log forwarding is being used #681

@benhoyt

Description

@benhoyt

When there's an active log forwarding target (log gatherer), the LogManager.Stop call takes 2s during shutdown.

I'll provide a full repro later, but in the meantime, here are PEBBLE_DEBUG=1 logs showing the issue (from Polar Signals):

2025-08-05T14:19:32.136Z [pebble] DEBUG Attempting to stop service "writer1" by sending SIGTERM
2025-08-05T14:19:32.136Z [pebble] DEBUG Service "writer1" transitioning to state "terminating"
2025-08-05T14:19:32.136Z [pebble] DEBUG Reaped PID 3606052 which exited with code 143.
...
2025-08-05T14:19:32.138Z [pebble] DEBUG Reaped PID 3606053 which exited with code 143.
2025-08-05T14:19:32.138Z [pebble] DEBUG Service "writer9" exited with code 143.
2025-08-05T14:19:32.138Z [pebble] Service "writer9" stopped
2025-08-05T14:19:32.138Z [pebble] DEBUG Service "writer9" transitioning to state "stopped"
2025-08-05T14:19:32.197Z [pebble] DEBUG All services stopped.
2025-08-05T14:19:32.198Z [pebble] DEBUG State engine is stopping *state.TaskRunner
2025-08-05T14:19:32.204Z [pebble] DEBUG State engine is stopping *logstate.LogManager
2025-08-05T14:19:34.205Z [pebble] DEBUG gatherer "test": force killing log pullers
2025-08-05T14:19:34.205Z [pebble] DEBUG gatherer "test": pullers have finished
2025-08-05T14:19:34.206Z [pebble] DEBUG Reaper stopped.
2025-08-05T14:19:34.215Z [pebble] Time elapse (shutdown): 2.11s

Working theory (from Polar Signals): The log stream does not terminate the way it is expected to. It appears that logs.RingBuffer gets created, but never has Close called on it, which is what would call releaseIterators, which closes the nextChan would make the Next call return and exit the tomb.

Metadata

Metadata

Assignees

No one assigned

    Labels

    26.04An item we hope to do in the 26.04 cyclerainy daySmall items done in ~10% of each week's time

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions