fix(tcp): auto-grow epoll event buffer to prevent connection starvation#67
Open
louisponet wants to merge 1 commit intomainfrom
Open
fix(tcp): auto-grow epoll event buffer to prevent connection starvation#67louisponet wants to merge 1 commit intomainfrom
louisponet wants to merge 1 commit intomainfrom
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+635
to
+643
| fn maybe_grow_events(&mut self, n_events: usize) { | ||
| // mio returns at most `capacity` events; hitting the limit means | ||
| // there were likely more fds ready than we could service. | ||
| if n_events >= self.event_capacity { | ||
| let new_cap = self.event_capacity * 2; | ||
| info!(old = self.event_capacity, new = new_cap, "epoll event buffer full, growing"); | ||
| self.event_capacity = new_cap; | ||
| self.events = Events::with_capacity(new_cap); | ||
| } |
Contributor
There was a problem hiding this comment.
🟡 maybe_grow_events never grows from zero capacity due to 0 * 2 == 0
If with_event_capacity(0) is called, maybe_grow_events enters a degenerate state: n_events >= 0 is always true for usize, so the growth branch fires every poll cycle, but 0 * 2 = 0 means the capacity never actually increases. This causes: (1) the connector never processes any IO events since Events::with_capacity(0) can never return events from poll(), making the connector completely non-functional, and (2) an info! log is emitted on every single poll_with call, creating infinite log spam.
Was this helpful? React with 👍 or 👎 to provide feedback.
louisponet
force-pushed
the
lopo/epoll-event-buffer-auto-grow
branch
2 times, most recently
from
b702902 to
76f268a
Compare
…igurable nodelay, EOF logging Four fixes/improvements for TcpConnector: 1. Auto-grow epoll event buffer to prevent connection starvation. 2. Apply TCP_USER_TIMEOUT to accepted inbound connections (was only set on outbound). 3. Make TCP_NODELAY configurable via with_nodelay(bool). Default remains true (Nagle disabled). 4. Add debug logging to silent EOF paths in read_frame. Previously, Ok(0) during header/payload reads returned Disconnected with no log, making it impossible to distinguish peer-closed from I/O errors. Now logs peer address and read progress.
louisponet
force-pushed
the
lopo/epoll-event-buffer-auto-grow
branch
from
76f268a to
895a7c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Events::with_capacity(128)inTcpConnectorcaused epoll event starvation when >128 connections were active (e.g. the data gather receiver).Mio uses edge-triggered epoll — connections that don't fit in the 128-event batch are deferred to the next
poll()cycle. With heavy connections dominating each batch, lighter connections experienced ~1s read delays. The kernel measured this asrcv_rtt: 999ms, auto-tunedrcv_spacedown to ~14KB, and advertised a tiny TCP receive window — causing the sender to berwnd_limited: 99.7%of the time.Diagnosis (from
ss -tnipon both sides)Sender side:
rwnd_limited: 99.7%— blocked on receiver's window almost all the timesnd_wnd: 73728(72KB) — tiny window from receiverSend-Q: 931816— ~1MB queued in kernelReceiver side (starved connection):
rcv_space: 14480(14KB) — kernel auto-tuned window downrcv_rtt: 999ms— kernel thinks app takes ~1s between readsReceiver side (healthy connection):
rcv_space: 4734960(4.7MB) — normalbytes_received: 3.8GB— orders of magnitude more throughputFix
poll(), if the number of returned events equals the buffer capacity, double itinfolevel when growth occurs ("epoll event buffer full, growing").with_event_capacity(n)builder method for callers that want a different starting size