The "small packet" problem when reply confirm

[wsky]

The small-packet problem

the typical result is that 41 byte packets (one byte of data, 40 bytes of header) are transmitted for each byte of useful data. 

This 4000% overhead is annoying but tolerable on lightly loaded networks. 

On heavily loaded networks, however, 

the congestion resulting from this overhead can result in lost datagrams and retransmissions, 

as well as excessive propagation time caused by congestion in switching nodes and gateways.

In practice, throughput may drop so low that TCP connections are aborted.

http://www.freesoft.org/CIE/RFC/896/4.htm

The solution to the small-packet problem
http://www.freesoft.org/CIE/RFC/896/5.htm

http://www.informit.com/articles/article.aspx?p=99812&seqNum=5

TcpNoDelay=true then disable nagle.

[wsky]

some data

localhost, message size=1k, 3 client

• 64K tcpbuffer
  • push 250MB/s 232,245packet/s client100% java188% total 70%
  • push-confirm 17.4MB/s 165,368packet/s client100% java150% total 50%
• 4095K tcpbuffer
  • push 56MB/s 86,689packet/s client 40% java255% kernal 40%
  • push-confirm 35MB/s 63,619packet/s client100% server done fast，but memory increase to 3.5G
• Conclusion
  • enlarge tcpbuffer can course packet rate decrease，but throughput fall fown while too big buffer
  • buffer should be set a appropriate size by RTT, usually, 64K-128K is enough.