FR: When tunneling over DERP (TCP), drop packets to manipulate the inner TCP congestion control

[sfllaw]

What are you trying to do?

DERP relays listen for their traffic over TCP on the HTTPS port, because everyone’s network allows outbound traffic to port 443. The DERP client actually generates packet loss with its magic sock:

tailscale/wgengine/magicsock/derp.go

Lines 431 to 432 in 46505ca

	ctx, cancel := context.WithCancel(c.connCtx)
	ch := make(chan derpWriteRequest, bufferedDerpWritesBeforeDrop())

tailscale/wgengine/magicsock/magicsock.go

Lines 1387 to 1398 in 46505ca

	select {
	case <-c.donec:
	metricSendDERPErrorClosed.Add(1)
	return false, errConnClosed
	case ch <- derpWriteRequest{addr, pubKey, pkt, isDisco}:
	metricSendDERPQueued.Add(1)
	return true, nil
	default:
	metricSendDERPErrorQueue.Add(1)
	// Too many writes queued. Drop packet.
	return false, errDropDerpPacket
	}

This packet loss signals to the inner TCP congestion algorithm to slow down. But it looks like there might be some head-of-line blocking that could cause TCP meltdown, where the inner TCP spends all its time retransmitting. Which coincidentally, is the Sysiphean boulder that @apenwarr keeps pushing up that hill.

How should we solve this?

TBD

What is the impact of not solving this?

The workaround is to use a totally reliable network connection between you and a DERP relay that never experiences high latency or drops packets. /s

[raggi]

• DERPs do drop packets
• The flows also traverse non-DERP paths

[sfllaw]

• DERPs do drop packets

Sorry, I don’t understand what you mean by this?

Was it a reference to the workaround? Sorry, I will add the sarcasm marker. It was tongue-in-cheek.

• The flows also traverse non-DERP paths

Under Linux, each Tailscale node has its own route, doesn’t it? Unless I’m misunderstanding things, a node doesn’t send partial traffic over DERP. So if we know that a connection from our node to the remote node is relayed by DERP, can’t we set the TCP RTO for the inner route so it doesn’t timeout?

[raggi]

Sorry, I don’t understand what you mean by this?

DERP internally contains a per peer packet queue. This queue is not unbounded, and the derper will drop packets from this queue.

Under Linux, each Tailscale node has its own route, doesn’t it?

On linux by default this is true. The default configuration varies by platform here (see a flag named something like "one cgnat route"). Adjusting route configuration regularly will cause some poorly behaved software (and most particularly Chrome) to forcefully close connections associated with that route, so we can not make decisions at a high rate. The way that magic sock operates today is that it sends packets along both UDP and DERP paths concurrently, it will back off from UDP when it is not working. The stateful notion can change at a high frequency and is not represented in a "one way or another" kind of modality as would be required to do this well.

[sfllaw]

DERP internally contains a per peer packet queue. This queue is not unbounded, and the derper will drop packets from this queue.

Lemme see if I understand correctly. Clients can send TCP over the tailscale0 tunnel, which gets encrypted into Wireguard packets and sent through the magicsock UDP into the per-region queue?

tailscale/wgengine/magicsock/derp.go

Lines 431 to 432 in 46505ca

	ctx, cancel := context.WithCancel(c.connCtx)
	ch := make(chan derpWriteRequest, bufferedDerpWritesBeforeDrop())

And the dropping happens here?

tailscale/wgengine/magicsock/magicsock.go

Lines 1387 to 1398 in 46505ca

	select {
	case <-c.donec:
	metricSendDERPErrorClosed.Add(1)
	return false, errConnClosed
	case ch <- derpWriteRequest{addr, pubKey, pkt, isDisco}:
	metricSendDERPQueued.Add(1)
	return true, nil
	default:
	metricSendDERPErrorQueue.Add(1)
	// Too many writes queued. Drop packet.
	return false, errDropDerpPacket
	}

So we are relying on these dropped packets to cause loss so that the inner TCP congestion control kicks in?

Doesn’t the DERP-over-HTTP client still have to reliably send in order to drain the derpWriteRequest channel? And if the outer TCP takes too long to send a message in this channel, then the inner TCP will time out and retransmit. But then the rest of the channel has to drain, so that means the channel’s contents will cause head-of-line blocking? Maybe a buffer of 32 or 256 messages isn’t that bad, but wouldn’t the back-pressure be more responsive if the buffer were shorter?

tailscale/wgengine/magicsock/derp.go

Lines 692 to 704 in 46505ca

	case wr := <-ch:
	err := dc.Send(wr.pubKey, wr.b)
	if err != nil {
	c.logf("magicsock: derp.Send(%v): %v", wr.addr, err)
	metricSendDERPError.Add(1)
	if !wr.isDisco {
	c.metrics.outboundPacketsDroppedErrors.Add(1)
	}
	} else if !wr.isDisco {
	c.metrics.outboundPacketsDERPTotal.Add(1)
	c.metrics.outboundBytesDERPTotal.Add(int64(len(wr.b)))
	}
	}

Or if the buffering is desirable, shouldn’t we keep track of how long the packets have been waiting in this channel? If they’ve been waiting too long, then they’ve probably been retransmitted so there’s no point in sending them now.

Am I’m missing something?

Under Linux, each Tailscale node has its own route, doesn’t it?

On linux by default this is true. The default configuration varies by platform here (see a flag named something like "one cgnat route").

tailscale/ipn/ipnlocal/local.go

Lines 5058 to 5063 in 46505ca

	// shouldUseOneCGNATRoute reports whether we should prefer to make one big
	// CGNAT /10 route rather than a /32 per peer.
	//
	// The versionOS is a Tailscale-style version ("iOS", "macOS") and not
	// a runtime.GOOS.
	func shouldUseOneCGNATRoute(logf logger.Logf, mon *netmon.Monitor, controlKnobs *controlknobs.Knobs, versionOS string) bool {

Adjusting route configuration regularly will cause some poorly behaved software (and most particularly Chrome) to forcefully close connections associated with that route, so we can not make decisions at a high rate.

I guess you’re referring to #3102. 🙁

And I get it, setting RTO per /32 will not help if we really are trying to cause TCP congestion control to do its job.

The way that magic sock operates today is that it sends packets along both UDP and DERP paths concurrently, it will back off from UDP when it is not working. The stateful notion can change at a high frequency and is not represented in a "one way or another" kind of modality as would be required to do this well.

Yes, I understand now:

tailscale/wgengine/magicsock/endpoint.go

Lines 962 to 1009 in 46505ca

	if udpAddr.IsValid() {
	_, err = de.c.sendUDPBatch(udpAddr, buffs)
	// If the error is known to indicate that the endpoint is no longer
	// usable, clear the endpoint statistics so that the next send will
	// re-evaluate the best endpoint.
	if err != nil && isBadEndpointErr(err) {
	de.noteBadEndpoint(udpAddr)
	}
	var txBytes int
	for _, b := range buffs {
	txBytes += len(b)
	}
	switch {
	case udpAddr.Addr().Is4():
	de.c.metrics.outboundPacketsIPv4Total.Add(int64(len(buffs)))
	de.c.metrics.outboundBytesIPv4Total.Add(int64(txBytes))
	case udpAddr.Addr().Is6():
	de.c.metrics.outboundPacketsIPv6Total.Add(int64(len(buffs)))
	de.c.metrics.outboundBytesIPv6Total.Add(int64(txBytes))
	}
	// TODO(raggi): needs updating for accuracy, as in error conditions we may have partial sends.
	if stats := de.c.stats.Load(); err == nil && stats != nil {
	stats.UpdateTxPhysical(de.nodeAddr, udpAddr, len(buffs), txBytes)
	}
	}
	if derpAddr.IsValid() {
	allOk := true
	var txBytes int
	for _, buff := range buffs {
	const isDisco = false
	ok, _ := de.c.sendAddr(derpAddr, de.publicKey, buff, isDisco)
	txBytes += len(buff)
	if !ok {
	allOk = false
	}
	}
	if stats := de.c.stats.Load(); stats != nil {
	stats.UpdateTxPhysical(de.nodeAddr, derpAddr, len(buffs), txBytes)
	}
	if allOk {
	return nil
	}
	}

But doesn’t the system settle after a while, until there’s a network change? You don’t suddenly lose NAT punching and then regain it rapidly, do you? And if you are able to connect directly, then surely we don’t need to send to the DERP relay any more?

[sfllaw]

I think what @raggi is trying to say is that the inner TCP does get back-pressure from magicsock, so setting a large TCP RTO won’t help. Instead, we should probably be tweaking this back-pressure to send better signals to the inner TCP congestion control.

[raggi]

I think what @raggi is trying to say is that the inner TCP does get back-pressure from magicsock, so setting a large TCP RTO won’t help. Instead, we should probably be tweaking this back-pressure to send better signals to the inner TCP congestion control.

A good first step would be to reliably produce some of the would-be real world scenarios so that we can measure efficacy of changes.

The natlab package and tests are the place to start, they're nascent, but start to provide some of the necessary controls.

As for forward packets being dropped, see the code inside DERP itself, in the derp/ package. Packets are sometimes dropped inside DERP, after reception, before being sent on to peers.

FWIW as well, the performance of DERP for end users is generally not constrained at all by this set of problems, it's constrained by strict caps we set on total DERP bandwidth. We have metrics demonstrating clients can and do "eat as much as we give them", so optimization at this layer/from this line of focus is in many cases not going to have an end-user effect.

Where there is a deeper space in this is high latency links, there are a number of problems in that space, but those problems are also more serious in other layers right now. It would be a lot to summarize here, but I'll be happy to hop on a call and provide a brain dump sometime.

[raggi]

Oh, also on packet drops in particular, we've been more keen on adding ECN pass-through as is implemented in kernel Wireguard (so there's precedent), but we've just yet to get around to it. I'd still like to see that be effective of course, and we may need to enable ECN in a few more places / on more platforms.

[sfllaw]

A good first step would be to reliably produce some of the would-be real world scenarios so that we can measure efficacy of changes.

I agree.

FWIW as well, the performance of DERP for end users is generally not constrained at all by this set of problems, it's constrained by strict caps we set on total DERP bandwidth. We have metrics demonstrating clients can and do "eat as much as we give them", so optimization at this layer/from this line of focus is in many cases not going to have an end-user effect.

I started looking into this to improve resource usage, to avoid wasting resources doing nothing. After all, those DERP relays aren’t free to operate. If everyone were better behaved, and users saw better latency or bandwidth, then that would be a happy side effect.

Oh, also on packet drops in particular, we've been more keen on adding ECN pass-through as is implemented in kernel Wireguard (so there's precedent), but we've just yet to get around to it. I'd still like to see that be effective of course, and we may need to enable ECN in a few more places / on more platforms.

Yes, in this model, DERP relays should act as if they are ECN-aware routers.

Aha, I found the discussion between @zx2c4 and @jwhited about implementing ECN: WireGuard/wireguard-go#64 (review)

[raggi]

Do you have some evidence that spurious retransmission rate is high?

[sfllaw]

By dropping from the head of the DERP write queue, we reduce the amount of duplicate work done by TCP retransmission: #16977.

[raggi]

By dropping from the head of the DERP write queue, we reduce the amount of duplicate work done by TCP retransmission: #16977.

all of the logged drops so far were disco, further investigation is required to know if it has any relationship to this bug.