Replace Java DNS resolver with Netty DNS resolver to avoid test timeouts

[nodece]

Some BookKeeper tests occasionally fail due to long delays in reverse DNS lookups. The root cause is the use of InitialDirContext.getAttributes() in org.apache.bookkeeper.net.DNS#reverseDns, which can block for up to 120 seconds if DNS resolution hangs. This leads to flaky or timing-out tests such as:

• Flaky-test: org.apache.bookkeeper.replication.BookieAutoRecoveryTest.testOpenLedgers #4564
• Flaky-test: org.apache.bookkeeper.replication.BookieAutoRecoveryTest.testEmptyLedgerLosesQuorumEventually #4565

To improve reliability, we should replace the current blocking implementation with Netty's asynchronous DNS resolver. This will allow us to set timeouts, avoid blocking test threads, and reduce flakiness.

Test with Netty DNS:

[nodece]

I will make a PR to improve the DNS resolve.

[lhotari]

@nodece Adding -Dsun.net.inetaddr.ttl=1 -Dsun.net.inetaddr.negative.ttl=1 to test JVM args could possible help without switching to Netty DNS resolver. I don't think that DNS resolution itself hangs. It's more likely to be a result of cached addresses. The default negative cache is 10 seconds and default cache is 30 seconds on recent JVM versions.

[lhotari]

I created #4586 to tune the DNS TTL settings for tests and for CI

[lhotari]

Oh, now I see what there is:

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/net/DNS.java

Lines 53 to 103 in bcd6b52

	/**
	* Returns the hostname associated with the specified IP address by the
	* provided nameserver.
	* <p/>
	* Loopback addresses
	*
	* @param hostIp The address to reverse lookup
	* @param ns The host name of a reachable DNS server
	* @return The host name associated with the provided IP
	* @throws NamingException If a NamingException is encountered
	*/
	public static String reverseDns(InetAddress hostIp, String ns)
	throws NamingException {
	//
	// Builds the reverse IP lookup form
	// This is formed by reversing the IP numbers and appending in-addr.arpa
	//
	String[] parts = hostIp.getHostAddress().split("\\.");
	if (parts.length != 4) {
	//Not proper address. May be IPv6
	throw new NamingException("IPV6");
	}
	String reverseIP = parts[3] + "." + parts[2] + "." + parts[1] + "."
	+ parts[0] + ".in-addr.arpa";
	DirContext ictx = new InitialDirContext();
	Attributes attribute;
	try {
	attribute = ictx.getAttributes("dns://" // Use "dns:///" if the default
	+ ((ns == null) ? "" : ns)
	// nameserver is to be used
	+ "/" + reverseIP, new String[]{"PTR"});
	} finally {
	ictx.close();
	}
	if (null == attribute) {
	throw new NamingException("No attribute is found");
	}
	Attribute ptrAttr = attribute.get("PTR");
	if (null == ptrAttr) {
	throw new NamingException("No PTR attribute is found");
	}
	if (null == ptrAttr.get()) {
	throw new NamingException("PTR attribute value is null");
	}
	return ptrAttr.get().toString();
	}

@nodece I'd suggest to make this configurable so that this isn't performed at all when reverse lookup is disabled. Having this type of config mandatory in BookKeeper doesn't make much sense to me.

[nodece]

@lhotari It looks like we can get the hostname by the java.net.InetAddress#getCanonicalHostName.

[nodece]

-Dsun.net.inetaddr.ttl=1 -Dsun.net.inetaddr.negative.ttl=1 works fine.

[lhotari]

@nodece Alternative approach: #4587

[nodece]

Alternative approach: #4588

[nodece]

Closed by #4588