Attempt next available server returned by Locate v2 API when receiving '503 Service Unavailable'

[hardikbhalodi]

java.net.ProtocolException: Expected HTTP 101 response but was '503 Service Unavailable'

This happening randomly!

[stephen-soltesz]

Hello, @hardikbhalodi - there are a combination of factors resulting in this message.

1. The ndt-server must be able to decide whether to accept a measurement or not based on whether it believes the conditions are good for measurement quality. The ndt-server includes a "transmit controller" that monitors actively used bandwidth and only accepts new measurements when usage is below a configured threshold.
2. The transmit controller does not distinguish between upload or download tests. rate-limiter errors for very fast clients and fallback considerations ndt-server#334
3. The Locate API returns four candidate servers for a client (e.g. https://locate.measurementlab.net/v2/nearest/ndt/ndt7). And, the client library is expected to try the next available server in the event of an error from one. I'm not fluent in Kotlin, but it appears that

   ndt7-client-android/libndt7/src/main/java/net/measurementlab/ndt7/android/NDTTest.kt

   Lines 42 to 71 in 3e831b7

   	getHostname()?.enqueue(
   	object : Callback {
   	override fun onFailure(call: Call, e: IOException) {
   	onFinished(null, e, testType)
   	executorService?.shutdown()
   	runLock.release()
   	}
   	override fun onResponse(call: Call, response: Response) {
   	try {
   	val hostInfo: HostnameResponse = Gson().fromJson(response.body?.string(), HostnameResponse::class.java)
   	val numUrls = hostInfo.results?.size!!
   	for (i in 0 until numUrls) {
   	try {
   	selectTestType(testType, hostInfo.results[i].urls, speedtestLock)
   	return
   	} catch (e: Exception) {
   	if (i == numUrls - 1) throw e
   	}
   	}
   	} catch (e: Exception) {
   	onFinished(null, e, testType)
   	executorService?.shutdown()
   	runLock.release()
   	}
   	}
   	}
   	)
   	}

   errors after a measurement has started do not result in trying the next available result.

Because load on target servers can fluctuate, the errors may appear random from the client perspective. Because load may be higher in some regions than others, the frequency of these errors may be higher or lower for some users than others based only on location, number of users, and available server capacity.

So, on one level, the ndt-server is doing the right thing here. On another, we could be doing a better job differentiating between upload and download tests. And, this community supported library (ndt7-client-android) should do a better job trying the next available server (if in fact it does not today).

@hardikbhalodi - we would welcome changes to the ndt7-client-android library to try the next available server. This would make the client more resilient to transient load on target servers by trying the next available server.

FYI: @robertodauria - this may be helped by adding support to the tx controller to differentiate upload from download. m-lab/ndt-server#334

[stephen-soltesz]

The transmit controller now distinguishes between upload or download tests. Upload requests should not be impacted by recent download measurements. But, the first connection for download could still be subject to the transmit controller and the connection closed.

So, all NDT clients should support trying the next available service URL returned by the Locate API.