Regular HttpClient Timeouts experienced when running high volume #438
Description
We have 5 isolated durable functions running within individual consumption plans (i.e. 1 slot, per plan), all with Netherite backends. At least once a day we typically experience a lot of the following TaskCanceledException:
Status(StatusCode="Cancelled", Detail="", DebugException="System.OperationCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.") The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. The operation was canceled. The operation was canceled.
App Insights will look like this:
Each function is running Microsoft.Azure.Functions.Worker.Extensions.DurableTask.Netherite 2.1.0, has a max scale-out set to 5, and we have the following host.json configuration for each:
"durableTask": {
"hubName": "<hub name>", // unique per function
"useGracefulShutdown": "true",
"storageProvider": {
"type": "Netherite",
"storageConnectionName": "AzureWebJobsStorage", // unique per function
"eventHubsConnectionName": "EventHubConnectionString" // unique per function
}
}
Is this FASTER db crashing and taking a while to recover? A function restart will often address this issue and allow FASTER db to reinitialize with its task hub but if that doesn't happen for any reason, once the exceptions stop being thrown, the functions can often idle without picking up new work.
The functions themselves process reasonably high volumes (i.e. millions of messages/month) which is why we moved from Azure Table Storage to Netherite, but Netherite is proving to be more unreliable than Table Storage, as it stands.
Can this scenario be mitigated without upgrading to an EP plan? Will we see the same issue even if we upgrade?