Validate endpoint fails for two datasets due to taking longer than max time allowed for AZ Function Apps

[simon-20]

Brief Description
The /api/pub/validate endpoint is currently failing for two datasets, ids: 4ca17edd-429a-4aff-bc27-53214b1cc60a and 86ac46a7-ce96-415f-a33e-8d1390aea79e. These files always fail to validate against the production server, and also fail to validate using the Validator website. They do validate successfully when running js-validator-api locally, but they take a long time: > 5 minutes.

When they do timeout, they usually take about 240 / 250 seconds on both dev and production. This corresponds to the stated built-in timeout for Azure Functions, which is 230 seconds: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-azure-function-activity#timeout-and-long-running-functions

Fixing this issue should address some of the speed up request by @robredpath in this ticket: #543

Severity

Critical. In and of itself for the Validator it would be high, but the problem seems to be causing the validate stage of the Pipeline to crash, for an as-yet unknown reason.

Important Note

There is a bug in curl which means that, given normal usage, these files are truncated if curl is used to POST them to the Validator, such as with the following command:

curl -v -X POST --data @affectedFileName.xml "https://api.iatistandard.org/validator/validate" --header "Ocp-Apim-Subscription-Key: KEYHERE"

Using such a command, the Validator will return 400. This is because curl truncates the file, such that it is no longer valid XML.

curl does this because the problematic files contain some single CR characters, apparently intended as new lines. curl does not handle these files correctly. But it can be forced to do so using the command:

curl -v -X POST --data-binary @affectedFileName.xml "https://api.iatistandard.org/validator/validate" --header "Ocp-Apim-Subscription-Key: KEYHERE"

Note that while I've used the public-facing API endpoints in the examples above, the error occurs on both dev and production when accessing the validator directly using the *.azurewebsites.net URLs.