InferenceLostConnection error for detect step with more than 5000 data points using Multivariate Anomaly Detection API in Python

[csital]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

When I submit more than 5,000 data points for the multivariate anomaly detector detect API call https://{endpoint}/multivariate/models/{model_id}/detect), I consistently get the following error. According to the documentation, the maximum number of data points per inference call is 20,000. There is no error when I use less than 5,000 data points for the API call.

Supplying at most 5,000 data points is much more time-consuming because any single detect API call takes around the same time to complete (regardless of the number of data points) in my experience.

Any log messages given by the failure

Error code: InferenceLostConnection. Message: Lost connection with the worker that processed the inference work. This may be caused by service upgrade. Re-trigger inference should help resolve this.

Expected/desired behavior

No error when I supply at most 20,000 data points for a single detect API call.

Mention any other details that might be useful