MultiGet Json exception when only HttpCompression is enabled

[wesley0172]

Elastic.Clients.Elasticsearch version: 9.1.8

Elasticsearch version: 9.1.2

.NET runtime version: 8.0.19

Operating system version: Windows 11 24H2

Description of the problem including expected versus actual behavior:
We have a problem when performing a MultiGetAsync request. A UnexpectedTransportException is returned telling The JSON value could not be converted to Elastic.Clients.Elasticsearch.MultiGetResponse. When we change the MultiGet to a loop with single Get's it works fine, so we're assuming there is something wrong with the MultiGet.

Also, when debugging a found some kind of workaround. by changing from:
var settings = new ElasticsearchClientSettings(new Uri(url))
.Authentication(new BasicAuthentication(username, password))
.EnableHttpCompression();

To:
var settings = new ElasticsearchClientSettings(new Uri(url))
.Authentication(new BasicAuthentication(username, password))
.EnableHttpCompression()
.EnableDebugMode();

It works again
Steps to reproduce:

1. Perform a multiget with only HttpCompression enabled
2. The error is returned
3. Add EnableDebugMode
4. Perform the multiget again
5. No error is return

Expected behavior
I expect the MultiGet to work with only HttpCompression enabled

Provide ConnectionSettings (if relevant):
.Authentication(new BasicAuthentication(username, password))
.EnableHttpCompression()

Provide DebugInformation (if relevant):
Elastic.Transport.UnexpectedTransportException: The JSON value could not be converted to Elastic.Clients.Elasticsearch.MultiGetResponse1[Aster.Maki.Reports.Tickets.TicketActionSearchReport]. Path: $ | LineNumber: 0 | BytePositionInLine: 10.
---> System.Text.Json.JsonException: The JSON value could not be converted to Elastic.Clients.Elasticsearch.MultiGetResponse1[Aster.Maki.Reports.Tickets.TicketActionSearchReport]. Path: $ | LineNumber: 0 | BytePositionInLine: 10. ---> System.InvalidOperationException: Cannot skip tokens on partial JSON. Either get the whole payload and create a Utf8JsonReader instance where isFinalBlock is true or call TrySkip. at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_CannotSkipOnPartial() at System.Text.Json.Utf8JsonReader.Skip() at Elastic.Clients.Elasticsearch.Serialization.JsonUnionSelector.MatchProperty(Utf8JsonReader& reader, JsonSerializerOptions options, String name, UnionTag result) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonUnionSelector.cs:line 122 at Elastic.Clients.Elasticsearch.Serialization.JsonUnionSelector.<>c__DisplayClass7_0.<ByPropertyOfT1>b__0(Utf8JsonReader& r, JsonSerializerOptions o) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonUnionSelector.cs:line 160 at Elastic.Clients.Elasticsearch.Serialization.JsonUnionSelector.Match(Utf8JsonReader& reader, JsonSerializerOptions options, JsonUnionSelectorFunc case1, JsonUnionSelectorFunc case2) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonUnionSelector.cs:line 82 at Elastic.Clients.Elasticsearch.Serialization.JsonUnionSelector.ByPropertyOfT1(Utf8JsonReader& reader, JsonSerializerOptions options, String name) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonUnionSelector.cs:line 159 at Elastic.Clients.Elasticsearch.Core.MGet.Json.MultiGetResponseItemConverter1.<>c.b__0_0(Utf8JsonReader& r, JsonSerializerOptions o) in /_/src/Elastic.Clients.Elasticsearch/Generated/Types/Core/MultiGetResponseItem.Converters.g.cs:line 30
at Elastic.Clients.Elasticsearch.Core.MGet.Json.MultiGetResponseItemConverter1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Types/Core/MultiGetResponseItem.Converters.g.cs:line 31 at Elastic.Clients.Elasticsearch.Serialization.JsonReaderExtensions.<>c__111.b__11_0(Utf8JsonReader& r, JsonSerializerOptions o) in //src/Elastic.Clients.Elasticsearch/Shared/Next/JsonReaderExtensions.cs:line 326
at Elastic.Clients.Elasticsearch.Serialization.JsonReaderExtensions.ReadCollectionValue[T](Utf8JsonReader& reader, JsonSerializerOptions options, JsonReadFunc1 readElement) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonReaderExtensions.cs:line 332 at Elastic.Clients.Elasticsearch.Json.MultiGetResponseConverter1.<>c.b__1_0(Utf8JsonReader& r, JsonSerializerOptions o) in //src/Elastic.Clients.Elasticsearch/_Generated/Api/MultiGetResponse.Converters.g.cs:line 36
at Elastic.Clients.Elasticsearch.Json.MultiGetResponseConverter1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Api/MultiGetResponse.Converters.g.cs:line 32 at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state) --- End of inner exception stack trace --- at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex) at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.ContinueDeserialize(ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, T& value) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.DeserializeAsync(Stream utf8Json, CancellationToken cancellationToken)
at Elastic.Transport.Products.Elasticsearch.ElasticsearchResponseBuilder.SetBodyCoreAsync[TResponse](Boolean isAsync, ApiCallDetails details, BoundConfiguration boundConfiguration, Stream responseStream, CancellationToken cancellationToken)
at Elastic.Transport.DefaultResponseFactory.CreateCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, Exception ex, Nullable1 statusCode, Dictionary2 headers, Stream responseStream, String contentType, Int64 contentLength, IReadOnlyDictionary2 threadPoolStats, IReadOnlyDictionary2 tcpStats, CancellationToken cancellationToken)
at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)
at Elastic.Transport.RequestPipeline.CallProductEndpointCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, Auditor auditor, CancellationToken cancellationToken)
at Elastic.Transport.DistributedTransport1.RequestCoreAsync[TResponse](Boolean isAsync, EndpointPath path, PostData data, Action1 configureActivity, IRequestConfiguration localConfiguration, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elastic.Transport.DistributedTransport1.ThrowUnexpectedTransportException[TResponse](Exception killerException, List1 seenExceptions, Endpoint endpoint, TResponse response, IReadOnlyCollection1 auditTrail) at Elastic.Transport.DistributedTransport1.RequestCoreAsync[TResponse](Boolean isAsync, EndpointPath path, PostData data, Action1 configureActivity, IRequestConfiguration localConfiguration, CancellationToken cancellationToken) at Aster.Maki.Elasticsearch.SearchClient.GetManyAsync[T](IList1 ids) in C:\projects\aster.maki\src\Aster.Maki.Elasticsearch\SearchClient.cs:line 116
at Aster.Maki.Services.Implementation.Tickets.TicketActionSearchQuery.GetManyAsync(List1 actionIds, Boolean showInternalNotes) in C:\projects\aster.maki\src\Aster.Maki.Services\Implementation\Tickets\TicketActionSearchQuery.cs:line 61 at Aster.Maki.Services.Implementation.Tickets.TicketSearchQuery.MapAsync(IList1 tickets, Boolean showInternalNotes) in C:\projects\aster.maki\src\Aster.Maki.Services\Implementation\Tickets\TicketSearchQuery.cs:line 593
at Aster.Maki.Services.Implementation.Tickets.TicketSearchQuery.GetAsync(Guid ticketId, Boolean showInternalNotes) in C:\projects\aster.maki\src\Aster.Maki.Services\Implementation\Tickets\TicketSearchQuery.cs:line 145
at Aster.Maki.Web.Features.Api.Tickets.Ticket.TicketController.Get(Guid ticketId, TicketUrl url, Headers headers) in C:\projects\aster.maki\src\Aster.Maki.Web\Features\Api\Tickets\Ticket\TicketController.cs:line 40
at lambda_method1272(Closure, Object)
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask1 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Aster.Maki.Web.Framework.SinglePageMiddleware.Invoke(HttpContext context) in C:\projects\aster.maki\src\Aster.Maki.Web\Framework\SinglePageMiddleware.cs:line 65 at Aster.Maki.Web.Framework.RequestLoggingMiddleware.Invoke(HttpContext context) in C:\projects\aster.maki\src\Aster.Maki.Web\Framework\RequestLoggingMiddleware.cs:line 26 at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Aster.Maki.Web.Framework.Websocket.WebsocketHandler.Invoke(HttpContext context) in C:\projects\aster.maki\src\Aster.Maki.Web\Framework\Websocket\WebsocketHandler.cs:line 44 at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

[flobernd]

Hi @wesley0172,

could you please share some additional information:

I assume you are using await client.MultiGetAsync(...) to retrieve the documents? Do you also request the source (.Source(true))?

How many documents are you trying to get in a single batch and what is the average size of each document?

I tried to reproduce the issue without success on my side. EnableDebugMode disables streaming in the client which explains why this seems to solve the issue (this will however have negative effects on client memory usage and performance).

[wesley0172]

Hi @flobernd,

Thanks for your reply.

Below is the function we use for the MultiGet:
`public async Task<IList> GetManyAsync(IList ids) where T : class, ISearchReport
{
if (ids == null || ids.Count == 0)
return new List();

var allIds = ids
    .Where(id => id != Guid.Empty)
    .Select(id => id.ToString())
    .Distinct()
    .ToList();

if (allIds.Count == 0)
    return new List<T>();

var index = IndexNameGenerator.Get(typeof(T));

var request = new MultiGetRequest
{
    Docs = allIds.Select(id => new MultiGetOperation
    {
        Id = id,
        Index = index
    }).ToList()
};

MultiGetResponse<T> response;
try
{
    response = await client.MultiGetAsync<T>(request);
}
catch (Exception ex)
{
    log.Error("MultiGetAsync failed", ex);
    throw;
}

if (!response.IsValidResponse || response.Docs is null)
    return new List<T>();

var results = new List<T>();

foreach (var doc in response.Docs)
{
    doc.Match(
        getResult =>
        {
            if (getResult.Found && getResult.Source != null)
            {
                results.Add(getResult.Source);
            }
            else
            {
                log.Warn("Missing or empty document");
            }
        },
        error =>
        {
            log.Warn($"Error retrieving document: {error.Error.Reason}");
        });
}

return results;

}`

We're getting 12 small documents in a single batch approx. 1.5kB to 5kB each.

Other MultiGet calls are working fine, just for this specific ticket we want to get the ticketActions (12 documents) from it's not working.

[wesley0172]

Did some more debugging and found the document that's causing the multiget to fail by excluding the ID before doing the multiget.

This is the document out of elastic:
{ "took": 49, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": 12.900787, "hits": [ { "_index": "aster-maki-ticketactionsearchreport", "_id": "c00cbe75-d7fb-4f82-ba7e-6f028efe098b", "_score": 12.900787, "_source": { "files": [], "ticketId": "57142a06-1214-4546-91d8-ba4832540a8e", "date": "2017-01-24T18:20:21.0702927Z", "newStatus": 4, "isCancelled": false, "textMarkdown": "Ticket met xxx besproken. Wij mogen de ticket sluiten, want xxx pakt dit verder op.", "textNGram": "Ticket met xxx besproken. Wij mogen de ticket sluiten, want x pakt dit verder op.", "textEdgeNGram": "Ticket met xxx besproken. Wij mogen de ticket sluiten, want xxx pakt dit verder op.", "notifyContact": true, "ratingDate": "0001-01-01T00:00:00", "closeTicketAfterReminderIsSent": false, "isInternalNote": false, "id": "c00cbe75-d7fb-4f82-ba7e-6f028efe098b", "typeName": "TicketActionSearchReport" } } ] } }

I don't see anything strange here so i don't know why it fails at this document specifically

[flobernd]

Hi @wesley0172 , thanks for the update. I think I have identified the problem. I'll release a new version soon. Since I can't reproduce the issue locally, I would need your help to confirm the fix after it's live 🙂

[flobernd]

@wesley0172 The new packages 8.19.8 and 9.1.9 should be available on NuGet in the next few minutes.

[wesley0172]

It’s working after the update. Thank you very much for your quick action, we really appreciate it.