gRPC-Web request rejected with 415 Error

[sishbi]

Version

5.0.4

Context

We use Vert.x gRPC Server with gRPC-Web clients and have done so successfully for the past few years.
We use the emissary-ingress to convert from gRPC-Web to gRPC as Vert.x didn't natively support gRPC-Web until recently. TLS is terminated by emissary and we use plain HTTP within our VPC.
We use generated grpc-java/grpc-kotlin server stubs with the Vert.x gRPC Server. And we generate gRPC-Web clients for angular/typescript
This week I upgraded to 5.0.4 and we now get 415 errors when sending a gRPC-Web request from the javascript client.
During the upgrade I had to switch to the 'gRPC IO Server' due to the removal of the MethodDescription support from plain gRPC Server - other than switching the constructor the change was simple.
Reverting back to 4.5.x resolves the issue so this is definitely related to the Vert.x 5.0 upgrade.
All other gRPC client requests (Java/Kotlin and Android native) work fine - only gRPC web requests have this issue.

I tried enabling trace logging level for io.vertx and io.netty but I don't see any log messages that would explain why we get the 415 error.
Looking at the code I see a few references to 415 error, mostly for Content-Type and HTTP version mismatches.

Is there anything I can do to further debug this issue and if there is a workaround you know of then that would be very helpful.
I plan on attaching a debugger to the running server pod in AWS to see if I can trace through to the rejection point.
I may also try to reproduce using a local server, but would also need to setup a local emissary proxy with the same configuration we use in AWS to complete the reproduction which is non-trivial.
Thanks for your support.

Steps to reproduce

1. Upgrade to 5.0.4
2. Send a gRPC-Web request to emissary-ingress
3. emissary-ingress converts the request to gRPC
4. gRPC request is forwarded to gRPC IO Server with a registered gRPC service method
5. gRPC request is rejected with a 415 error

Do you have a reproducer?

No response

[sishbi]

I can see in main branch for 5.x in:

vertx-grpc/vertx-grpc-server/src/main/java/io/vertx/grpc/server/impl/GrpcServerImpl.java

Line 77 in 56172de

GrpcServerRequestInspector.RequestInspectionDetails details = GrpcServerRequestInspector.inspect(httpRequest);

that the HTTP/gRPC request is now validated before processing whereas prior to 5.x no validation happened.
It would be good if the results of the inspection were logged so that I could determine the cause from the logs instead of having to attach a debugger but that is where I will start looking.
I suspect that emissary is not converting the gRPC-Web request correctly, possibly forwarding as HTTP/1.1 or not converting the Content-Type - so I am going to look at emissary logs too.

[sishbi]

The root cause of this issue was in the emissary mapping, the fix was to add:

  add_request_headers:
    content-type:
      append: False
      value: "application/grpc"

For example, here is a complete mapping file:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: my-grpc-service
  labels:
    app: my-grpc-service
    service: my-grpc-service
    protocol: grpc
spec:
  grpc: True
  add_request_headers:
    content-type:
      append: False
      value: "application/grpc"
  prefix: /my.grpc.Service
  rewrite: ""
  service: my-grpc-service

This is required because of a bug in Safari where the grpc-web request was not being handled correctly and caused immediate timeouts. The fix for safari was to add the headers to query parameters.
For example here is the URL to the backend service handled by emissary:
https://my.service/my.grpc.Service/ServiceMethod?%24httpHeaders=Content-Type%3Aapplication%2Fgrpc-web%2Bproto%0D%0AX-User-Agent%3Agrpc-web-javascript%2F0.1%0D%0AX-Grpc-Web%3A1%0D%0A
The issue is that the Content-Type header value in the query param is not mapped to the request and remains unchanged, as seen by the server: Content-Type: application/x-www-form-urlencoded;charset=utf-8
It should have been mapped to application/grpc by emissary, which is what Vert.x requires.

A small PR would be nice to log the reason for the 415 error response, but I didn't see any log messages from vertx for this case. I had to add my own logging in a HTTP handle method that is called before passing the request on to the grpcServer.handle method.