Need improved/HTTPS-centric GKE Ingress + ESP + gRPC example

[bgetsug]

I had a tough time figuring out how to get an Ingress on GKE to behave with ESP, with HTTPS end-to-end. I believe the underlying issues have to do with health checks and readiness probes and how ingress-gke handles those. I commented on the following issue in that project: kubernetes/ingress-gce#18 (comment)

Once health checks can be customized, perhaps an update can be made to esp_echo_gke_ingress.yaml.

[paoesco]

Hi,

same issue here.
As ESP does not allow root path (https://cloud.google.com/endpoints/docs/openapi/openapi-limitations#operations_on_url_root_path) and Ingress GLB does a health check on / (https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer), it seems impossible to configure the GKE Ingress + ESP + gRPC stack.

UPDATE :

• To make it working, I've updated manually the health check endpoint in Compute Engine > Health checks section

UPDATE 2 :
According to this link https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc, Ingress scans for readiness probes and configures automatically its health check endpoint.
It's working for me now

[alethenorio]

@paoesco Can you post how you got Grpc + Esp to work on GKE? I have been struggling finding detailed information and I seem to have hit a wall on the matter.

My health check is always yellow. I have tried configuring readiness probes but it did not help

[paoesco]

And here it is :

apiVersion: v1
kind: Service
metadata:
  name: myapp-svc
spec:
  # NodePort is mandatory for Ingress to perform load balancer
  type: NodePort
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8081
  selector:
    app: myapp-app
  sessionAffinity: None
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp-app
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: myapp-app
  template:
    metadata:
      labels:
        app: myapp-app
    spec:
      containers:
      # ESP container for Google Cloud Endpoints
      # Every requests are caught by ESP first to perform
      # validation/monitoring/authentication and then redirects
      # to our app
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1
        args: [
          "--http_port", "8081",
          "--backend", "127.0.0.1:8080",
          "--service", "myapp.endpoints.PROJECT-XXX.cloud.goog",
          "--rollout_strategy=managed",
          "-z", "healthz"
        ]
        # HEALTH CHECK START
        # Used by ingress to perform health check
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8081
        ports:
          - containerPort: 8081
        # HEALTH CHECK END
      # Our app container
      - name: myapp-app
        image: gcr.io/PROJECT-XXX/myapp-app-image:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

[alethenorio]

Thank you very much but is that gRPC? The esp is configured with http and not http2.

Do you not need to set the esp backend to be grpc://127.0.0.1:8080?

[bgetsug]

I was also never able to get the readiness probe to behave correctly. Perhaps the key is making sure containerPort is set? (kubernetes/ingress-gce#241 (comment)). On ingress-gce@master the docs directory looks very much in flux and the examples directory has been removed. Here's a link to the latest tag of the health checks doc: https://github.com/kubernetes/ingress-gce/blob/v1.4.1/examples/health-checks/README.md. Perhaps that will help? Based on the discussion in kubernetes/ingress-gce#42, I'm still not convinced readiness probes always work.

[alethenorio]

I did configure containerPort and I never saw my load balancer configuration point on the cloud console point to any other port than the http2 port

[alethenorio]

I have done some further digging and found what seems to be a solution for configuration esp + ingress + grpc on GKE

There are 2 issues at hand which were not clear to me in the beginning. Nginx does not seem to support http/2 without ssl so you have to configure SSL on esp. Another problem is that you need to enable to the esp health check to answer on the root path (/)

Let's break down the issue in 2

1. SSL

If we look at how ESP is configured we can see that if you choose ssl (using the --ssl_port flag) instead of --http2_port , it will configure SSL with http2

This of course means you need to mount the certificates in the esp container as per the the esp documentation

You should be able to mount the same secret used by the ingress itself

2. Health Check

I gave up trying to get the readiness probes to work and looked for a way to configure esp to respond on / with 200 while proxying everything else to the backend.

Turns out that this is possible. ESP has a -z (also --healthz) flag to supply which path to respond to health checks on. The issue here is the template looks like this

% if healthz:
    location = /${healthz} {
      return 200;
      access_log off;
    }
% endif

So we cannot supply / as the path in the argument because this would give us location = // { which is not valid.

The trick here is to supply a whitespace as the argument for the health check path.

conclusion

This is how I configured esp on my pod to take care of both issues above

- name: endpoints-proxy
  image: gcr.io/endpoints-release/endpoints-runtime:1
  args: [
  "--ssl_port=8080",
  "--backend=grpc://127.0.0.1:50051",
  "--service=myendpoint.endpoints.myproject-17272.cloud.goog",
  "--rollout_strategy=managed",
  "--service_account_key=/etc/nginx/creds/endpoints-credentials.json",
  "-z",
  " "
  ]
  ports:
  - containerPort: 8080
    protocol: TCP
  volumeMounts:
  - name: ingress-tls
    subPath: tls.crt
    mountPath: /etc/nginx/ssl/nginx.crt
    readOnly: true
  - name: ingress-tls
    subPath: tls.key
    mountPath: /etc/nginx/ssl/nginx.key
    readOnly: true

This worked for me though I have not yet performed extensive testing. I point the service to port 8080 above and ingress to whatever port the service exposes. Ingress then configures the load balancer, it stays healthy and I was able to perform grpc calls on it using the google endpoints domain (once I configured the endpoints to point to the Ingress load balancer IP).

Now if anybody could tell me how to configure GRPC-WEB through google endpoints, that would complete the package

[jcramb]

Did you ever make progress on getting grpc-web support through google endpoints?

[alethenorio]

@jcramb Yes I did.

You can find more details on the grpc-web PR on the esp code.

cloudendpoints/esp#283

It works but only if you use the binary grpc-web wire format which does not support streaming

[unludo]

Hello @alethenorio,
If it's not too much trouble, could you please post the rest of your configuration for grpc endpoint with ssl (service, ingress)? I don't manage to make it work and can't find what I am missing.
thanks

[alethenorio]

@unludo sure

The configurations above are based on the fictional config I posted in my previous post (so, for example, it should be obvious the secret name pointed by ingress is the same secret mounted in the pod)

My service looks like this (assuming the pod it points to has the label app: myApp)

apiVersion: v1
kind: Service
metadata:
  name: myService
  annotations:
    cloud.google.com/app-protocols: '{"http2":"HTTP2"}'
spec:
  type: NodePort
  ports:
  - name: http2
    port: 8080
    targetPort: 8080
    protocol: TCP
  selector:
     app: myApp

And my Ingress looks like this (Assuming use of let's encrypt through cert manager

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myService-ingress-endpoints
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: "http01"
 spec:
   tls:
   - hosts:
     - "myapp.endpoints.myprojectID.cloud.goog"
     secretName: ingress-tls
   backend:
     serviceName: myService
     servicePort: 8080