Propopal: Add InferenceClass to Inference Pool

[Xunzhuo]

What would you like to be added:

InferenceClass is to support other EPP implementation, like the semantic router, or the inference-plugin we`ve done in AIBrix.

The inference pool is list-watch by different implementation, but the inference class determine which epp implementation to take the inference pool and inference model.

This introduces a way for letting inference api to be like gwapi, the epp provided by GIE is just one of its implementation.

For the Gateway Controller, they only care about the EndpointPickerConfig, create ext-proc filter and cluster, and modify route,cluster in envoy to support routing based on the header/metadata modified by ext-proc server.

For the EPP implementation, they implement the inference pool and model, different implementation may have different focus, the end-user can choose to use the epp implementation they need, and specify the inferenceClass.

As the way goes, the gateway controller introduces a pluggable epp solution, like I can support envoy ai gateway to integrate with GIE or AIBrix inference plugin or semantic router.

GIE can be:

InferenceClass: gie

AIBrix can be:

InferenceClass: aibrix

Semantic Router:

InferenceClass: semantic

When creating the inference pool, when need to specify the inference class, to define which epp to schedule the traffic.

The standard communication from envoyproxy and ext-proc, defined by EndpointPickerConfig, which can be implemented by GW control plane.

Why is this needed:

Support multiple epp implementation

[nirrozenbaum]

@Xunzhuo we have in the InferencePool CRD a field to specify the ExtensionReference.
see here:
https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/api/v1alpha2/inferencepool_types.go#L90

with this field, one can point the traffic to other EPP implementation than GIE.

can you please explain why this field is not answering your needs?

[Xunzhuo]

The reason why I raised this is, even though we have the ExtensionReference, but it is used by the gateway cp to generate the ext-proc config, which controls the envoy to talk to the target ext-proc server, at the same time it is watched by epp as well, no more fields for them to determine if them should reconcile this inferencePool.

[kfswain]

This seems related to the discussion in: #770

TL;DR: Semantic is not an endpoint picking strategy. The way it was presented in our community meetings was semantic routing selected the model, and so there would still need to be an inference specific scheduling algorithm.

I've looked for documentation on an AIBrix inference plugin, and didn't see much in the way of results. If this inference plugin selects endpoints in a way similar to the EPP. What is stopping a user from just using the AIBrix image with the pre-existing config?