feat: make model metrics endpoints configurable

[nayihz]

fix: #16

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nayihz
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	d86effa
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6854b9cdb5b6cb000878ff17
😎 Deploy Preview	https://deploy-preview-1000--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[nirrozenbaum]

I have some doubts about adding additional fields to InferencePool.
there are alternative ways to achieve the same goal, like using command line args.
changes to CRDs should be discussed and get broad agreement using proposals.

cc @kfswain @ahg-g @robscott @danehans @elevran

/hold for others to comment.

[nayihz]

@nirrozenbaum , thanks for you advise. I think you are right.
Command line args or environment variable, which way do you prefer?

[nirrozenbaum]

@nirrozenbaum , thanks for you advise. I think you are right. Command line args or environment variable, which way do you prefer?

@nayihz I would start with command-line args with default values (the existing ones).
we can always iterate if needed.

[elevran]

@nayihz @nirrozenbaum this introduces a fixed endpoint for all model servers in the pool.
Would it make sense to use the Prometheus-formatted annotations as the source of truth when present and fallback to the configuration when missing?
For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"

[nirrozenbaum]

@nayihz @nirrozenbaum this introduces a fixed endpoint for all model servers in the pool. Would it make sense to use the Prometheus-formatted annotations as the source of truth when present and fallback to the configuration when missing? For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080”

@elevran we already have a fixed endpoint, so this PR is not introducing it :). the intention was to make that endpoint configurable.
but your suggestion makes sense as an improvement.

[nirrozenbaum]

/unhold

[kfswain]

Sorry to be a pain, but we may want to consider making these env vars so that they can work more cleanly with our helm charts via https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/config/charts/inferencepool#install-with-custom-environment-variables

We could add support for flags as well, but this is an already established path.

[nayihz]

When user sets both env variables and the flags at the same time, which one has higher priority?

[danehans]

When both command-line flags and environment variables are used to configure the same setting, the standard behavior is to prioritize command-line flags. This is because command-line flags represent explicit, while environment variables often represent more general or persistent configuration. Viper is a good reference implementation of this precedence.

[danehans]

PTAL at https://gist.github.com/danehans/7eae063e205141323ffae5428acb736a for an example implementation snippet.

[kfswain]

looks good for the most part, added a comment about switching to env vars to have better interfacing with our helm chart