docs: Add LMEval TLS documentation (#79)

* docs: Add LMEval authentication tutorial

* Change troubleshooting list to table

Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>

* docs: Add LMEval TLS documentation

Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>

* docs: Update LMEval TLS certificates docs with ConfigMap setup

* fix: linting

* docs: Add link to service account token creation

---------

Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>

Author: ruivieira

docs/modules/ROOT/nav.adoc

@@ -15,6 +15,7 @@
 *** xref:saliency-explanations-with-kserve.adoc[]
 ** xref:lm-eval-tutorial.adoc[]
 *** xref:lmeval-oauth-authentication.adoc[OAuth Authentication]
+*** xref:lmeval-tls-certificates.adoc[TLS Certificates]
 *** xref:lm-eval-tutorial-toxicity.adoc[Toxicity Measurement]
 ** xref:gorch-tutorial.adoc[]
 *** xref:hf-serving-runtime-tutorial.adoc[Using Hugging Face models with GuardrailsOrchestrator]

docs/modules/ROOT/pages/lmeval-tls-certificates.adoc

@@ -0,0 +1,363 @@
+= LMEvalJob with TLS Certificates from OAuth-Protected Models
+:sectnums:
+:icons: font
+
+== Overview
+
+This guide explains how to configure LMEvalJob Custom Resources to use TLS certificates from OAuth-protected KServe InferenceServices for proper SSL verification. Instead of disabling SSL verification (`verify_certificate: "False"`), this approach mounts the actual TLS certificates and CA bundle for secure communication.
+
+== TLS Certificate Architecture
+
+When using TLS certificates with OAuth-protected InferenceServices:
+
+1. **TLS Secret**: Contains the InferenceService's TLS certificate and private key
+2. **CA Bundle ConfigMap**: Contains the Certificate Authority bundle for trust chain
+3. **Volume Mounts**: Mount certificates into the evaluation pod
+4. **SSL Verification**: Configure evaluation client to use mounted certificates
+
+== Step-by-Step Setup
+
+=== Step 1: Identify TLS Resources
+
+First, identify the TLS secret and CA bundle for your InferenceService.
+
+==== Find the TLS Secret
+
+[source,bash]
+----
+kubectl get secret -n $NAMESPACE | grep serving-cert
+----
+
+Expected output shows secrets like `$MODEL-predictor-serving-cert`.
+
+==== Examine the TLS Secret Structure
+
+[source,bash]
+----
+kubectl get secret $MODEL-predictor-serving-cert -n $NAMESPACE -o yaml
+----
+
+The secret contains:
+
+- `tls.crt`: The TLS certificate
+- `tls.key`: The private key
+
+==== Verify CA Bundle ConfigMap
+
+[source,bash]
+----
+kubectl get configmap openshift-service-ca.crt -n openshift-config-managed -o yaml
+----
+
+This contains the `service-ca.crt` key with the Certificate Authority bundle.
+
+IMPORTANT: The `openshift-service-ca.crt` ConfigMap exists in the `openshift-config-managed` namespace, but Kubernetes cannot mount ConfigMaps across namespaces. To use this CA bundle in your LMEvalJob, you have two options:
+
+1. **Copy the ConfigMap** to your target namespace:
++
+[source,bash]
+----
+kubectl get configmap openshift-service-ca.crt -n openshift-config-managed -o yaml | \
+  sed 's/namespace: openshift-config-managed/namespace: $NAMESPACE/' | \
+  kubectl apply -f -
+----
+
+2. **Use OpenShift's service CA injection** (recommended):
++
+Create a ConfigMap with the injection annotation:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: service-ca-bundle
+  namespace: $NAMESPACE
+  annotations:
+    service.beta.openshift.io/inject-cabundle: "true"
+data: {}  # left empty; the operator will add service-ca.crt
+----
++
+Then reference this ConfigMap in your volume configuration.
+
+=== Step 2: Create LMEvalJob with TLS Configuration
+
+Before creating the LMEvalJob, make sure you have created the service CA bundle ConfigMap in your namespace using one of the methods described above.
+
+Create an LMEvalJob that mounts TLS certificates and configures SSL verification.
+
+Create `tls-lmeval-job.yaml`:
+
+[source,yaml]
+----
+apiVersion: trustyai.opendatahub.io/v1alpha1
+kind: LMEvalJob
+metadata:
+  name: tls-eval-job
+  namespace: $NAMESPACE  # <1>
+spec:
+  model: local-completions  # <2>
+  taskList:
+    taskNames: ["mmlu"]
+  logSamples: true
+  batchSize: "1"
+  allowOnline: true
+  allowCodeExecution: true
+  modelArgs:  # <3>
+    - name: model
+      value: granite
+    - name: base_url
+      value: https://$MODEL-predictor.$NAMESPACE.svc.cluster.local:8443/v1/completions  # <4>
+    - name: num_concurrent
+      value: "1"
+    - name: max_retries
+      value: "3"
+    - name: tokenized_requests
+      value: "false"
+    - name: tokenizer
+      value: ibm-granite/granite-7b-instruct
+    - name: verify_certificate
+      value: "/etc/ssl/certs/ca-bundle.crt"  # <5>
+  pod:
+    container:
+      env:
+        - name: OPENAI_API_KEY  # <6>
+          valueFrom:
+            secretKeyRef:
+              name: lmeval-sa-token
+              key: token
+      volumeMounts:  # <7>
+        - name: tls-certs
+          mountPath: /etc/ssl/certs/tls.crt
+          subPath: tls.crt
+          readOnly: true
+        - name: ca-bundle
+          mountPath: /etc/ssl/certs/ca-bundle.crt
+          subPath: service-ca.crt
+          readOnly: true
+    volumes:  # <8>
+      - name: tls-certs
+        secret:
+          secretName: $MODEL-predictor-serving-cert
+      - name: ca-bundle
+        configMap:
+          name: service-ca-bundle
+----
+<1> Replace `$NAMESPACE` with your target namespace
+<2> Use `local-completions` for OpenAI-compatible API endpoints
+<3> Model arguments configure the evaluation client
+<4> HTTPS endpoint of the OAuth-protected InferenceService
+<5> Path to mounted CA bundle for SSL verification
+<6> Service account token for OAuth authentication (see xref:lmeval-oauth-authentication.adoc#_step_2_create_service_account_token_secret[Create Service Account Token Secret])
+<7> Mount TLS certificate and CA bundle into container
+<8> Define volumes for TLS secret and CA bundle ConfigMap
+
+Apply the LMEvalJob:
+
+[source,bash]
+----
+kubectl apply -f tls-lmeval-job.yaml -n $NAMESPACE
+----
+
+== Configuration Reference
+
+=== TLS Certificate Paths
+
+[cols="1,2,1"]
+|===
+|Mount Path |Description |Source
+
+|`/etc/ssl/certs/tls.crt`
+|InferenceService TLS certificate
+|Secret `tls.crt` key
+
+|`/etc/ssl/certs/ca-bundle.crt`
+|Certificate Authority bundle
+|ConfigMap `service-ca.crt` key
+
+|`/etc/ssl/certs/tls.key`
+|Private key (if needed)
+|Secret `tls.key` key
+|===
+
+=== SSL Verification Options
+
+[cols="1,2,1"]
+|===
+|Value |Description |Use Case
+
+|`"False"`
+|Disable SSL verification (insecure)
+|Development only
+
+|`"/etc/ssl/certs/ca-bundle.crt"`
+|Use mounted CA bundle
+|Production with proper certificates
+
+|`"True"`
+|Use system default CA bundle
+|Standard SSL verification
+|===
+
+=== Volume Configuration Patterns
+
+==== TLS Secret Volume
+[source,yaml]
+----
+volumes:
+  - name: tls-certs
+    secret:
+      secretName: $MODEL-predictor-serving-cert
+      items:  # Optional: specify specific keys
+        - key: tls.crt
+          path: tls.crt
+        - key: tls.key
+          path: tls.key
+----
+
+==== CA Bundle ConfigMap Volume
+
+**Option 1: Using Service CA Injection (Recommended)**
+[source,yaml]
+----
+volumes:
+  - name: ca-bundle
+    configMap:
+      name: service-ca-bundle  # <1>
+      items:  # Optional: specify specific keys
+        - key: service-ca.crt
+          path: service-ca.crt
+----
+<1> ConfigMap created with `service.beta.openshift.io/inject-cabundle: "true"` annotation
+
+**Option 2: Using Copied ConfigMap**
+[source,yaml]
+----
+volumes:
+  - name: ca-bundle
+    configMap:
+      name: openshift-service-ca.crt  # <1>
+      items:  # Optional: specify specific keys
+        - key: service-ca.crt
+          path: service-ca.crt
+----
+<1> ConfigMap manually copied from `openshift-config-managed` namespace
+
+==== Volume Mount Options
+[source,yaml]
+----
+volumeMounts:
+  - name: tls-certs
+    mountPath: /etc/ssl/certs/tls.crt
+    subPath: tls.crt  # Mount specific file, not entire volume
+    readOnly: true
+  - name: ca-bundle
+    mountPath: /etc/ssl/certs/ca-bundle.crt
+    subPath: service-ca.crt
+    readOnly: true
+----
+
+== Verification and Troubleshooting
+
+=== Verify TLS Configuration
+
+Check that the LMEvalJob has proper TLS configuration:
+
+[source,bash]
+----
+kubectl get lmevaljob tls-eval-job -n $NAMESPACE -o yaml
+----
+
+Look for:
+
+- `verify_certificate: "/etc/ssl/certs/ca-bundle.crt"` in modelArgs
+- `volumeMounts` section in pod.container
+- `volumes` section in pod
+
+=== Check Pod Volume Mounts
+
+[source,bash]
+----
+kubectl describe pod tls-eval-job -n $NAMESPACE
+----
+
+Verify:
+
+- Volumes are listed in the pod spec
+- Volume mounts are correctly configured
+- No mount conflicts or permission issues
+
+=== Verify Certificate Access
+
+Check that certificates are accessible inside the pod:
+
+[source,bash]
+----
+kubectl exec tls-eval-job -n $NAMESPACE -- ls -la /etc/ssl/certs/
+kubectl exec tls-eval-job -n $NAMESPACE -- cat /etc/ssl/certs/ca-bundle.crt | head -5
+----
+
+=== Check SSL Warnings
+
+Monitor job logs for SSL-related messages:
+
+[source,bash]
+----
+kubectl logs tls-eval-job -n $NAMESPACE | grep -i ssl
+kubectl logs tls-eval-job -n $NAMESPACE | grep -i certificate
+kubectl logs tls-eval-job -n $NAMESPACE | grep -i verify
+----
+
+== Troubleshooting
+
+[cols="1,2,2"]
+|===
+|Problem |Causes |Solution
+
+|Certificate Not Found +
+*(File not found errors for certificate paths)*
+a|* Secret or configmap missing
+* Incorrect volume mount paths
+* Wrong secret/configmap names
+* ConfigMap in wrong namespace
+a|* Verify secret and configmap exist in the same namespace as LMEvalJob
+* Check volume mount paths and subPath values
+* Ensure correct secret/configmap names
+* Use service CA injection or copy ConfigMap to target namespace
+
+|SSL Verification Still Disabled +
+*(`InsecureRequestWarning` in logs, `verify_certificate: False` in modelArgs)*
+a|* Incorrect `verify_certificate` value in modelArgs
+* Certificate path not accessible
+* Wrong CA bundle content
+a|* Set `verify_certificate: "/etc/ssl/certs/ca-bundle.crt"`
+* Verify certificate files are mounted correctly
+* Check CA bundle contains valid certificates
+
+|Permission Denied +
+*(Permission errors accessing mounted certificates)*
+a|* Volume mount permissions
+* Secret and configmap access issues
+* Pod security context restrictions
+a|* Ensure volumes are mounted as `readOnly: true`
+* Check secret and configmap permissions
+* Verify pod security context if applicable
+
+|Certificate Trust Issues +
+*(SSL verification fails even with certificates mounted)*
+a|* Wrong CA bundle for the certificate chain
+* Certificate expired or invalid
+* Trust chain incomplete
+a|* Verify certificate validity: `openssl x509 -in /etc/ssl/certs/tls.crt -text -noout`
+* Check certificate chain matches CA bundle
+* Use correct ConfigMap for your cluster's CA
+
+|Cross-Namespace ConfigMap Error +
+*(ConfigMap `openshift-service-ca.crt` not found in target namespace)*
+a|* Trying to mount ConfigMap from `openshift-config-managed` namespace
+* Kubernetes cannot mount ConfigMaps across namespaces
+* Missing service CA injection setup
+a|* Create ConfigMap with `service.beta.openshift.io/inject-cabundle: "true"` annotation
+* Or copy ConfigMap: `kubectl get configmap openshift-service-ca.crt -n openshift-config-managed -o yaml \| sed 's/namespace: openshift-config-managed/namespace: $NAMESPACE/' \| kubectl apply -f -`
+* Update volume to reference the namespaced ConfigMap
+|===