chore(tracking): Test demos on nightly versions for 25.7.0

[NickLarsenNZ]

Part of stackabletech/issues#722

This is testing that the nightly release of the operators and products do not negatively impact
the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

• hourglass: In progress / waiting
• red_circle: In progress with errors
• orange_circle: Completed with minor issues or anomalies
• green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.

• airflow-scheduled-job: adapt to listener changes #247 @maltesander
• data-lakehouse-iceberg-trino-spark @maltesander
• end-to-end-security @maltesander
• hbase-hdfs-load-cycling-data @adwk67
• jupyterhub-keycloak @Techassi
• jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data @razvan
• logging @Techassi
• nifi-kafka-druid-earthquake-data @adwk67
• nifi-kafka-druid-water-level-data @adwk67 fix: Water-level demo, docs and jobs #256
• signal-processing @Techassi
• spark-k8s-anomaly-detection-taxi-data @maltesander
• trino-iceberg @dervoeti
• fix(demo/trino-taxi-data): Use new listener service #254 @NickLarsenNZ

Additional stacks

If we have time (otherwise they can be tested and patched after the release)

• monitoring @sbernauer
• logging
• dual-hive-hdfs-s3
• observability @NickLarsenNZ
• tutorial-openldap
• openldap

Nightly from Scratch Testing Instructions

These instructions are for deploying and completing the nightly demo from scratch.

Tip

Be sure to select the nightly docs version on https://docs.stackable.tech/home/nightly/demos/.

# Install demo (stable operators) for the nightly release.
stackablectl demo install <DEMO_NAME> --release dev

# --- IMPORTANT ---
# Run through the nightly demo instructions (refer to the list above).

[adwk67]

🟢 hbase-hdfs-load-cycling-data

No issues encountered.

[Techassi]

🟢 logging

No issues encountered.

[Techassi]

🟢 signal-processing

I encountered the following errors during a fresh install of the current nightly (soon to be 25.7.0):

The listenerClass field seems to have been moved from the cluser Config to the role-level config. It is interessting, that I didn't encounter this issue during the upgrade testing here: #236 (comment)

Details

An unrecoverable error occured: demo command error

Caused by these errors (recent errors listed first):
 1: failed to install demo "signal-processing"
 2: failed to install stack
 3: failed to install stack manifests
 4: failed to deploy manifests using the kube client
 5: failed to patch/create Kubernetes object
 6: ApiError: failed to create typed patch object (/nifi; nifi.stackable.tech/v1alpha1, Kind=NifiCluster): .spec.clusterConfig.listenerClass: field not declared in schema:  (ErrorResponse { status: "Failure", message: "failed to create typed patch object (/nifi; nifi.stackable.tech/v1alpha1, Kind=NifiCluster): .spec.clusterConfig.listenerClass: field not declared in schema", reason: "", code: 500 })
 7: failed to create typed patch object (/nifi; nifi.stackable.tech/v1alpha1, Kind=NifiCluster): .spec.clusterConfig.listenerClass: field not declared in schema:

---

The NiFi endpoint in the create-nifi-ingestion-job needed to be updated because of the listener changes.

Details

diff --git a/demos/signal-processing/create-nifi-ingestion-job.yaml b/demos/signal-processing/create-nifi-ingestion-job.yaml
index 41c5699..b2722cd 100644
--- a/demos/signal-processing/create-nifi-ingestion-job.yaml
+++ b/demos/signal-processing/create-nifi-ingestion-job.yaml
@@ -74,7 +74,7 @@ data:
     import urllib3
 
     # As of 2022-08-29 we cant use "https://nifi:8443" here because <h2>The request contained an invalid host header [<code>nifi:8443</code>] in the request [<code>/nifi-api</code>]. Check for request manipulation or third-party intercept.</h2>
-    ENDPOINT = f"https://nifi-node-default-0.nifi-node-default.{os.environ['NAMESPACE']}.svc.cluster.local:8443" # For local testing / developing replace it, afterwards change back to f"https://nifi-node-default-0.nifi-node-default.{os.environ['NAMESPACE']}.svc.cluster.local:8443"
+    ENDPOINT = f"https://nifi-node-default-headless.{os.environ['NAMESPACE']}.svc.cluster.local:8443" # For local testing / developing replace it, afterwards change back to f"https://nifi-node-default-headless.{os.environ['NAMESPACE']}.svc.cluster.local:8443"
     USERNAME = "admin"
     PASSWORD = open("/nifi-admin-credentials/admin").read()

---

The issues are fixed by #243.

[adwk67]

🟢 nifi-kafka-druid-earthquake-data

The docs need to be corrected in main (see PR below)

• the Nifi login screenshot seems to be in the wrong place
• the earthquake scatterplot is missing the map as a backdrop (to do with our mapbox key, maybe?)

PR:

• fix: Changes made to earthquake demo and underlying stack #244

[Techassi]

🟢 jupyterhub-keycloak

No issues encountered. It was a little tricky to figure out how to access the JupyterHub interface for which I had to reach out to @adwk67. I will raise a PR to clarify this section in the docs.

PR: #249

[maltesander]

🟢 end-to-end-security

Some listener adaptations required.

PR:

• end-to-end-security: adapt to listener changes #245

[dervoeti]

🟢 trino-iceberg

The listenerClass was moved in the CRD, was fixed by #248

The coordinator-metrics endpoint does not show up anymore when listing the stacklets. I guess this will need to be changed everywhere in our docs. Will create a PR for this.

[maltesander]

🟢 spark-k8s-anomaly-detection-taxi-data

• Added http_scheme: false to the superset asset zip to avoid HTTP instead of HTTPS connection causing this error

ERROR: (builtins.NoneType) None
[SQL: failed to execute: ('Connection aborted.', BadStatusLine('\x15\x03\x03\x00\x02\x02P'))]
(Background on this error at: https://sqlalche.me/e/14/dbapi)

• Adapted listener services

PR:

• fix(spark-anomaly-detection): Adjust Trino HTTPS connection #251