fix: add retry logic to OpenSearch setup scripts to handle non-JSON responses

[nilushancosta]

Purpose

openchoreo/openchoreo#2887
The OpenSearch init scripts in observability-logs-opensearch and observability-tracing-opensearch exit prematurely when OpenSearch is still starting up. The health check endpoint returns a non-JSON response during startup, causing jq parsing to fail with a non-zero exit code. Due to set -euo pipefail, this immediately terminates the script, bypassing the retry logic entirely.

Goals

Ensure the init scripts gracefully handle non-JSON responses from the OpenSearch health check endpoint and continue retrying until OpenSearch is fully up or the maximum retry attempts are exhausted.

Approach

• Wrap the jq parsing in a conditional check so that invalid JSON responses do not cause the script to exit.
• If the response is not valid JSON, log an informative message indicating that OpenSearch might still be starting up, and continue the retry loop.

Summary by CodeRabbit

• Chores

  • Updated observability-logs-opensearch Helm chart to version 0.3.11
  • Updated observability-tracing-openobserve Helm chart to version 0.2.1

• Bug Fixes

  • Improved cluster health validation with JSON response parsing checks
  • Enhanced error handling and logging for cluster readiness checks in setup scripts

[coderabbitai]

📝 Walkthrough

Walkthrough

Version bumps for two Helm charts (observability-logs-opensearch to 0.3.11, observability-tracing-openobserve to 0.2.1) reflected in README and Chart.yaml files. Two setup scripts enhanced with JSON validation and curl error handling for improved cluster readiness polling.

Changes

Cohort / File(s)	Summary
Documentation Version Updates observability-logs-opensearch/README.md, observability-tracing-openobserve/README.md	Helm chart version bumps in installation command examples: 0.3.10→0.3.11 and 0.2.0→0.2.1 respectively.
Helm Chart Metadata Updates observability-logs-opensearch/helm/Chart.yaml, observability-tracing-openobserve/helm/Chart.yaml	Version and appVersion fields incremented in chart metadata: 0.3.10→0.3.11 and 0.2.0→0.2.1 respectively.
OpenSearch Setup Script Improvements observability-logs-opensearch/init/setup-opensearch.sh, observability-tracing-opensearch/init/setup-opensearch.sh	Enhanced cluster health polling with JSON validation before parsing and improved curl error handling. Scripts now validate response is valid JSON before extraction and log errors on curl failures before retrying.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 Versions hop along, one step ahead,
Charts and readmes in sync, all properly spread,
But the real magic lies in the scripts so divine—
Checking their JSON, waiting in line,
With errors now caught and resilience defined! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'fix: add retry logic to OpenSearch setup scripts to handle non-JSON responses' directly and accurately describes the main objective of the PR—handling non-JSON responses in OpenSearch scripts through improved error handling.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[coderabbitai]

🧹 Nitpick comments (2)

observability-tracing-opensearch/init/setup-opensearch.sh (1)

31-31: Suppress stdout from JSON validation check.

Same issue as in the logs-opensearch script: jq . prints parsed JSON to stdout when successful.

Proposed fix

-        if echo "$clusterHealth" | jq . 2>/dev/null; then
+        if echo "$clusterHealth" | jq . >/dev/null 2>&1; then

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@observability-tracing-opensearch/init/setup-opensearch.sh` at line 31, The
JSON validation check using `jq .` on the variable `clusterHealth` currently
emits the parsed JSON to stdout; update the conditional that evaluates `if echo
"$clusterHealth" | jq . 2>/dev/null; then` so that successful jq output is
suppressed (e.g., use jq's exit-only mode or redirect stdout to /dev/null) while
preserving stderr redirection; modify the `echo "$clusterHealth" | jq ...`
invocation used in that if-condition (the `clusterHealth` check) to avoid
printing the parsed JSON on success.

observability-logs-opensearch/init/setup-opensearch.sh (1)

34-34: Suppress stdout from JSON validation check.

jq . prints the parsed JSON to stdout when successful, polluting the logs. Redirect both stdout and stderr to /dev/null.

Proposed fix

-        if echo "$clusterHealth" | jq . 2>/dev/null; then
+        if echo "$clusterHealth" | jq . >/dev/null 2>&1; then

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@observability-logs-opensearch/init/setup-opensearch.sh` at line 34, The JSON
validation check using the if condition that pipes echo "$clusterHealth" into jq
(the if echo "$clusterHealth" | jq . invocation) is currently leaving parsed
JSON on stdout; suppress both stdout and stderr from jq by redirecting its
output to /dev/null in that condition so the check only returns the exit status
without polluting logs.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@observability-logs-opensearch/init/setup-opensearch.sh`:
- Line 34: The JSON validation check using the if condition that pipes echo
"$clusterHealth" into jq (the if echo "$clusterHealth" | jq . invocation) is
currently leaving parsed JSON on stdout; suppress both stdout and stderr from jq
by redirecting its output to /dev/null in that condition so the check only
returns the exit status without polluting logs.

In `@observability-tracing-opensearch/init/setup-opensearch.sh`:
- Line 31: The JSON validation check using `jq .` on the variable
`clusterHealth` currently emits the parsed JSON to stdout; update the
conditional that evaluates `if echo "$clusterHealth" | jq . 2>/dev/null; then`
so that successful jq output is suppressed (e.g., use jq's exit-only mode or
redirect stdout to /dev/null) while preserving stderr redirection; modify the
`echo "$clusterHealth" | jq ...` invocation used in that if-condition (the
`clusterHealth` check) to avoid printing the parsed JSON on success.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 620b312b-90ba-40d4-9bd9-0ac07af48f85

📥 Commits

Reviewing files that changed from the base of the PR and between 4144080 and 6ff8a0a.

📒 Files selected for processing (6)

• observability-logs-opensearch/README.md
• observability-logs-opensearch/helm/Chart.yaml
• observability-logs-opensearch/init/setup-opensearch.sh
• observability-tracing-openobserve/README.md
• observability-tracing-openobserve/helm/Chart.yaml
• observability-tracing-opensearch/init/setup-opensearch.sh