From 068575dd87d467267a14bd06e2df2fd60708c770 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Wed, 11 Mar 2026 14:41:54 +0000 Subject: [PATCH 01/17] Add verify-procedure skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a new docs-tools skill that executes AsciiDoc procedures against a live system to prove they work end-to-end. Requires an active connection (oc login, SSH, etc.) — for offline review, use the technical-reviewer agent instead. The Ruby script (verify_proc.rb) parses the .adoc file, extracts numbered steps and their source blocks, then runs each one sequentially. It validates YAML syntax and dry-runs Kubernetes resources, executes bash commands, skips example output and placeholder blocks, and reports a pass/fail summary. Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/SKILL.md | 170 ++++++ .../verify-procedure/scripts/verify_proc.rb | 497 ++++++++++++++++++ 2 files changed, 667 insertions(+) create mode 100644 plugins/docs-tools/skills/verify-procedure/SKILL.md create mode 100644 plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb diff --git a/plugins/docs-tools/skills/verify-procedure/SKILL.md b/plugins/docs-tools/skills/verify-procedure/SKILL.md new file mode 100644 index 00000000..67f56e21 --- /dev/null +++ b/plugins/docs-tools/skills/verify-procedure/SKILL.md @@ -0,0 +1,170 @@ +--- +name: verify-procedure +description: Execute and test AsciiDoc procedures on a live system. Runs every command and validates every YAML block against a real cluster, VM, or host. Requires an active connection to the target system. For static review without a live system, use the docs-tools:technical-reviewer agent instead. +author: Red Hat Documentation Team +allowed-tools: Bash, Read, Edit, Glob +--- + +# Procedure Verification Skill + +This skill executes documented procedures against a live system to prove they work end-to-end. It is the "guided exercise tester" — it runs every command, applies every YAML block, and reports what passes and what breaks. + +**This is not a review tool.** For reviewing documentation quality, prerequisites, and structure without a live system, use the `docs-tools:technical-reviewer` agent. + +## Prerequisites + +You must be connected to the target system before invoking this skill: + +- **OpenShift/Kubernetes**: `oc login` or valid `~/.kube/config` +- **RHEL/Linux**: Local access or SSH session to the target host +- **Ansible**: `ansible --version` succeeds and inventory is accessible + +At invocation, the skill runs a connectivity check (e.g., `oc whoami`). If the check fails, the skill exits and directs the user to log in first or use `docs-tools:technical-reviewer` for offline review. + +## How it works + +1. **Parse**: Read the `.adoc` file and extract all `[source,terminal]`, `[source,bash]`, `[source,yaml]`, and `[source,json]` blocks, associating each with its numbered step. +2. **Execute**: Run the `verify_proc.rb` script against the file: + ```bash + ruby /scripts/verify_proc.rb + ruby /scripts/verify_proc.rb --cleanup + ``` +3. **Report**: Present the script output and flag any additional observations. + +## What the Ruby script does + +The `scripts/verify_proc.rb` script is a procedure runner that processes an AsciiDoc file sequentially: + +### Working directory + +Creates a temporary working directory (`/tmp/verify-proc-*`) for each run. All YAML files referenced in the procedure are saved here, and all bash commands execute with this as their working directory. This ensures: +- Relative file paths in commands (e.g., `oc create -f foo.yaml`) resolve correctly +- The user's working directory is not polluted with temporary files +- Each run is isolated from previous runs + +### Step extraction with hierarchical numbering + +- Parses AsciiDoc numbered steps (`. Step text`, `.. Substep`, `... Sub-substep`) and tracks depth +- Associates each source block with its parent step +- Uses hierarchical step labels: `1`, `1.a`, `1.b`, `2.a.i` instead of flat sequential numbers +- This makes it easy to correlate script output with the actual procedure structure + +### Save-YAML-to-file linking + +Detects the common documentation pattern where a YAML block is preceded by a step like: + +``` +.. Save the following YAML in the `foo.yaml` file: +``` + +When this pattern is detected: +1. The YAML is validated for syntax +2. The YAML is written to `/foo.yaml` +3. Subsequent commands like `oc create -f foo.yaml` find the file and execute successfully + +Filename extraction matches backtick-quoted filenames (`` `foo.yaml` ``) or bare `.yaml`/`.yml` filenames in the step text. + +### Smart skipping + +- **Example output blocks**: Detects blocks preceded by "Example output", "sample output", "expected output", or "output is shown" and skips them +- **Placeholder blocks**: Detects `` patterns, `${VAR}` syntax, `CHANGEME`, or `REPLACE` markers and skips those steps + +### AsciiDoc attribute resolution + +When a source block has `subs="attributes+"`, the script resolves common AsciiDoc attributes before validation or execution: + +- `{product-version}` → detected from the live cluster via `oc version` +- `{product-title}` → `OpenShift Container Platform` +- `{op-system-base}` → `RHEL` +- `{op-system}` → `RHCOS` + +This prevents false YAML validation failures on blocks that contain attribute placeholders. + +### YAML validation + +- Parses every `[source,yaml]` block with Ruby's YAML parser for syntax errors +- If the YAML contains `apiVersion:` (Kubernetes resource), runs `oc apply --dry-run=client` or `kubectl apply --dry-run=client` (auto-detects which CLI is available) +- Reports `[VALID]` or `[FAILURE]` with the specific error + +### JSON validation + +- Parses `[source,json]` blocks with Ruby's JSON parser for syntax errors +- Reports `[VALID]` or `[FAILURE]` + +### Bash execution + +- Strips leading `$ ` prompts from command lines +- Joins backslash-continued lines +- Executes each command via `Open3.capture3` in the working directory +- For verification steps (containing words like "verify", "check", "confirm"), displays the command output +- On failure, logs the error but continues to the next step + +### Best practices check + +- Scans the full file for login/setup patterns (`oc login`, `ssh`, `sudo`, `subscription-manager`, `dnf install`, `ansible-playbook`, etc.) +- Warns if no setup pattern is found in a procedure over 500 characters, suggesting potential "magic steps" + +### Resource tracking and cleanup + +The script tracks resources created during verification: +- Records `oc create -f` / `oc apply -f` commands and their file paths +- Captures resource identifiers from stdout (e.g., `namespace/openshift-ptp created`) + +When invoked with `--cleanup`: +- Deletes tracked resources in reverse order (last created, first deleted) +- Uses `--ignore-not-found` to handle partially-created resources gracefully +- Removes the temporary working directory + +Without `--cleanup`, the working directory and resources are retained so the user can inspect them. + +### Summary + +- Reports total executable steps, pass count, and fail count +- Uses hierarchical step labels matching the procedure structure +- Lists each failed step with its error message +- Flags if no verification step exists in the procedure + +## Output format + +The script produces structured terminal output: + +``` +--- Starting Procedure Validation: --- +[INFO] Working directory: /tmp/verify-proc-abc123 + +[Step 1] Create a namespace for the PTP Operator. + +[Step 1.a] Save the following YAML in the `ptp-namespace.yaml` file: +[VALID] YAML syntax for Step 1.a is correct. +[INFO] Saved YAML to /tmp/verify-proc-abc123/ptp-namespace.yaml +[VALID] Resource logic (dry-run via oc) passed for Step 1.a. + +[Step 1.b] Create the `Namespace` CR: +Executing: oc create -f ptp-namespace.yaml +[SUCCESS] Step 1.b executed. + +[Step 4] To verify that the Operator is installed, enter the following command: +Executing: oc get csv -n openshift-ptp -o custom-columns=... +[SUCCESS] Step 4 executed. +Output: Name Phase +ptp-operator.v4.21.0-... Succeeded +-> Verification successfully performed. + +[Step 4] To verify that the Operator is installed, enter the following command: +[SKIP] Example output - not executed + +============================================================ +FINAL SUMMARY +============================================================ +Total executable steps: 7 +Passed: 7 +Failed: 0 +============================================================ +✓ All steps PASSED +============================================================ + +[INFO] Working directory retained at: /tmp/verify-proc-abc123 +[INFO] Run with --cleanup to auto-delete resources and working directory after verification. +``` + +After the script output, add observations about any patterns noticed during execution (timing issues, missing waits, ordering problems). diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb new file mode 100644 index 00000000..4ccaba7a --- /dev/null +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -0,0 +1,497 @@ +require 'open3' +require 'yaml' +require 'json' +require 'tempfile' +require 'fileutils' +require 'timeout' + +# ProcedureVerifier: Validates AsciiDoc procedures as "guided exercises" +class ProcedureVerifier + def initialize(file_path, cleanup: false) + @file_path = file_path + @content = File.read(file_path) + @results = [] + @cleanup = cleanup + @created_resources = [] # Track resources for cleanup + @workdir = nil + @cli_tool = detect_cli_tool + end + + def run_verification + puts "--- Starting Procedure Validation: #{@file_path} ---" + + # 1. Check for Best Practices (Instructional Design) + check_best_practices + + # 2. Create a clean working directory + @workdir = Dir.mktmpdir('verify-proc-') + puts "[INFO] Working directory: #{@workdir}" + + begin + # 3. Extract and Process Blocks + blocks = extract_code_blocks + + if blocks.empty? + puts "[ERROR] No executable steps or source blocks found." + return + end + + blocks.each do |block| + process_step(block) + end + + summarize + ensure + if @cleanup + run_cleanup + else + puts "\n[INFO] Working directory retained at: #{@workdir}" + puts "[INFO] Run with --cleanup to auto-delete resources and working directory after verification." + end + end + end + + private + + def detect_cli_tool + if system('which oc > /dev/null 2>&1') + 'oc' + elsif system('which kubectl > /dev/null 2>&1') + 'kubectl' + else + nil + end + end + + # Resolve AsciiDoc attribute substitutions like {product-version} + def resolve_attributes(content, has_subs) + return content unless has_subs + + resolved = content.dup + + # Detect cluster version if available + if @cli_tool && resolved.include?('{product-version}') + version = detect_cluster_version + resolved.gsub!('{product-version}', version) if version + end + + # Common attributes + resolved.gsub!('{product-title}', 'OpenShift Container Platform') + resolved.gsub!('{op-system-base}', 'RHEL') + resolved.gsub!('{op-system}', 'RHCOS') + + resolved + end + + def detect_cluster_version + return @cluster_version if defined?(@cluster_version) + + stdout, _, status = Open3.capture3("#{@cli_tool} version -o json 2>/dev/null") + if status.success? + begin + data = JSON.parse(stdout) + # oc version returns serverVersion or openshiftVersion + @cluster_version = data.dig('openshiftVersion')&.match(/^(\d+\.\d+)/)&.[](1) + @cluster_version ||= data.dig('serverVersion', 'minor')&.then { |m| "1.#{m}" } + rescue + @cluster_version = nil + end + else + @cluster_version = nil + end + + @cluster_version + end + + def extract_code_blocks + blocks = [] + lines = @content.lines + i = 0 + # Hierarchical step tracking: [major, sub, subsub] + step_counters = [0, 0, 0] + current_step_label = nil + current_step_depth = 0 + + while i < lines.length + line = lines[i] + + # Match numbered steps (., .., ..., etc.) to track context + if line =~ /^(\.{1,3})\s+(.+)$/ + depth = $1.length # 1 = major, 2 = sub, 3 = subsub + step_text = $2.strip + + # Update counters: increment at this depth, reset deeper levels + step_counters[depth - 1] += 1 + (depth...3).each { |d| step_counters[d] = 0 } + + # Reset sub-counters when a new major step starts + if depth == 1 + step_counters[1] = 0 + step_counters[2] = 0 + elsif depth == 2 + step_counters[2] = 0 + end + + current_step_label = format_step_label(step_counters, depth) + current_step_depth = depth + current_step_text = step_text + end + + # Match source blocks with various formats: + # [source,terminal], [source,bash], [source,yaml] + # [source,terminal,subs="attributes+"], etc. + if line =~ /^\[source,(terminal|bash|yaml|shell|json)(?:,(.*?))?\]\s*$/ + source_type = $1 + source_attrs = $2 || '' + has_subs = source_attrs.include?('subs=') + + # Check if this is an example output block (preceded by "Example output" or similar) + is_example = false + lookback = [i - 1, 0].max + 5.times do + break if lookback < 0 + prev_line = lines[lookback].to_s.downcase + if prev_line.include?("example output") || prev_line.include?("output is shown") || + prev_line.include?("sample output") || prev_line.include?("expected output") + is_example = true + break + end + break if prev_line =~ /^\.{1,3}\s+/ # Stop at previous step + lookback -= 1 + end + + # Find the opening ---- delimiter + i += 1 + while i < lines.length && lines[i] !~ /^----\s*$/ + i += 1 + end + + # Extract content between ---- delimiters + i += 1 + content_lines = [] + while i < lines.length && lines[i] !~ /^----\s*$/ + content_lines << lines[i] + i += 1 + end + + content = content_lines.join.strip + + # Normalize terminal/shell to bash for execution + type = %w[terminal shell].include?(source_type) ? 'bash' : source_type + + # Only add non-empty blocks + unless content.empty? + blocks << { + label: current_step_label || "?", + instruction: current_step_text || "Unknown step", + type: type, + content: content, + is_example: is_example, + has_subs: has_subs + } + end + end + + i += 1 + end + + # Second pass: link YAML blocks to subsequent file-apply commands + link_yaml_to_files(blocks) + + blocks + end + + def format_step_label(counters, depth) + parts = [] + parts << counters[0].to_s if counters[0] > 0 + if depth >= 2 && counters[1] > 0 + parts << ('a'..'z').to_a[counters[1] - 1] + end + if depth >= 3 && counters[2] > 0 + parts << ('i'..'xxvi').to_a[counters[2] - 1] + end + parts.join('.') + end + + # Link "Save YAML to file" blocks with their filenames so subsequent + # oc create/apply commands can find the files + def link_yaml_to_files(blocks) + blocks.each do |block| + next unless block[:type] == 'yaml' + + # Check if the instruction mentions saving to a file + # Patterns: "Save the following YAML in the `foo.yaml` file" + # "Create a file named `foo.yaml`" + # "Save the following YAML as `foo.yaml`" + if block[:instruction] =~ /`([^`]+\.ya?ml)`/ + block[:save_as] = $1 + elsif block[:instruction] =~ /(\S+\.ya?ml)/ + block[:save_as] = $1 + end + end + end + + def check_best_practices + # Ensure no "Magic Steps" - Identify assumed knowledge + # Detect common CLI login/auth patterns across products + login_patterns = [ + 'oc login', # OpenShift + 'ssh ', # Remote access + 'sudo ', # Privilege escalation + 'ansible-navigator', # Ansible + 'ansible-playbook', # Ansible + 'subscription-manager', # RHEL + 'dnf install', # RHEL/Fedora + 'yum install', # RHEL/CentOS + 'kubectl', # Kubernetes + 'export ', # Environment setup + 'source ', # Environment setup + ] + + has_setup = login_patterns.any? { |p| @content.include?(p) } + + if @content.length > 500 && !has_setup + puts "[ADVICE] Warning: No login, environment setup, or tool invocation found. Check for 'magic steps'." + end + end + + def process_step(block) + label = block[:label] + instruction = block[:instruction] + type = block[:type] + content = block[:content] + is_example = block[:is_example] + has_subs = block[:has_subs] + + puts "\n[Step #{label}] #{instruction}" + + if is_example + puts "[SKIP] Example output - not executed" + return + end + + # Resolve AsciiDoc attributes if subs="attributes+" is present + content = resolve_attributes(content, has_subs) + + # Check for placeholders that need user input + if has_placeholders?(content) + puts "[SKIP] Contains placeholders requiring user input" + puts "Content preview: #{content[0..100]}..." + return + end + + case type + when 'yaml' + validate_yaml(content, label, block[:save_as]) + when 'bash' + execute_bash(content, label, instruction) + when 'json' + validate_json(content, label) + end + end + + def has_placeholders?(content) + # Match angle-bracket placeholders containing underscores, spaces, or hyphens + # (e.g., , , ) + # but not single words that are likely legitimate values + # (e.g., , , ,

,
) + content.match?(/<[a-z][a-z0-9]*[_\s-][a-z0-9_\s-]+>/i) || # Multi-word placeholders + content.match?(/\$\{[^}]+\}/) || # Variable placeholders like ${VAR} + content.include?('CHANGEME') || + content.include?('REPLACE') + end + + def validate_yaml(content, label, save_as = nil) + begin + # Lint the YAML for syntax errors + YAML.safe_load(content) + puts "[VALID] YAML syntax for Step #{label} is correct." + @results << { step: label, status: :passed, output: "YAML syntax valid" } + + # Save the YAML to the working directory if a filename was detected + if save_as + dest = File.join(@workdir, save_as) + File.write(dest, content) + puts "[INFO] Saved YAML to #{dest}" + end + + # If it looks like a Kubernetes/OpenShift resource, try dry-run validation + if content.include?("apiVersion:") && @cli_tool + Tempfile.open(['resource', '.yaml']) do |f| + f.write(content) + f.close + stdout, stderr, status = Open3.capture3("#{@cli_tool} apply -f #{f.path} --dry-run=client") + if status.success? + puts "[VALID] Resource logic (dry-run via #{@cli_tool}) passed for Step #{label}." + else + puts "[FAILURE] Resource validation failed: #{stderr}" + # Replace the passed result with a failure + @results.pop + @results << { step: label, status: :failed, error: stderr.strip } + end + end + elsif content.include?("apiVersion:") && !@cli_tool + puts "[SKIP] No oc or kubectl CLI found — skipping resource dry-run for Step #{label}." + end + rescue Psych::SyntaxError => e + puts "[FAILURE] YAML Syntax error in Step #{label}: #{e.message}" + @results << { step: label, status: :failed, error: e.message } + end + end + + def validate_json(content, label) + begin + JSON.parse(content) + puts "[VALID] JSON syntax for Step #{label} is correct." + @results << { step: label, status: :passed, output: "JSON syntax valid" } + rescue JSON::ParserError => e + puts "[FAILURE] JSON Syntax error in Step #{label}: #{e.message}" + @results << { step: label, status: :failed, error: e.message } + end + end + + def execute_bash(command, label, instruction) + # Clean up command: remove $ prompt symbols from each line + clean_command = command.lines.map { |line| line.sub(/^\$ /, '') }.join + + # Handle multi-line commands with backslash continuations + if clean_command.include?('\\') + # Join lines that end with backslash + clean_command = clean_command.gsub(/\\\n\s*/, ' ').strip + else + clean_command = clean_command.strip + end + + puts "Executing: #{clean_command[0..150]}#{clean_command.length > 150 ? '...' : ''}" + + # Run commands in the working directory with a 120-second timeout + begin + stdout, stderr, status = Timeout.timeout(120) do + Open3.capture3(clean_command, chdir: @workdir) + end + rescue Timeout::Error + puts "[FAILURE] Step #{label} timed out after 120 seconds." + @results << { step: label, status: :failed, error: "Command timed out after 120 seconds" } + puts "[WARNING] Continuing with remaining steps despite failure..." + return + end + + if status.success? + puts "[SUCCESS] Step #{label} executed." + + # Track created resources for cleanup + track_resource(clean_command, stdout) + + # Show output if it's a verification/check command + if instruction.downcase.match?(/verify|check|confirm|retrieve|identify/) + puts "Output: #{stdout.strip[0..200]}" unless stdout.strip.empty? + puts "-> Verification successfully performed." + end + + @results << { step: label, status: :passed, output: stdout.strip } + else + puts "[FAILURE] Step #{label} failed." + puts "STDERR: #{stderr.strip}" unless stderr.strip.empty? + puts "STDOUT: #{stdout.strip}" unless stdout.strip.empty? + @results << { step: label, status: :failed, error: stderr.strip } + + # Don't exit immediately - continue with warnings + puts "[WARNING] Continuing with remaining steps despite failure..." + end + end + + # Track resources created by oc/kubectl create/apply for cleanup + def track_resource(command, stdout) + # Match "oc create -f file.yaml" or "oc apply -f file.yaml" + if command =~ /\b(oc|kubectl)\s+(create|apply)\s+-f\s+(\S+)/ + tool = $1 + file = $3 + filepath = File.join(@workdir, file) + if File.exist?(filepath) + @created_resources << { tool: tool, file: filepath } + end + end + + # Match inline resource creation from stdout like "namespace/openshift-ptp created" + if stdout =~ %r{^(\S+/\S+)\s+created} + @created_resources << { resource: $1 } + end + end + + def run_cleanup + puts "\n--- Cleanup ---" + + # Delete resources in reverse order + @created_resources.reverse.each do |res| + if res[:file] + tool = res[:tool] || @cli_tool || 'oc' + puts "Deleting resources from: #{res[:file]}" + stdout, stderr, status = Open3.capture3("#{tool} delete -f #{res[:file]} --ignore-not-found") + if status.success? + puts "[CLEANED] #{stdout.strip}" + else + puts "[WARN] Cleanup failed: #{stderr.strip}" + end + elsif res[:resource] + tool = @cli_tool || 'oc' + puts "Deleting: #{res[:resource]}" + stdout, stderr, status = Open3.capture3("#{tool} delete #{res[:resource]} --ignore-not-found") + if status.success? + puts "[CLEANED] #{stdout.strip}" + else + puts "[WARN] Cleanup failed: #{stderr.strip}" + end + end + end + + # Remove the working directory + FileUtils.rm_rf(@workdir) + puts "[CLEANED] Removed working directory: #{@workdir}" + end + + def summarize + puts "\n" + "="*60 + puts "FINAL SUMMARY" + puts "="*60 + + passed = @results.count { |r| r[:status] == :passed } + failed = @results.count { |r| r[:status] == :failed } + + puts "Total executable steps: #{@results.size}" + puts "Passed: #{passed}" + puts "Failed: #{failed}" + + if failed > 0 + puts "\nFailed steps:" + @results.select { |r| r[:status] == :failed }.each do |result| + puts " - Step #{result[:step]}: #{result[:error]&.split("\n")&.first}" + end + end + + # Check if a global verification example was included + unless @content.downcase.include?("verify") || @content.downcase.include?(".verification") + puts "\n[ADVICE] Consider adding an end-to-end verification step to this procedure." + end + + puts "="*60 + puts passed == @results.size ? "✓ All steps PASSED" : "✗ Some steps FAILED" + puts "="*60 + end +end + +# Execution +if ARGV.empty? + puts "Usage: ruby verify_proc.rb [--cleanup] " + puts " --cleanup Delete created resources and working directory after verification" + exit 1 +end + +cleanup = ARGV.delete('--cleanup') +file_path = ARGV[0] + +unless file_path && File.exist?(file_path) + puts "Error: File not found: #{file_path}" + exit 1 +end + +ProcedureVerifier.new(file_path, cleanup: !!cleanup).run_verification From 09890da06c678e87427eae8fd86c73fd30fdfd99 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Wed, 18 Mar 2026 14:21:24 +0000 Subject: [PATCH 02/17] Fix verify-procedure skill: remove OCP bias, improve file detection, add ifdef warnings Address five validation issues: - Replace noisy magic-steps command scanning with .Prerequisites section check - Make dry-run validation CLI-agnostic (works with both oc and kubectl) - Add ifdef/ifndef conditional warnings instead of silently mis-numbering steps - Broaden file-save detection to match real-world phrasing from openshift-docs - Replace hardcoded OCP attributes with document-extracted attribute resolution Bump docs-tools version to 0.0.11. Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/SKILL.md | 43 +++-- .../verify-procedure/scripts/verify_proc.rb | 167 ++++++++++++------ 2 files changed, 143 insertions(+), 67 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/SKILL.md b/plugins/docs-tools/skills/verify-procedure/SKILL.md index 67f56e21..1b7ed55b 100644 --- a/plugins/docs-tools/skills/verify-procedure/SKILL.md +++ b/plugins/docs-tools/skills/verify-procedure/SKILL.md @@ -51,18 +51,35 @@ Creates a temporary working directory (`/tmp/verify-proc-*`) for each run. All Y ### Save-YAML-to-file linking -Detects the common documentation pattern where a YAML block is preceded by a step like: +Detects steps that instruct the user to save content to a file. The step text doesn't need to contain the word "YAML" — any backtick-quoted or bare filename with a recognized extension is matched. Real-world phrasing varies widely across Red Hat docs: ``` .. Save the following YAML in the `foo.yaml` file: +.. Create a file named load-sctp-module.yaml that contains... +.. Save the following YAML manifest as integration-source-aws-ddb.yaml : +.. Create a route definition called hello-openshift-route.yaml : +.. Create an osc-operatorgroup.yaml manifest file: +.. Create a ra.yaml file that includes the following content: ``` -When this pattern is detected: -1. The YAML is validated for syntax -2. The YAML is written to `/foo.yaml` +Steps without a filename are not matched (no false positives): + +``` +.. Apply the following YAML for a specific backing store: +.. Create a config map in the Velero namespace... +.. Use the following example YAML file to create the deployment: +``` + +Supported extensions: `.yaml`, `.yml`, `.json`, `.conf`, `.cfg`, `.sh`, `.txt`, `.toml`, `.ini`, `.properties` + +When a filename is detected: +1. The content is validated for syntax (YAML or JSON) +2. The content is written to `/` 3. Subsequent commands like `oc create -f foo.yaml` find the file and execute successfully -Filename extraction matches backtick-quoted filenames (`` `foo.yaml` ``) or bare `.yaml`/`.yml` filenames in the step text. +### Conditional directive warnings + +The parser cannot evaluate `ifdef::`/`ifndef::` conditionals. When these directives are encountered, the script emits a warning with the line number so the user knows that step numbering inside the conditional block may be inaccurate. This is a deliberate choice — silently producing wrong step associations is worse than a visible warning. ### Smart skipping @@ -71,14 +88,14 @@ Filename extraction matches backtick-quoted filenames (`` `foo.yaml` ``) or bare ### AsciiDoc attribute resolution -When a source block has `subs="attributes+"`, the script resolves common AsciiDoc attributes before validation or execution: +When a source block has `subs="attributes+"`, the script resolves `{attr-name}` references before validation or execution. Attributes are loaded from two sources, in priority order: + +1. **The document itself** — `:attr-name: value` lines in the `.adoc` file +2. **An `_attributes.adoc` file** — looked up in the same directory as the file, then one level up -- `{product-version}` → detected from the live cluster via `oc version` -- `{product-title}` → `OpenShift Container Platform` -- `{op-system-base}` → `RHEL` -- `{op-system}` → `RHCOS` +This approach works across Red Hat docs repos regardless of which attribute names they use. No attribute names are hardcoded. -This prevents false YAML validation failures on blocks that contain attribute placeholders. +As a special case, `{product-version}` falls back to live cluster detection via `oc version` if no document-level definition is found. ### YAML validation @@ -101,8 +118,8 @@ This prevents false YAML validation failures on blocks that contain attribute pl ### Best practices check -- Scans the full file for login/setup patterns (`oc login`, `ssh`, `sudo`, `subscription-manager`, `dnf install`, `ansible-playbook`, etc.) -- Warns if no setup pattern is found in a procedure over 500 characters, suggesting potential "magic steps" +- Checks for a `.Prerequisites` section (or equivalent heading/anchor) in the document +- Warns if no prerequisites section is found, since its absence is a stronger signal of undocumented assumptions than scanning for specific commands ### Resource tracking and cleanup diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index 4ccaba7a..42c9fd43 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -63,23 +63,52 @@ def detect_cli_tool end end + # Extract AsciiDoc attribute definitions (:attr-name: value) from the document. + # Each docs repo uses different attribute names, so we read them from the source + # rather than hardcoding product-specific values. + def extract_doc_attributes + return @doc_attributes if defined?(@doc_attributes) + + @doc_attributes = {} + + # Parse :attr: value lines from the document + @content.scan(/^:([A-Za-z0-9_-]+):\s*(.+)$/) do |name, value| + @doc_attributes[name] = value.strip + end + + # Also load from an optional attributes file (same dir as the .adoc, or parent) + [File.dirname(@file_path), File.join(File.dirname(@file_path), '..')].each do |dir| + attrs_file = File.join(dir, '_attributes.adoc') + next unless File.exist?(attrs_file) + + File.read(attrs_file).scan(/^:([A-Za-z0-9_-]+):\s*(.+)$/) do |name, value| + @doc_attributes[name] ||= value.strip # doc-level attrs take precedence + end + end + + @doc_attributes + end + # Resolve AsciiDoc attribute substitutions like {product-version} def resolve_attributes(content, has_subs) return content unless has_subs resolved = content.dup - - # Detect cluster version if available - if @cli_tool && resolved.include?('{product-version}') - version = detect_cluster_version - resolved.gsub!('{product-version}', version) if version + attrs = extract_doc_attributes + + # Replace all {attr-name} references with values found in the document + resolved.gsub!(/\{([A-Za-z0-9_-]+)\}/) do |match| + attr_name = $1 + if attrs.key?(attr_name) + attrs[attr_name] + elsif attr_name == 'product-version' && @cli_tool + # Special case: try to detect version from a connected cluster + detect_cluster_version || match + else + match # Leave unresolved attributes as-is + end end - # Common attributes - resolved.gsub!('{product-title}', 'OpenShift Container Platform') - resolved.gsub!('{op-system-base}', 'RHEL') - resolved.gsub!('{op-system}', 'RHCOS') - resolved end @@ -112,9 +141,24 @@ def extract_code_blocks current_step_label = nil current_step_depth = 0 + ifdef_depth = 0 + while i < lines.length line = lines[i] + # Track ifdef/ifndef/endif conditionals — the regex parser cannot + # evaluate these, so warn the user that step associations may be wrong. + if line =~ /^ifn?def::(\S+)\[/ + ifdef_depth += 1 + if ifdef_depth == 1 + puts "[WARNING] Conditional directive at line #{i + 1}: #{line.strip}" + puts " The parser cannot evaluate AsciiDoc conditionals. Steps inside" + puts " this block may be mis-numbered or associated with the wrong context." + end + elsif line =~ /^endif::/ + ifdef_depth -= 1 if ifdef_depth > 0 + end + # Match numbered steps (., .., ..., etc.) to track context if line =~ /^(\.{1,3})\s+(.+)$/ depth = $1.length # 1 = major, 2 = sub, 3 = subsub @@ -213,45 +257,58 @@ def format_step_label(counters, depth) parts.join('.') end - # Link "Save YAML to file" blocks with their filenames so subsequent - # oc create/apply commands can find the files + # Link source blocks to filenames mentioned in the step instruction. + # + # Real-world phrasing varies widely across Red Hat docs. Examples from + # openshift-docs (via NotebookLM analysis): + # + # With filename: + # "Create a file named load-sctp-module.yaml that contains..." + # "Save the following YAML manifest as integration-source-aws-ddb.yaml :" + # "Create a route definition called hello-openshift-route.yaml :" + # "Create an osc-operatorgroup.yaml manifest file:" + # "Create a my-pod.yaml pod manifest..." + # "Create a ra.yaml file that includes the following content:" + # + # Without filename (should NOT match): + # "Apply the following YAML for a specific backing store:" + # "Create a config map in the Velero namespace..." + # "Use the following example YAML file to create the deployment:" + # "See the following example:" + # + # The instruction doesn't need to mention "YAML" — any backtick-quoted or + # bare filename with a recognized extension is matched. + SAVE_FILE_EXTENSIONS = /\.(?:ya?ml|json|conf|cfg|sh|txt|toml|ini|properties)/i + def link_yaml_to_files(blocks) blocks.each do |block| - next unless block[:type] == 'yaml' + next unless %w[yaml json].include?(block[:type]) - # Check if the instruction mentions saving to a file - # Patterns: "Save the following YAML in the `foo.yaml` file" - # "Create a file named `foo.yaml`" - # "Save the following YAML as `foo.yaml`" - if block[:instruction] =~ /`([^`]+\.ya?ml)`/ + instruction = block[:instruction] + + # 1. Backtick-quoted filenames: `foo.yaml` + if instruction =~ /`([^`]+#{SAVE_FILE_EXTENSIONS.source})`/i block[:save_as] = $1 - elsif block[:instruction] =~ /(\S+\.ya?ml)/ + # 2. Bare filenames — must contain a path-like character pattern + # (word chars, hyphens, dots) ending with a recognized extension. + # The \b prevents matching partial words. + elsif instruction =~ /\b([\w][\w.-]*#{SAVE_FILE_EXTENSIONS.source})\b/i block[:save_as] = $1 end end end def check_best_practices - # Ensure no "Magic Steps" - Identify assumed knowledge - # Detect common CLI login/auth patterns across products - login_patterns = [ - 'oc login', # OpenShift - 'ssh ', # Remote access - 'sudo ', # Privilege escalation - 'ansible-navigator', # Ansible - 'ansible-playbook', # Ansible - 'subscription-manager', # RHEL - 'dnf install', # RHEL/Fedora - 'yum install', # RHEL/CentOS - 'kubectl', # Kubernetes - 'export ', # Environment setup - 'source ', # Environment setup - ] - - has_setup = login_patterns.any? { |p| @content.include?(p) } - - if @content.length > 500 && !has_setup - puts "[ADVICE] Warning: No login, environment setup, or tool invocation found. Check for 'magic steps'." + # Check for a .Prerequisites section — its absence is a stronger signal + # of "magic steps" than scanning for specific commands in the procedure body. + # Commands like `oc login`, `sudo`, `ssh` appear in procedure steps themselves, + # so scanning the whole file for them produces false positives. + has_prereqs = @content.match?(/^\.Prerequisites\b/i) || + @content.match?(/^\[id=.*prereq/i) || + @content.match?(/^== Prerequisites/i) + + unless has_prereqs + puts "[ADVICE] No .Prerequisites section found. Verify that prerequisites are documented or linked." end end @@ -315,23 +372,25 @@ def validate_yaml(content, label, save_as = nil) puts "[INFO] Saved YAML to #{dest}" end - # If it looks like a Kubernetes/OpenShift resource, try dry-run validation - if content.include?("apiVersion:") && @cli_tool - Tempfile.open(['resource', '.yaml']) do |f| - f.write(content) - f.close - stdout, stderr, status = Open3.capture3("#{@cli_tool} apply -f #{f.path} --dry-run=client") - if status.success? - puts "[VALID] Resource logic (dry-run via #{@cli_tool}) passed for Step #{label}." - else - puts "[FAILURE] Resource validation failed: #{stderr}" - # Replace the passed result with a failure - @results.pop - @results << { step: label, status: :failed, error: stderr.strip } + # If it looks like a Kubernetes resource, try dry-run validation. + # This works with both oc and kubectl — it's a K8s API feature, not OCP-specific. + if content.include?("apiVersion:") + if @cli_tool + Tempfile.open(['resource', '.yaml']) do |f| + f.write(content) + f.close + stdout, stderr, status = Open3.capture3("#{@cli_tool} apply -f #{f.path} --dry-run=client") + if status.success? + puts "[VALID] Resource dry-run (#{@cli_tool}) passed for Step #{label}." + else + puts "[FAILURE] Resource validation failed: #{stderr}" + @results.pop + @results << { step: label, status: :failed, error: stderr.strip } + end end + else + puts "[SKIP] No oc or kubectl found — skipping resource dry-run for Step #{label}." end - elsif content.include?("apiVersion:") && !@cli_tool - puts "[SKIP] No oc or kubectl CLI found — skipping resource dry-run for Step #{label}." end rescue Psych::SyntaxError => e puts "[FAILURE] YAML Syntax error in Step #{label}: #{e.message}" From a0422f240cb44c910aa420f6028f7ce132031f5e Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Wed, 18 Mar 2026 14:41:52 +0000 Subject: [PATCH 03/17] Add RHEL procedure support to verify-procedure skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extend the skill beyond OCP to handle RHEL documentation patterns: - Parse ini, toml, text, python, ruby source block types - Strip RHEL root prompts (#, [root@host]#, ~]#) in addition to $ - Validate Python and Ruby script blocks for syntax errors - Handle absolute paths (/etc/foo.conf) — validate but never write - Track systemctl and dnf/yum changes for cleanup - Add RHEL examples to SKILL.md from NotebookLM analysis Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/SKILL.md | 36 +++- .../verify-procedure/scripts/verify_proc.rb | 154 +++++++++++++++--- 2 files changed, 160 insertions(+), 30 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/SKILL.md b/plugins/docs-tools/skills/verify-procedure/SKILL.md index 1b7ed55b..1847b686 100644 --- a/plugins/docs-tools/skills/verify-procedure/SKILL.md +++ b/plugins/docs-tools/skills/verify-procedure/SKILL.md @@ -23,7 +23,7 @@ At invocation, the skill runs a connectivity check (e.g., `oc whoami`). If the c ## How it works -1. **Parse**: Read the `.adoc` file and extract all `[source,terminal]`, `[source,bash]`, `[source,yaml]`, and `[source,json]` blocks, associating each with its numbered step. +1. **Parse**: Read the `.adoc` file and extract all source blocks (`[source,terminal]`, `[source,bash]`, `[source,yaml]`, `[source,json]`, `[source,ini]`, `[source,toml]`, `[source,text]`, `[source,python]`, `[source,ruby]`), associating each with its numbered step. 2. **Execute**: Run the `verify_proc.rb` script against the file: ```bash ruby /scripts/verify_proc.rb @@ -54,12 +54,17 @@ Creates a temporary working directory (`/tmp/verify-proc-*`) for each run. All Y Detects steps that instruct the user to save content to a file. The step text doesn't need to contain the word "YAML" — any backtick-quoted or bare filename with a recognized extension is matched. Real-world phrasing varies widely across Red Hat docs: ``` +OCP examples: .. Save the following YAML in the `foo.yaml` file: .. Create a file named load-sctp-module.yaml that contains... .. Save the following YAML manifest as integration-source-aws-ddb.yaml : .. Create a route definition called hello-openshift-route.yaml : -.. Create an osc-operatorgroup.yaml manifest file: -.. Create a ra.yaml file that includes the following content: + +RHEL examples: +.. Edit the `/etc/chrony.conf` configuration file: +.. Create a playbook file, for example ~/playbook.yml, with the following content: +.. Create a YAML file named sap-netweaver.yml with the following content: +.. Create a configuration file named `default` and add it to the `pxelinux.cfg/` directory: ``` Steps without a filename are not matched (no false positives): @@ -70,6 +75,8 @@ Steps without a filename are not matched (no false positives): .. Use the following example YAML file to create the deployment: ``` +Absolute paths (e.g., `/etc/chrony.conf`) are validated for syntax but never written to the filesystem — the script should not modify system files during verification. + Supported extensions: `.yaml`, `.yml`, `.json`, `.conf`, `.cfg`, `.sh`, `.txt`, `.toml`, `.ini`, `.properties` When a filename is detected: @@ -108,9 +115,19 @@ As a special case, `{product-version}` falls back to live cluster detection via - Parses `[source,json]` blocks with Ruby's JSON parser for syntax errors - Reports `[VALID]` or `[FAILURE]` +### Config and script validation + +- `[source,ini]`, `[source,toml]`, `[source,text]` blocks are recorded and saved to the working directory (for relative paths) or validated without writing (for absolute paths like `/etc/chrony.conf`) +- `[source,python]` blocks are syntax-checked via `python3 -c "compile(...)"` +- `[source,ruby]` blocks are syntax-checked via `ruby -c` +- Absolute paths in step instructions (common in RHEL procedures) are validated but never written to the filesystem + ### Bash execution -- Strips leading `$ ` prompts from command lines +- Strips prompt symbols from command lines, handling conventions across products: + - OCP/K8s: `$ oc get pods` + - RHEL root: `# dnf install`, `[root@host ~]# systemctl`, `~]# subscription-manager` + - Mixed: `$ sudo dnf install` - Joins backslash-continued lines - Executes each command via `Open3.capture3` in the working directory - For verification steps (containing words like "verify", "check", "confirm"), displays the command output @@ -123,13 +140,14 @@ As a special case, `{product-version}` falls back to live cluster detection via ### Resource tracking and cleanup -The script tracks resources created during verification: -- Records `oc create -f` / `oc apply -f` commands and their file paths -- Captures resource identifiers from stdout (e.g., `namespace/openshift-ptp created`) +The script tracks resources created during verification across products: +- **K8s/OCP**: Records `oc create -f` / `oc apply -f` commands and captures resource identifiers from stdout (e.g., `namespace/openshift-ptp created`) +- **RHEL**: Tracks `systemctl enable/start` services and `dnf/yum install` packages When invoked with `--cleanup`: -- Deletes tracked resources in reverse order (last created, first deleted) -- Uses `--ignore-not-found` to handle partially-created resources gracefully +- Deletes K8s resources in reverse order using `--ignore-not-found` +- Stops and disables tracked services via `systemctl` +- Removes installed packages via `dnf remove` - Removes the temporary working directory Without `--cleanup`, the working directory and resources are retained so the user can inspect them. diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index 42c9fd43..b760f497 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -181,10 +181,11 @@ def extract_code_blocks current_step_text = step_text end - # Match source blocks with various formats: - # [source,terminal], [source,bash], [source,yaml] - # [source,terminal,subs="attributes+"], etc. - if line =~ /^\[source,(terminal|bash|yaml|shell|json)(?:,(.*?))?\]\s*$/ + # Match source blocks with various formats across Red Hat products: + # OCP/K8s: [source,terminal], [source,bash], [source,yaml], [source,json] + # RHEL: [source,ini], [source,toml], [source,text], [source,python] + # General: [source,shell], [source,ruby] + if line =~ /^\[source,(terminal|bash|yaml|shell|json|ini|toml|text|python|ruby)(?:,(.*?))?\]\s*$/ source_type = $1 source_attrs = $2 || '' has_subs = source_attrs.include?('subs=') @@ -220,8 +221,12 @@ def extract_code_blocks content = content_lines.join.strip - # Normalize terminal/shell to bash for execution - type = %w[terminal shell].include?(source_type) ? 'bash' : source_type + # Normalize executable types to bash; keep data formats as-is for validation + type = case source_type + when 'terminal', 'shell' then 'bash' + when 'ini', 'toml', 'text' then 'config' + else source_type + end # Only add non-empty blocks unless content.empty? @@ -282,19 +287,26 @@ def format_step_label(counters, depth) def link_yaml_to_files(blocks) blocks.each do |block| - next unless %w[yaml json].include?(block[:type]) + next unless %w[yaml json config].include?(block[:type]) instruction = block[:instruction] - # 1. Backtick-quoted filenames: `foo.yaml` - if instruction =~ /`([^`]+#{SAVE_FILE_EXTENSIONS.source})`/i + # 1. Backtick-quoted paths: `/etc/chrony.conf` or `foo.yaml` + if instruction =~ /`([^`]*#{SAVE_FILE_EXTENSIONS.source})`/i block[:save_as] = $1 - # 2. Bare filenames — must contain a path-like character pattern - # (word chars, hyphens, dots) ending with a recognized extension. - # The \b prevents matching partial words. + # 2. Backtick-quoted absolute paths without recognized extension: + # e.g., `~/playbook.yml`, `/etc/sysctl.d/99-custom` + elsif instruction =~ /`((?:\/|~\/)[^`]+)`/ + block[:save_as] = $1 + # 3. Bare filenames — word chars, hyphens, dots ending with recognized extension elsif instruction =~ /\b([\w][\w.-]*#{SAVE_FILE_EXTENSIONS.source})\b/i block[:save_as] = $1 end + + # For absolute paths, flag them — they can't be saved to workdir safely + if block[:save_as] && block[:save_as].start_with?('/') + block[:absolute_path] = true + end end end @@ -344,6 +356,10 @@ def process_step(block) execute_bash(content, label, instruction) when 'json' validate_json(content, label) + when 'config' + validate_config(content, label, block[:save_as]) + when 'python', 'ruby' + validate_script(content, label, type, block[:save_as]) end end @@ -365,11 +381,17 @@ def validate_yaml(content, label, save_as = nil) puts "[VALID] YAML syntax for Step #{label} is correct." @results << { step: label, status: :passed, output: "YAML syntax valid" } - # Save the YAML to the working directory if a filename was detected + # Save the YAML to the working directory if a filename was detected. + # Absolute paths (e.g., /etc/foo.conf from RHEL procedures) are validated + # but not written — the script should not modify system files. if save_as - dest = File.join(@workdir, save_as) - File.write(dest, content) - puts "[INFO] Saved YAML to #{dest}" + if save_as.start_with?('/') || save_as.start_with?('~/') + puts "[INFO] Absolute path #{save_as} — YAML validated but not written to filesystem." + else + dest = File.join(@workdir, File.basename(save_as)) + File.write(dest, content) + puts "[INFO] Saved YAML to #{dest}" + end end # If it looks like a Kubernetes resource, try dry-run validation. @@ -409,9 +431,71 @@ def validate_json(content, label) end end + # Validate config file content (INI, TOML, plain text). + # For RHEL procedures that edit files like /etc/chrony.conf, systemd units, etc. + # We validate what we can (TOML syntax) and record the rest as seen. + def validate_config(content, label, save_as = nil) + puts "[VALID] Configuration content for Step #{label} recorded." + @results << { step: label, status: :passed, output: "Config content recorded" } + + if save_as + if save_as.start_with?('/') || save_as.start_with?('~/') + puts "[INFO] Absolute path #{save_as} — content validated but not written to filesystem." + else + dest = File.join(@workdir, File.basename(save_as)) + File.write(dest, content) + puts "[INFO] Saved config to #{dest}" + end + end + end + + # Validate script content (Python, Ruby) for syntax errors without executing. + def validate_script(content, label, lang, save_as = nil) + case lang + when 'python' + # Use python3 -c "compile()" for syntax check + stdout, stderr, status = Open3.capture3('python3', '-c', "compile(#{content.inspect}, '', 'exec')") + if status.success? + puts "[VALID] Python syntax for Step #{label} is correct." + @results << { step: label, status: :passed, output: "Python syntax valid" } + else + puts "[FAILURE] Python syntax error in Step #{label}: #{stderr.strip}" + @results << { step: label, status: :failed, error: stderr.strip } + end + when 'ruby' + Tempfile.open(['step', '.rb']) do |f| + f.write(content) + f.close + stdout, stderr, status = Open3.capture3("ruby -c #{f.path}") + if status.success? + puts "[VALID] Ruby syntax for Step #{label} is correct." + @results << { step: label, status: :passed, output: "Ruby syntax valid" } + else + puts "[FAILURE] Ruby syntax error in Step #{label}: #{stderr.strip}" + @results << { step: label, status: :failed, error: stderr.strip } + end + end + end + + if save_as && !(save_as.start_with?('/') || save_as.start_with?('~/')) + dest = File.join(@workdir, File.basename(save_as)) + File.write(dest, content) + puts "[INFO] Saved script to #{dest}" + end + end + def execute_bash(command, label, instruction) - # Clean up command: remove $ prompt symbols from each line - clean_command = command.lines.map { |line| line.sub(/^\$ /, '') }.join + # Clean up command: remove prompt symbols from each line. + # Handles prompts across Red Hat products: + # OCP/K8s: $ oc get pods + # RHEL: # dnf install, [root@host ~]# systemctl, ~]# subscription-manager + # Mixed: $ sudo dnf install + clean_command = command.lines.map do |line| + line + .sub(/^\[[\w@.\-]+ [~\w\/]*\][#$]\s*/, '') # [root@host ~]# or [user@host dir]$ + .sub(/^~\][#$]\s*/, '') # ~]# or ~]$ + .sub(/^[#$]\s/, '') # bare # or $ prompt + end.join # Handle multi-line commands with backslash continuations if clean_command.include?('\\') @@ -459,9 +543,10 @@ def execute_bash(command, label, instruction) end end - # Track resources created by oc/kubectl create/apply for cleanup + # Track resources created during verification for cleanup. + # Handles both K8s resources (oc/kubectl) and RHEL system changes (systemctl, dnf). def track_resource(command, stdout) - # Match "oc create -f file.yaml" or "oc apply -f file.yaml" + # K8s: "oc create -f file.yaml" or "oc apply -f file.yaml" if command =~ /\b(oc|kubectl)\s+(create|apply)\s+-f\s+(\S+)/ tool = $1 file = $3 @@ -471,10 +556,21 @@ def track_resource(command, stdout) end end - # Match inline resource creation from stdout like "namespace/openshift-ptp created" + # K8s: inline resource creation from stdout like "namespace/openshift-ptp created" if stdout =~ %r{^(\S+/\S+)\s+created} @created_resources << { resource: $1 } end + + # RHEL: systemctl enable/start — track for disable/stop on cleanup + if command =~ /\bsystemctl\s+(enable|start|enable\s+--now)\s+(\S+)/ + @created_resources << { service: $2, action: $1 } + end + + # RHEL: dnf/yum install — track for removal on cleanup + if command =~ /\b(?:dnf|yum)\s+install\s+(?:-y\s+)?(.+)/ + packages = $1.strip.split(/\s+/).reject { |p| p.start_with?('-') } + @created_resources << { packages: packages } unless packages.empty? + end end def run_cleanup @@ -500,6 +596,22 @@ def run_cleanup else puts "[WARN] Cleanup failed: #{stderr.strip}" end + elsif res[:service] + # RHEL: stop and disable services that were started/enabled + puts "Stopping service: #{res[:service]}" + Open3.capture3("sudo systemctl stop #{res[:service]}") + Open3.capture3("sudo systemctl disable #{res[:service]}") + puts "[CLEANED] Service #{res[:service]} stopped and disabled." + elsif res[:packages] + # RHEL: remove packages that were installed + pkg_list = res[:packages].join(' ') + puts "Removing packages: #{pkg_list}" + stdout, stderr, status = Open3.capture3("sudo dnf remove -y #{pkg_list}") + if status.success? + puts "[CLEANED] Packages removed." + else + puts "[WARN] Package removal failed: #{stderr.strip}" + end end end From 20545872bbb2af81333e57a0cde9d954494732c5 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Wed, 18 Mar 2026 15:05:40 +0000 Subject: [PATCH 04/17] Fix dry-run failure overwriting YAML syntax pass result When no cluster is connected, the dry-run via oc/kubectl fails with connection errors. Previously this replaced the YAML syntax pass with a failure (via @results.pop), making all valid YAML blocks appear broken. Now: - Connection errors (no such host, connection refused) skip the dry-run gracefully while preserving the syntax pass - Genuine resource errors (invalid field, unknown kind) are recorded as a separate result instead of replacing the syntax check Before: 0 passed / 7 failed (no cluster) After: 3 passed / 4 failed (YAML syntax passes preserved) Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/scripts/verify_proc.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index b760f497..813398f1 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -396,6 +396,8 @@ def validate_yaml(content, label, save_as = nil) # If it looks like a Kubernetes resource, try dry-run validation. # This works with both oc and kubectl — it's a K8s API feature, not OCP-specific. + # Dry-run is a bonus check — it does NOT replace the syntax result. + # If there's no cluster connection, the syntax pass still stands. if content.include?("apiVersion:") if @cli_tool Tempfile.open(['resource', '.yaml']) do |f| @@ -404,10 +406,13 @@ def validate_yaml(content, label, save_as = nil) stdout, stderr, status = Open3.capture3("#{@cli_tool} apply -f #{f.path} --dry-run=client") if status.success? puts "[VALID] Resource dry-run (#{@cli_tool}) passed for Step #{label}." + elsif stderr.include?("Unable to connect") || stderr.include?("no such host") || stderr.include?("connection refused") + # No cluster connectivity — don't fail the YAML validation for this + puts "[SKIP] No cluster connection — dry-run skipped for Step #{label}." else - puts "[FAILURE] Resource validation failed: #{stderr}" - @results.pop - @results << { step: label, status: :failed, error: stderr.strip } + # Genuine resource validation error (e.g., invalid field, unknown kind) + puts "[FAILURE] Resource validation failed: #{stderr.strip.lines.first}" + @results << { step: "#{label}-dryrun", status: :failed, error: stderr.strip } end end else From 594d0896bbcfd65b32476a6808f595044e28d298 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Fri, 20 Mar 2026 13:59:55 +0000 Subject: [PATCH 05/17] Fix CodeRabbit issues in verify-procedure skill - Fix Roman numeral labels: replace ('i'..'xxvi').to_a (produces i,j,k) with explicit Roman numeral array - Fix extracted file writes: pass save_as to validate_json, preserve nested paths instead of flattening with File.basename, mkdir_p parents - Fix command timeout: add run_command_with_timeout using popen3 with explicit TERM/KILL to terminate child processes on timeout - Fix package manager tracking: capture dnf vs yum during install and use the same manager during cleanup - Fix systemctl regex: reorder alternation so 'enable --now' matches before 'enable', check exit codes during cleanup - Add language identifiers to unlabeled fenced code blocks in SKILL.md - Use relative script path in SKILL.md per repo conventions Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/SKILL.md | 10 +-- .../verify-procedure/scripts/verify_proc.rb | 89 ++++++++++++++----- 2 files changed, 74 insertions(+), 25 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/SKILL.md b/plugins/docs-tools/skills/verify-procedure/SKILL.md index 1847b686..3652240a 100644 --- a/plugins/docs-tools/skills/verify-procedure/SKILL.md +++ b/plugins/docs-tools/skills/verify-procedure/SKILL.md @@ -26,8 +26,8 @@ At invocation, the skill runs a connectivity check (e.g., `oc whoami`). If the c 1. **Parse**: Read the `.adoc` file and extract all source blocks (`[source,terminal]`, `[source,bash]`, `[source,yaml]`, `[source,json]`, `[source,ini]`, `[source,toml]`, `[source,text]`, `[source,python]`, `[source,ruby]`), associating each with its numbered step. 2. **Execute**: Run the `verify_proc.rb` script against the file: ```bash - ruby /scripts/verify_proc.rb - ruby /scripts/verify_proc.rb --cleanup + ruby scripts/verify_proc.rb + ruby scripts/verify_proc.rb --cleanup ``` 3. **Report**: Present the script output and flag any additional observations. @@ -53,7 +53,7 @@ Creates a temporary working directory (`/tmp/verify-proc-*`) for each run. All Y Detects steps that instruct the user to save content to a file. The step text doesn't need to contain the word "YAML" — any backtick-quoted or bare filename with a recognized extension is matched. Real-world phrasing varies widely across Red Hat docs: -``` +```text OCP examples: .. Save the following YAML in the `foo.yaml` file: .. Create a file named load-sctp-module.yaml that contains... @@ -69,7 +69,7 @@ RHEL examples: Steps without a filename are not matched (no false positives): -``` +```text .. Apply the following YAML for a specific backing store: .. Create a config map in the Velero namespace... .. Use the following example YAML file to create the deployment: @@ -163,7 +163,7 @@ Without `--cleanup`, the working directory and resources are retained so the use The script produces structured terminal output: -``` +```text --- Starting Procedure Validation: --- [INFO] Working directory: /tmp/verify-proc-abc123 diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index 813398f1..afd8fe47 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -257,7 +257,8 @@ def format_step_label(counters, depth) parts << ('a'..'z').to_a[counters[1] - 1] end if depth >= 3 && counters[2] > 0 - parts << ('i'..'xxvi').to_a[counters[2] - 1] + roman = %w[i ii iii iv v vi vii viii ix x xi xii xiii xiv xv xvi xvii xviii xix xx xxi xxii xxiii xxiv xxv xxvi] + parts << roman[counters[2] - 1] end parts.join('.') end @@ -355,7 +356,7 @@ def process_step(block) when 'bash' execute_bash(content, label, instruction) when 'json' - validate_json(content, label) + validate_json(content, label, block[:save_as]) when 'config' validate_config(content, label, block[:save_as]) when 'python', 'ruby' @@ -388,7 +389,8 @@ def validate_yaml(content, label, save_as = nil) if save_as.start_with?('/') || save_as.start_with?('~/') puts "[INFO] Absolute path #{save_as} — YAML validated but not written to filesystem." else - dest = File.join(@workdir, File.basename(save_as)) + dest = File.join(@workdir, save_as) + FileUtils.mkdir_p(File.dirname(dest)) File.write(dest, content) puts "[INFO] Saved YAML to #{dest}" end @@ -425,11 +427,22 @@ def validate_yaml(content, label, save_as = nil) end end - def validate_json(content, label) + def validate_json(content, label, save_as = nil) begin JSON.parse(content) puts "[VALID] JSON syntax for Step #{label} is correct." @results << { step: label, status: :passed, output: "JSON syntax valid" } + + if save_as + if save_as.start_with?('/') || save_as.start_with?('~/') + puts "[INFO] Absolute path #{save_as} — JSON validated but not written to filesystem." + else + dest = File.join(@workdir, save_as) + FileUtils.mkdir_p(File.dirname(dest)) + File.write(dest, content) + puts "[INFO] Saved JSON to #{dest}" + end + end rescue JSON::ParserError => e puts "[FAILURE] JSON Syntax error in Step #{label}: #{e.message}" @results << { step: label, status: :failed, error: e.message } @@ -447,7 +460,8 @@ def validate_config(content, label, save_as = nil) if save_as.start_with?('/') || save_as.start_with?('~/') puts "[INFO] Absolute path #{save_as} — content validated but not written to filesystem." else - dest = File.join(@workdir, File.basename(save_as)) + dest = File.join(@workdir, save_as) + FileUtils.mkdir_p(File.dirname(dest)) File.write(dest, content) puts "[INFO] Saved config to #{dest}" end @@ -483,12 +497,41 @@ def validate_script(content, label, lang, save_as = nil) end if save_as && !(save_as.start_with?('/') || save_as.start_with?('~/')) - dest = File.join(@workdir, File.basename(save_as)) + dest = File.join(@workdir, save_as) + FileUtils.mkdir_p(File.dirname(dest)) File.write(dest, content) puts "[INFO] Saved script to #{dest}" end end + def run_command_with_timeout(command, chdir:, timeout:) + stdout = +'' + stderr = +'' + status = nil + + Open3.popen3(command, chdir: chdir) do |stdin, out, err, wait_thr| + stdin.close + out_reader = Thread.new { stdout << out.read } + err_reader = Thread.new { stderr << err.read } + + begin + Timeout.timeout(timeout) { status = wait_thr.value } + rescue Timeout::Error + Process.kill('TERM', wait_thr.pid) rescue nil + Process.kill('KILL', wait_thr.pid) rescue nil + Process.wait(wait_thr.pid) rescue nil + raise + ensure + out.close unless out.closed? + err.close unless err.closed? + out_reader.join + err_reader.join + end + end + + [stdout, stderr, status] + end + def execute_bash(command, label, instruction) # Clean up command: remove prompt symbols from each line. # Handles prompts across Red Hat products: @@ -512,11 +555,11 @@ def execute_bash(command, label, instruction) puts "Executing: #{clean_command[0..150]}#{clean_command.length > 150 ? '...' : ''}" - # Run commands in the working directory with a 120-second timeout + # Run commands in the working directory with a 120-second timeout. + # Uses popen3 with explicit TERM/KILL to ensure the child process + # is terminated on timeout (not just the Ruby caller). begin - stdout, stderr, status = Timeout.timeout(120) do - Open3.capture3(clean_command, chdir: @workdir) - end + stdout, stderr, status = run_command_with_timeout(clean_command, chdir: @workdir, timeout: 120) rescue Timeout::Error puts "[FAILURE] Step #{label} timed out after 120 seconds." @results << { step: label, status: :failed, error: "Command timed out after 120 seconds" } @@ -567,14 +610,15 @@ def track_resource(command, stdout) end # RHEL: systemctl enable/start — track for disable/stop on cleanup - if command =~ /\bsystemctl\s+(enable|start|enable\s+--now)\s+(\S+)/ + if command =~ /\bsystemctl\s+(enable\s+--now|enable|start)\s+(\S+)/ @created_resources << { service: $2, action: $1 } end # RHEL: dnf/yum install — track for removal on cleanup - if command =~ /\b(?:dnf|yum)\s+install\s+(?:-y\s+)?(.+)/ - packages = $1.strip.split(/\s+/).reject { |p| p.start_with?('-') } - @created_resources << { packages: packages } unless packages.empty? + if command =~ /\b(dnf|yum)\s+install\s+(?:-y\s+)?(.+)/ + pkg_manager = $1 + packages = $2.strip.split(/\s+/).reject { |p| p.start_with?('-') } + @created_resources << { packages: packages, pkg_manager: pkg_manager } unless packages.empty? end end @@ -604,14 +648,19 @@ def run_cleanup elsif res[:service] # RHEL: stop and disable services that were started/enabled puts "Stopping service: #{res[:service]}" - Open3.capture3("sudo systemctl stop #{res[:service]}") - Open3.capture3("sudo systemctl disable #{res[:service]}") - puts "[CLEANED] Service #{res[:service]} stopped and disabled." + _, stop_stderr, stop_status = Open3.capture3('sudo', 'systemctl', 'stop', res[:service].to_s) + _, disable_stderr, disable_status = Open3.capture3('sudo', 'systemctl', 'disable', res[:service].to_s) + if stop_status.success? && disable_status.success? + puts "[CLEANED] Service #{res[:service]} stopped and disabled." + else + puts "[WARN] Service cleanup failed: #{[stop_stderr, disable_stderr].reject(&:empty?).join(' | ')}" + end elsif res[:packages] - # RHEL: remove packages that were installed + # RHEL: remove packages using the same package manager that installed them + pkg_manager = res[:pkg_manager] || 'dnf' pkg_list = res[:packages].join(' ') - puts "Removing packages: #{pkg_list}" - stdout, stderr, status = Open3.capture3("sudo dnf remove -y #{pkg_list}") + puts "Removing packages (#{pkg_manager}): #{pkg_list}" + stdout, stderr, status = Open3.capture3('sudo', pkg_manager, 'remove', '-y', *res[:packages]) if status.success? puts "[CLEANED] Packages removed." else From e3932fec5bcf8db392db43b699626efb289656fb Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Fri, 20 Mar 2026 14:15:27 +0000 Subject: [PATCH 06/17] Fix CodeRabbit nits in verify_proc.rb - Add safe_workdir_path to prevent directory traversal via ../ in extracted file paths; use it for all file writes (YAML, JSON, config, script) - Use array form for Open3.capture3 calls to avoid shell interpretation: cli_tool dry-run, ruby syntax check, cleanup delete commands - Remove unused current_step_depth variable - Use _ prefix for unused stdout captures Co-Authored-By: Claude Opus 4.6 --- .../verify-procedure/scripts/verify_proc.rb | 60 +++++++++++-------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index afd8fe47..90fa840c 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -139,7 +139,6 @@ def extract_code_blocks # Hierarchical step tracking: [major, sub, subsub] step_counters = [0, 0, 0] current_step_label = nil - current_step_depth = 0 ifdef_depth = 0 @@ -177,7 +176,6 @@ def extract_code_blocks end current_step_label = format_step_label(step_counters, depth) - current_step_depth = depth current_step_text = step_text end @@ -375,6 +373,18 @@ def has_placeholders?(content) content.include?('REPLACE') end + # Ensure a path stays within the working directory (prevents traversal attacks) + def safe_workdir_path(relative_path) + return nil if relative_path.start_with?('/') || relative_path.start_with?('~/') + + dest = File.expand_path(relative_path, @workdir) + workdir_root = File.expand_path(@workdir) + File::SEPARATOR + return nil unless dest.start_with?(workdir_root) + + FileUtils.mkdir_p(File.dirname(dest)) + dest + end + def validate_yaml(content, label, save_as = nil) begin # Lint the YAML for syntax errors @@ -386,11 +396,10 @@ def validate_yaml(content, label, save_as = nil) # Absolute paths (e.g., /etc/foo.conf from RHEL procedures) are validated # but not written — the script should not modify system files. if save_as - if save_as.start_with?('/') || save_as.start_with?('~/') - puts "[INFO] Absolute path #{save_as} — YAML validated but not written to filesystem." + dest = safe_workdir_path(save_as) + if dest.nil? + puts "[INFO] Path #{save_as} — YAML validated but not written to filesystem." else - dest = File.join(@workdir, save_as) - FileUtils.mkdir_p(File.dirname(dest)) File.write(dest, content) puts "[INFO] Saved YAML to #{dest}" end @@ -405,7 +414,7 @@ def validate_yaml(content, label, save_as = nil) Tempfile.open(['resource', '.yaml']) do |f| f.write(content) f.close - stdout, stderr, status = Open3.capture3("#{@cli_tool} apply -f #{f.path} --dry-run=client") + _, stderr, status = Open3.capture3(@cli_tool, 'apply', '-f', f.path, '--dry-run=client') if status.success? puts "[VALID] Resource dry-run (#{@cli_tool}) passed for Step #{label}." elsif stderr.include?("Unable to connect") || stderr.include?("no such host") || stderr.include?("connection refused") @@ -434,11 +443,10 @@ def validate_json(content, label, save_as = nil) @results << { step: label, status: :passed, output: "JSON syntax valid" } if save_as - if save_as.start_with?('/') || save_as.start_with?('~/') - puts "[INFO] Absolute path #{save_as} — JSON validated but not written to filesystem." + dest = safe_workdir_path(save_as) + if dest.nil? + puts "[INFO] Path #{save_as} — JSON validated but not written to filesystem." else - dest = File.join(@workdir, save_as) - FileUtils.mkdir_p(File.dirname(dest)) File.write(dest, content) puts "[INFO] Saved JSON to #{dest}" end @@ -457,11 +465,10 @@ def validate_config(content, label, save_as = nil) @results << { step: label, status: :passed, output: "Config content recorded" } if save_as - if save_as.start_with?('/') || save_as.start_with?('~/') - puts "[INFO] Absolute path #{save_as} — content validated but not written to filesystem." + dest = safe_workdir_path(save_as) + if dest.nil? + puts "[INFO] Path #{save_as} — content validated but not written to filesystem." else - dest = File.join(@workdir, save_as) - FileUtils.mkdir_p(File.dirname(dest)) File.write(dest, content) puts "[INFO] Saved config to #{dest}" end @@ -485,7 +492,7 @@ def validate_script(content, label, lang, save_as = nil) Tempfile.open(['step', '.rb']) do |f| f.write(content) f.close - stdout, stderr, status = Open3.capture3("ruby -c #{f.path}") + _, stderr, status = Open3.capture3('ruby', '-c', f.path) if status.success? puts "[VALID] Ruby syntax for Step #{label} is correct." @results << { step: label, status: :passed, output: "Ruby syntax valid" } @@ -496,11 +503,12 @@ def validate_script(content, label, lang, save_as = nil) end end - if save_as && !(save_as.start_with?('/') || save_as.start_with?('~/')) - dest = File.join(@workdir, save_as) - FileUtils.mkdir_p(File.dirname(dest)) - File.write(dest, content) - puts "[INFO] Saved script to #{dest}" + if save_as + dest = safe_workdir_path(save_as) + if dest + File.write(dest, content) + puts "[INFO] Saved script to #{dest}" + end end end @@ -630,18 +638,18 @@ def run_cleanup if res[:file] tool = res[:tool] || @cli_tool || 'oc' puts "Deleting resources from: #{res[:file]}" - stdout, stderr, status = Open3.capture3("#{tool} delete -f #{res[:file]} --ignore-not-found") + _, stderr, status = Open3.capture3(tool, 'delete', '-f', res[:file], '--ignore-not-found') if status.success? - puts "[CLEANED] #{stdout.strip}" + puts "[CLEANED] Deleted resources from #{res[:file]}" else puts "[WARN] Cleanup failed: #{stderr.strip}" end elsif res[:resource] tool = @cli_tool || 'oc' puts "Deleting: #{res[:resource]}" - stdout, stderr, status = Open3.capture3("#{tool} delete #{res[:resource]} --ignore-not-found") + _, stderr, status = Open3.capture3(tool, 'delete', res[:resource], '--ignore-not-found') if status.success? - puts "[CLEANED] #{stdout.strip}" + puts "[CLEANED] Deleted #{res[:resource]}" else puts "[WARN] Cleanup failed: #{stderr.strip}" end @@ -660,7 +668,7 @@ def run_cleanup pkg_manager = res[:pkg_manager] || 'dnf' pkg_list = res[:packages].join(' ') puts "Removing packages (#{pkg_manager}): #{pkg_list}" - stdout, stderr, status = Open3.capture3('sudo', pkg_manager, 'remove', '-y', *res[:packages]) + _, stderr, status = Open3.capture3('sudo', pkg_manager, 'remove', '-y', *res[:packages]) if status.success? puts "[CLEANED] Packages removed." else From 771ed76682974d3153f13c1cf67c5152958c21ed Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:47:29 +0000 Subject: [PATCH 07/17] Fix subs= check to only resolve attributes when actually enabled The previous check `source_attrs.include?('subs=')` matched any subs override (e.g. subs=none, subs=quotes), which could corrupt literal {...} content. Now only triggers attribute resolution when the subs value explicitly includes 'attributes'. Co-Authored-By: Claude Opus 4.6 --- .../docs-tools/skills/verify-procedure/scripts/verify_proc.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index 90fa840c..f89200ac 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -186,7 +186,7 @@ def extract_code_blocks if line =~ /^\[source,(terminal|bash|yaml|shell|json|ini|toml|text|python|ruby)(?:,(.*?))?\]\s*$/ source_type = $1 source_attrs = $2 || '' - has_subs = source_attrs.include?('subs=') + has_subs = source_attrs =~ /subs=.*attributes/ # Check if this is an example output block (preceded by "Example output" or similar) is_example = false From 9f4379ae63620aeab55c9ecd2f197792b1c6c9f4 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:48:45 +0000 Subject: [PATCH 08/17] Fix home-relative paths and extensionless filenames in file extraction - Map ~/... paths into the sandboxed workdir instead of dropping them - Add rule for backtick-quoted relative filenames without extensions (e.g., `default`, `my-config`) Co-Authored-By: Claude Opus 4.6 --- .../verify-procedure/scripts/verify_proc.rb | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index f89200ac..e99e454d 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -293,11 +293,15 @@ def link_yaml_to_files(blocks) # 1. Backtick-quoted paths: `/etc/chrony.conf` or `foo.yaml` if instruction =~ /`([^`]*#{SAVE_FILE_EXTENSIONS.source})`/i block[:save_as] = $1 - # 2. Backtick-quoted absolute paths without recognized extension: - # e.g., `~/playbook.yml`, `/etc/sysctl.d/99-custom` + # 2. Backtick-quoted absolute or home-relative paths (with or without extension): + # e.g., `/etc/sysctl.d/99-custom`, `~/playbook.yml` elsif instruction =~ /`((?:\/|~\/)[^`]+)`/ block[:save_as] = $1 - # 3. Bare filenames — word chars, hyphens, dots ending with recognized extension + # 3. Backtick-quoted relative filenames without recognized extension: + # e.g., `default`, `my-config` + elsif instruction =~ /`([\w][\w.-]*)`/ + block[:save_as] = $1 + # 4. Bare filenames — word chars, hyphens, dots ending with recognized extension elsif instruction =~ /\b([\w][\w.-]*#{SAVE_FILE_EXTENSIONS.source})\b/i block[:save_as] = $1 end @@ -375,9 +379,12 @@ def has_placeholders?(content) # Ensure a path stays within the working directory (prevents traversal attacks) def safe_workdir_path(relative_path) - return nil if relative_path.start_with?('/') || relative_path.start_with?('~/') + return nil if relative_path.start_with?('/') + + # Map ~/... paths into the workdir instead of expanding to real $HOME + sandboxed = relative_path.sub(%r{^~/}, '') - dest = File.expand_path(relative_path, @workdir) + dest = File.expand_path(sandboxed, @workdir) workdir_root = File.expand_path(@workdir) + File::SEPARATOR return nil unless dest.start_with?(workdir_root) From 6f290a1d78c7189b184c27453f9cdbdc61e97499 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:50:02 +0000 Subject: [PATCH 09/17] Guard against >26 substeps in step label formatting Fall back to "sub" when depth-2 or depth-3 counters exceed the 26-element alpha/roman arrays, preventing nil labels. Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/scripts/verify_proc.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index e99e454d..0071ae91 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -252,11 +252,12 @@ def format_step_label(counters, depth) parts = [] parts << counters[0].to_s if counters[0] > 0 if depth >= 2 && counters[1] > 0 - parts << ('a'..'z').to_a[counters[1] - 1] + alpha = ('a'..'z').to_a + parts << (counters[1] <= 26 ? alpha[counters[1] - 1] : "sub#{counters[1]}") end if depth >= 3 && counters[2] > 0 roman = %w[i ii iii iv v vi vii viii ix x xi xii xiii xiv xv xvi xvii xviii xix xx xxi xxii xxiii xxiv xxv xxvi] - parts << roman[counters[2] - 1] + parts << (counters[2] <= 26 ? roman[counters[2] - 1] : "sub#{counters[2]}") end parts.join('.') end From 01da16f54a99ce28abbbbb98e9d709223ac8545f Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:52:37 +0000 Subject: [PATCH 10/17] Save script blocks as files when instruction indicates file creation Steps like "Create verify.sh with the following content" now save the bash/python/ruby block to disk instead of executing it. This ensures later steps that reference the file (e.g., python foo.py) can find it. - Extend link_yaml_to_files to detect save targets for all block types - Add SAVE_INSTRUCTION_PATTERN to identify "create/save/write file" - Short-circuit execution when a script block should be saved - Add .py and .rb to SAVE_FILE_EXTENSIONS Co-Authored-By: Claude Opus 4.6 --- .../verify-procedure/scripts/verify_proc.rb | 25 +++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index 0071ae91..ea1af3f2 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -283,11 +283,15 @@ def format_step_label(counters, depth) # # The instruction doesn't need to mention "YAML" — any backtick-quoted or # bare filename with a recognized extension is matched. - SAVE_FILE_EXTENSIONS = /\.(?:ya?ml|json|conf|cfg|sh|txt|toml|ini|properties)/i + SAVE_FILE_EXTENSIONS = /\.(?:ya?ml|json|conf|cfg|sh|txt|toml|ini|properties|py|rb)/i + + # Patterns in the instruction text that indicate "save this content as a file" + # rather than "execute this command". + SAVE_INSTRUCTION_PATTERN = /\b(?:creat|sav|writ|add|stor|nam)\w*\b.*\b(?:file|script|content)\b/i def link_yaml_to_files(blocks) blocks.each do |block| - next unless %w[yaml json config].include?(block[:type]) + next unless %w[yaml json config bash python ruby].include?(block[:type]) instruction = block[:instruction] @@ -353,6 +357,23 @@ def process_step(block) return end + # If a bash/script block has a save_as target and the instruction + # indicates file creation, save the content instead of executing it. + if block[:save_as] && %w[bash python ruby].include?(type) && + instruction =~ SAVE_INSTRUCTION_PATTERN + save_as = block[:save_as] + dest = safe_workdir_path(save_as) + if dest.nil? + puts "[INFO] Path #{save_as} — content recorded but not written to filesystem." + else + File.write(dest, content) + File.chmod(0755, dest) if type == 'bash' + puts "[INFO] Saved #{type} script to #{dest} (not executed)" + end + @results << { step: label, status: :passed, output: "Saved as #{save_as}" } + return + end + case type when 'yaml' validate_yaml(content, label, block[:save_as]) From 03b5234ba4a9d645fce99da73fbea340e59791db Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:53:45 +0000 Subject: [PATCH 11/17] Track all resource/package matches, not just the first Replace single =~ regex matches with scan to iterate all matches in commands and stdout. Fixes cleanup leaving behind resources when a block applies multiple manifests, installs multiple packages, or prints several "created" lines. Co-Authored-By: Claude Opus 4.6 --- .../verify-procedure/scripts/verify_proc.rb | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index ea1af3f2..3796763d 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -632,9 +632,8 @@ def execute_bash(command, label, instruction) # Handles both K8s resources (oc/kubectl) and RHEL system changes (systemctl, dnf). def track_resource(command, stdout) # K8s: "oc create -f file.yaml" or "oc apply -f file.yaml" - if command =~ /\b(oc|kubectl)\s+(create|apply)\s+-f\s+(\S+)/ - tool = $1 - file = $3 + # Scan all matches — a single command may apply multiple manifests. + command.scan(/\b(oc|kubectl)\s+(?:create|apply)\s+-f\s+(\S+)/) do |tool, file| filepath = File.join(@workdir, file) if File.exist?(filepath) @created_resources << { tool: tool, file: filepath } @@ -642,19 +641,19 @@ def track_resource(command, stdout) end # K8s: inline resource creation from stdout like "namespace/openshift-ptp created" - if stdout =~ %r{^(\S+/\S+)\s+created} - @created_resources << { resource: $1 } + # Parse each line — a single apply can create multiple resources. + stdout.scan(%r{^(\S+/\S+)\s+created}m) do |resource,| + @created_resources << { resource: resource } end # RHEL: systemctl enable/start — track for disable/stop on cleanup - if command =~ /\bsystemctl\s+(enable\s+--now|enable|start)\s+(\S+)/ - @created_resources << { service: $2, action: $1 } + command.scan(/\bsystemctl\s+(enable\s+--now|enable|start)\s+(\S+)/) do |action, service| + @created_resources << { service: service, action: action } end # RHEL: dnf/yum install — track for removal on cleanup - if command =~ /\b(dnf|yum)\s+install\s+(?:-y\s+)?(.+)/ - pkg_manager = $1 - packages = $2.strip.split(/\s+/).reject { |p| p.start_with?('-') } + command.scan(/\b(dnf|yum)\s+install\s+(?:-y\s+)?(.+)/) do |pkg_manager, pkg_list| + packages = pkg_list.strip.split(/\s+/).reject { |p| p.start_with?('-') } @created_resources << { packages: packages, pkg_manager: pkg_manager } unless packages.empty? end end From 52cf322cb297a8d33bf1ed7c43c721044c428b86 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:54:28 +0000 Subject: [PATCH 12/17] Revert only the actual systemctl action during cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - start → stop (don't change enablement) - enable → disable (don't stop) - enable --now → stop and disable Co-Authored-By: Claude Opus 4.6 --- .../verify-procedure/scripts/verify_proc.rb | 35 +++++++++++++++---- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index 3796763d..d6277271 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -682,14 +682,35 @@ def run_cleanup puts "[WARN] Cleanup failed: #{stderr.strip}" end elsif res[:service] - # RHEL: stop and disable services that were started/enabled - puts "Stopping service: #{res[:service]}" - _, stop_stderr, stop_status = Open3.capture3('sudo', 'systemctl', 'stop', res[:service].to_s) - _, disable_stderr, disable_status = Open3.capture3('sudo', 'systemctl', 'disable', res[:service].to_s) - if stop_status.success? && disable_status.success? - puts "[CLEANED] Service #{res[:service]} stopped and disabled." + # RHEL: revert only the action that was actually performed + service = res[:service].to_s + action = res[:action].to_s + errors = [] + + case action + when 'enable --now' + # Was both enabled and started — revert both + _, err, s = Open3.capture3('sudo', 'systemctl', 'stop', service) + errors << err unless s.success? + _, err, s = Open3.capture3('sudo', 'systemctl', 'disable', service) + errors << err unless s.success? + verb = 'stopped and disabled' + when 'enable' + # Only enabled — just disable, don't stop + _, err, s = Open3.capture3('sudo', 'systemctl', 'disable', service) + errors << err unless s.success? + verb = 'disabled' + when 'start' + # Only started — just stop, don't change enablement + _, err, s = Open3.capture3('sudo', 'systemctl', 'stop', service) + errors << err unless s.success? + verb = 'stopped' + end + + if errors.empty? + puts "[CLEANED] Service #{service} #{verb}." else - puts "[WARN] Service cleanup failed: #{[stop_stderr, disable_stderr].reject(&:empty?).join(' | ')}" + puts "[WARN] Service cleanup failed: #{errors.reject(&:empty?).join(' | ')}" end elsif res[:packages] # RHEL: remove packages using the same package manager that installed them From fd353eb95654a142f1149c5ed65ad886fb848a1a Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Tue, 24 Mar 2026 14:58:17 +0000 Subject: [PATCH 13/17] Use safe_load_stream for multi-document YAML validation YAML.safe_load fails on multi-document files (multiple --- separated docs) common in K8s manifests. safe_load_stream handles these correctly. Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/scripts/verify_proc.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb index d6277271..e621d4fe 100644 --- a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb @@ -416,8 +416,10 @@ def safe_workdir_path(relative_path) def validate_yaml(content, label, save_as = nil) begin - # Lint the YAML for syntax errors - YAML.safe_load(content) + # Lint the YAML for syntax errors. + # Use safe_load_stream to handle multi-document YAML (multiple --- + # separated docs), which is common in Kubernetes manifests. + YAML.safe_load_stream(content) puts "[VALID] YAML syntax for Step #{label} is correct." @results << { step: label, status: :passed, output: "YAML syntax valid" } From f1c135635570568ec270fcbd17565c61071166ed Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Wed, 25 Mar 2026 10:40:31 +0000 Subject: [PATCH 14/17] Refactor verify-procedure: Claude parses, bash executes Replace the 782-line Ruby parser/executor with a two-layer architecture: - Claude handles AsciiDoc parsing, intent classification, prompt stripping, step numbering, and judgment calls (include:: resolution, ifdef handling, nested blocks, callout stripping) - A thin 397-line bash script (verify_proc.sh) handles only execution: run commands, validate YAML/JSON, save files, track resources, cleanup Script improvements over the Ruby version: - Session-isolated temp files (concurrent runs don't collide) - Safe counter reads (grep/cut instead of source) - printf instead of echo (no spurious trailing newlines) - Python for YAML/JSON validation (more portable) SKILL.md improvements: - Pushier description for better skill triggering - Heredoc examples instead of fragile echo piping - Guidance on assemblies (multi-file procedures) - Guidance on partial cluster state and missing prerequisites - Focused on OpenShift/K8s (dropped RHEL/Ansible/Python/Ruby scope) Co-Authored-By: Claude Opus 4.6 --- .../skills/verify-procedure/SKILL.md | 325 ++++---- .../verify-procedure/scripts/verify_proc.rb | 781 ------------------ .../verify-procedure/scripts/verify_proc.sh | 397 +++++++++ 3 files changed, 566 insertions(+), 937 deletions(-) delete mode 100644 plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.rb create mode 100755 plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.sh diff --git a/plugins/docs-tools/skills/verify-procedure/SKILL.md b/plugins/docs-tools/skills/verify-procedure/SKILL.md index 3652240a..adc55ae4 100644 --- a/plugins/docs-tools/skills/verify-procedure/SKILL.md +++ b/plugins/docs-tools/skills/verify-procedure/SKILL.md @@ -1,205 +1,218 @@ --- name: verify-procedure -description: Execute and test AsciiDoc procedures on a live system. Runs every command and validates every YAML block against a real cluster, VM, or host. Requires an active connection to the target system. For static review without a live system, use the docs-tools:technical-reviewer agent instead. +description: Execute, test, and verify AsciiDoc procedures on a live OpenShift or Kubernetes cluster. Runs every command and validates every YAML block against a real cluster. Use this skill whenever the user asks to verify a procedure, test documentation steps, run through a guided exercise, prove a procedure works, check if steps are correct on a live system, or do a dry run of a doc. Requires an active oc/kubectl connection. For static review without a live system, use docs-tools:technical-reviewer instead. author: Red Hat Documentation Team -allowed-tools: Bash, Read, Edit, Glob +allowed-tools: Bash, Read, Edit, Glob, Grep --- # Procedure Verification Skill -This skill executes documented procedures against a live system to prove they work end-to-end. It is the "guided exercise tester" — it runs every command, applies every YAML block, and reports what passes and what breaks. +You execute documented OpenShift/Kubernetes procedures against a live cluster to prove they work end-to-end. **You** are the parser and judgment layer. The bash script `scripts/verify_proc.sh` is a thin executor — it only runs commands, validates YAML, and saves files. -**This is not a review tool.** For reviewing documentation quality, prerequisites, and structure without a live system, use the `docs-tools:technical-reviewer` agent. +**This is not a review tool.** For reviewing documentation quality without a live system, use `docs-tools:technical-reviewer`. -## Prerequisites +## Architecture: what you do vs. what the script does -You must be connected to the target system before invoking this skill: +| You (Claude) | Script (`verify_proc.sh`) | +|---|---| +| Read and parse the `.adoc` file | `init` — create workdir, detect oc/kubectl | +| Resolve `include::` directives by reading included files | `check-connection` — verify cluster login | +| Handle `ifdef::`/`ifndef::` conditionals intelligently | `execute

,
) - content.match?(/<[a-z][a-z0-9]*[_\s-][a-z0-9_\s-]+>/i) || # Multi-word placeholders - content.match?(/\$\{[^}]+\}/) || # Variable placeholders like ${VAR} - content.include?('CHANGEME') || - content.include?('REPLACE') - end - - # Ensure a path stays within the working directory (prevents traversal attacks) - def safe_workdir_path(relative_path) - return nil if relative_path.start_with?('/') - - # Map ~/... paths into the workdir instead of expanding to real $HOME - sandboxed = relative_path.sub(%r{^~/}, '') - - dest = File.expand_path(sandboxed, @workdir) - workdir_root = File.expand_path(@workdir) + File::SEPARATOR - return nil unless dest.start_with?(workdir_root) - - FileUtils.mkdir_p(File.dirname(dest)) - dest - end - - def validate_yaml(content, label, save_as = nil) - begin - # Lint the YAML for syntax errors. - # Use safe_load_stream to handle multi-document YAML (multiple --- - # separated docs), which is common in Kubernetes manifests. - YAML.safe_load_stream(content) - puts "[VALID] YAML syntax for Step #{label} is correct." - @results << { step: label, status: :passed, output: "YAML syntax valid" } - - # Save the YAML to the working directory if a filename was detected. - # Absolute paths (e.g., /etc/foo.conf from RHEL procedures) are validated - # but not written — the script should not modify system files. - if save_as - dest = safe_workdir_path(save_as) - if dest.nil? - puts "[INFO] Path #{save_as} — YAML validated but not written to filesystem." - else - File.write(dest, content) - puts "[INFO] Saved YAML to #{dest}" - end - end - - # If it looks like a Kubernetes resource, try dry-run validation. - # This works with both oc and kubectl — it's a K8s API feature, not OCP-specific. - # Dry-run is a bonus check — it does NOT replace the syntax result. - # If there's no cluster connection, the syntax pass still stands. - if content.include?("apiVersion:") - if @cli_tool - Tempfile.open(['resource', '.yaml']) do |f| - f.write(content) - f.close - _, stderr, status = Open3.capture3(@cli_tool, 'apply', '-f', f.path, '--dry-run=client') - if status.success? - puts "[VALID] Resource dry-run (#{@cli_tool}) passed for Step #{label}." - elsif stderr.include?("Unable to connect") || stderr.include?("no such host") || stderr.include?("connection refused") - # No cluster connectivity — don't fail the YAML validation for this - puts "[SKIP] No cluster connection — dry-run skipped for Step #{label}." - else - # Genuine resource validation error (e.g., invalid field, unknown kind) - puts "[FAILURE] Resource validation failed: #{stderr.strip.lines.first}" - @results << { step: "#{label}-dryrun", status: :failed, error: stderr.strip } - end - end - else - puts "[SKIP] No oc or kubectl found — skipping resource dry-run for Step #{label}." - end - end - rescue Psych::SyntaxError => e - puts "[FAILURE] YAML Syntax error in Step #{label}: #{e.message}" - @results << { step: label, status: :failed, error: e.message } - end - end - - def validate_json(content, label, save_as = nil) - begin - JSON.parse(content) - puts "[VALID] JSON syntax for Step #{label} is correct." - @results << { step: label, status: :passed, output: "JSON syntax valid" } - - if save_as - dest = safe_workdir_path(save_as) - if dest.nil? - puts "[INFO] Path #{save_as} — JSON validated but not written to filesystem." - else - File.write(dest, content) - puts "[INFO] Saved JSON to #{dest}" - end - end - rescue JSON::ParserError => e - puts "[FAILURE] JSON Syntax error in Step #{label}: #{e.message}" - @results << { step: label, status: :failed, error: e.message } - end - end - - # Validate config file content (INI, TOML, plain text). - # For RHEL procedures that edit files like /etc/chrony.conf, systemd units, etc. - # We validate what we can (TOML syntax) and record the rest as seen. - def validate_config(content, label, save_as = nil) - puts "[VALID] Configuration content for Step #{label} recorded." - @results << { step: label, status: :passed, output: "Config content recorded" } - - if save_as - dest = safe_workdir_path(save_as) - if dest.nil? - puts "[INFO] Path #{save_as} — content validated but not written to filesystem." - else - File.write(dest, content) - puts "[INFO] Saved config to #{dest}" - end - end - end - - # Validate script content (Python, Ruby) for syntax errors without executing. - def validate_script(content, label, lang, save_as = nil) - case lang - when 'python' - # Use python3 -c "compile()" for syntax check - stdout, stderr, status = Open3.capture3('python3', '-c', "compile(#{content.inspect}, '', 'exec')") - if status.success? - puts "[VALID] Python syntax for Step #{label} is correct." - @results << { step: label, status: :passed, output: "Python syntax valid" } - else - puts "[FAILURE] Python syntax error in Step #{label}: #{stderr.strip}" - @results << { step: label, status: :failed, error: stderr.strip } - end - when 'ruby' - Tempfile.open(['step', '.rb']) do |f| - f.write(content) - f.close - _, stderr, status = Open3.capture3('ruby', '-c', f.path) - if status.success? - puts "[VALID] Ruby syntax for Step #{label} is correct." - @results << { step: label, status: :passed, output: "Ruby syntax valid" } - else - puts "[FAILURE] Ruby syntax error in Step #{label}: #{stderr.strip}" - @results << { step: label, status: :failed, error: stderr.strip } - end - end - end - - if save_as - dest = safe_workdir_path(save_as) - if dest - File.write(dest, content) - puts "[INFO] Saved script to #{dest}" - end - end - end - - def run_command_with_timeout(command, chdir:, timeout:) - stdout = +'' - stderr = +'' - status = nil - - Open3.popen3(command, chdir: chdir) do |stdin, out, err, wait_thr| - stdin.close - out_reader = Thread.new { stdout << out.read } - err_reader = Thread.new { stderr << err.read } - - begin - Timeout.timeout(timeout) { status = wait_thr.value } - rescue Timeout::Error - Process.kill('TERM', wait_thr.pid) rescue nil - Process.kill('KILL', wait_thr.pid) rescue nil - Process.wait(wait_thr.pid) rescue nil - raise - ensure - out.close unless out.closed? - err.close unless err.closed? - out_reader.join - err_reader.join - end - end - - [stdout, stderr, status] - end - - def execute_bash(command, label, instruction) - # Clean up command: remove prompt symbols from each line. - # Handles prompts across Red Hat products: - # OCP/K8s: $ oc get pods - # RHEL: # dnf install, [root@host ~]# systemctl, ~]# subscription-manager - # Mixed: $ sudo dnf install - clean_command = command.lines.map do |line| - line - .sub(/^\[[\w@.\-]+ [~\w\/]*\][#$]\s*/, '') # [root@host ~]# or [user@host dir]$ - .sub(/^~\][#$]\s*/, '') # ~]# or ~]$ - .sub(/^[#$]\s/, '') # bare # or $ prompt - end.join - - # Handle multi-line commands with backslash continuations - if clean_command.include?('\\') - # Join lines that end with backslash - clean_command = clean_command.gsub(/\\\n\s*/, ' ').strip - else - clean_command = clean_command.strip - end - - puts "Executing: #{clean_command[0..150]}#{clean_command.length > 150 ? '...' : ''}" - - # Run commands in the working directory with a 120-second timeout. - # Uses popen3 with explicit TERM/KILL to ensure the child process - # is terminated on timeout (not just the Ruby caller). - begin - stdout, stderr, status = run_command_with_timeout(clean_command, chdir: @workdir, timeout: 120) - rescue Timeout::Error - puts "[FAILURE] Step #{label} timed out after 120 seconds." - @results << { step: label, status: :failed, error: "Command timed out after 120 seconds" } - puts "[WARNING] Continuing with remaining steps despite failure..." - return - end - - if status.success? - puts "[SUCCESS] Step #{label} executed." - - # Track created resources for cleanup - track_resource(clean_command, stdout) - - # Show output if it's a verification/check command - if instruction.downcase.match?(/verify|check|confirm|retrieve|identify/) - puts "Output: #{stdout.strip[0..200]}" unless stdout.strip.empty? - puts "-> Verification successfully performed." - end - - @results << { step: label, status: :passed, output: stdout.strip } - else - puts "[FAILURE] Step #{label} failed." - puts "STDERR: #{stderr.strip}" unless stderr.strip.empty? - puts "STDOUT: #{stdout.strip}" unless stdout.strip.empty? - @results << { step: label, status: :failed, error: stderr.strip } - - # Don't exit immediately - continue with warnings - puts "[WARNING] Continuing with remaining steps despite failure..." - end - end - - # Track resources created during verification for cleanup. - # Handles both K8s resources (oc/kubectl) and RHEL system changes (systemctl, dnf). - def track_resource(command, stdout) - # K8s: "oc create -f file.yaml" or "oc apply -f file.yaml" - # Scan all matches — a single command may apply multiple manifests. - command.scan(/\b(oc|kubectl)\s+(?:create|apply)\s+-f\s+(\S+)/) do |tool, file| - filepath = File.join(@workdir, file) - if File.exist?(filepath) - @created_resources << { tool: tool, file: filepath } - end - end - - # K8s: inline resource creation from stdout like "namespace/openshift-ptp created" - # Parse each line — a single apply can create multiple resources. - stdout.scan(%r{^(\S+/\S+)\s+created}m) do |resource,| - @created_resources << { resource: resource } - end - - # RHEL: systemctl enable/start — track for disable/stop on cleanup - command.scan(/\bsystemctl\s+(enable\s+--now|enable|start)\s+(\S+)/) do |action, service| - @created_resources << { service: service, action: action } - end - - # RHEL: dnf/yum install — track for removal on cleanup - command.scan(/\b(dnf|yum)\s+install\s+(?:-y\s+)?(.+)/) do |pkg_manager, pkg_list| - packages = pkg_list.strip.split(/\s+/).reject { |p| p.start_with?('-') } - @created_resources << { packages: packages, pkg_manager: pkg_manager } unless packages.empty? - end - end - - def run_cleanup - puts "\n--- Cleanup ---" - - # Delete resources in reverse order - @created_resources.reverse.each do |res| - if res[:file] - tool = res[:tool] || @cli_tool || 'oc' - puts "Deleting resources from: #{res[:file]}" - _, stderr, status = Open3.capture3(tool, 'delete', '-f', res[:file], '--ignore-not-found') - if status.success? - puts "[CLEANED] Deleted resources from #{res[:file]}" - else - puts "[WARN] Cleanup failed: #{stderr.strip}" - end - elsif res[:resource] - tool = @cli_tool || 'oc' - puts "Deleting: #{res[:resource]}" - _, stderr, status = Open3.capture3(tool, 'delete', res[:resource], '--ignore-not-found') - if status.success? - puts "[CLEANED] Deleted #{res[:resource]}" - else - puts "[WARN] Cleanup failed: #{stderr.strip}" - end - elsif res[:service] - # RHEL: revert only the action that was actually performed - service = res[:service].to_s - action = res[:action].to_s - errors = [] - - case action - when 'enable --now' - # Was both enabled and started — revert both - _, err, s = Open3.capture3('sudo', 'systemctl', 'stop', service) - errors << err unless s.success? - _, err, s = Open3.capture3('sudo', 'systemctl', 'disable', service) - errors << err unless s.success? - verb = 'stopped and disabled' - when 'enable' - # Only enabled — just disable, don't stop - _, err, s = Open3.capture3('sudo', 'systemctl', 'disable', service) - errors << err unless s.success? - verb = 'disabled' - when 'start' - # Only started — just stop, don't change enablement - _, err, s = Open3.capture3('sudo', 'systemctl', 'stop', service) - errors << err unless s.success? - verb = 'stopped' - end - - if errors.empty? - puts "[CLEANED] Service #{service} #{verb}." - else - puts "[WARN] Service cleanup failed: #{errors.reject(&:empty?).join(' | ')}" - end - elsif res[:packages] - # RHEL: remove packages using the same package manager that installed them - pkg_manager = res[:pkg_manager] || 'dnf' - pkg_list = res[:packages].join(' ') - puts "Removing packages (#{pkg_manager}): #{pkg_list}" - _, stderr, status = Open3.capture3('sudo', pkg_manager, 'remove', '-y', *res[:packages]) - if status.success? - puts "[CLEANED] Packages removed." - else - puts "[WARN] Package removal failed: #{stderr.strip}" - end - end - end - - # Remove the working directory - FileUtils.rm_rf(@workdir) - puts "[CLEANED] Removed working directory: #{@workdir}" - end - - def summarize - puts "\n" + "="*60 - puts "FINAL SUMMARY" - puts "="*60 - - passed = @results.count { |r| r[:status] == :passed } - failed = @results.count { |r| r[:status] == :failed } - - puts "Total executable steps: #{@results.size}" - puts "Passed: #{passed}" - puts "Failed: #{failed}" - - if failed > 0 - puts "\nFailed steps:" - @results.select { |r| r[:status] == :failed }.each do |result| - puts " - Step #{result[:step]}: #{result[:error]&.split("\n")&.first}" - end - end - - # Check if a global verification example was included - unless @content.downcase.include?("verify") || @content.downcase.include?(".verification") - puts "\n[ADVICE] Consider adding an end-to-end verification step to this procedure." - end - - puts "="*60 - puts passed == @results.size ? "✓ All steps PASSED" : "✗ Some steps FAILED" - puts "="*60 - end -end - -# Execution -if ARGV.empty? - puts "Usage: ruby verify_proc.rb [--cleanup] " - puts " --cleanup Delete created resources and working directory after verification" - exit 1 -end - -cleanup = ARGV.delete('--cleanup') -file_path = ARGV[0] - -unless file_path && File.exist?(file_path) - puts "Error: File not found: #{file_path}" - exit 1 -end - -ProcedureVerifier.new(file_path, cleanup: !!cleanup).run_verification diff --git a/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.sh b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.sh new file mode 100755 index 00000000..d9385077 --- /dev/null +++ b/plugins/docs-tools/skills/verify-procedure/scripts/verify_proc.sh @@ -0,0 +1,397 @@ +#!/usr/bin/env bash +# verify_proc.sh — Thin executor for procedure verification. +# +# Claude handles AsciiDoc parsing and intent classification. +# This script only does what requires a shell: execute commands, +# validate YAML against a live cluster, save files, and clean up. +# +# Usage: +# verify_proc.sh init → create workdir, detect CLI tool +# verify_proc.sh execute