lkj demos

[thorwhalen]

A few usage demos of lkj tools.

[thorwhalen]

Inserting Missing set -e in CI YAML Files

The following example demonstrates how to use the FindReplaceTool to process CI YAML files (such as those used with Google Actions)
and automatically insert a missing set -e into any run: | block where the first non-blank line is not already set -e.

In this demo, we define a regular expression pattern that matches a run: | block only if its first non-blank line does not start with set -e.
Then, we provide a callback function that inserts set -e (with proper indentation) immediately after the run: | header.
Finally, we apply the replacement to all matches in the file and print the modified YAML.

Applying this to isee/actions

from dol import TextFiles, add_ipython_key_completions
import os
import isee
from lkj import FindReplaceTool

s = add_ipython_key_completions(
    TextFiles(os.path.join(os.path.dirname(isee.__path__[0]), 'actions'))
)
list(s)

['install-packages/action.yml',
 'ci-diagnosis/action.yml',
 'run-azurite/action.yml',
 'check-in/action.yml',
 'pylint-validation/action.yml',
 'deploy/action.yml',
 'tag-repository/action.yml',
 'bump-version-number/action.yml',
 'configure-git-user/action.yml',
 'pytest-validation/action.yml',
 'publish/action.yml',
 'format-source-code/action.yml',
 'generate-documentation/action.yml',
 'run-localstack/action.yml',
 'publish-job/action.yml',
 'validation-job/action.yml',
 'package/action.yml']

len(s)

17

Determine what keys (files) will need editing (will be different when editing them
to add the "set -e" line at the beginning of the run block if it's not already there.)
The pattern is designed to match a run block header (including its trailing newline)
and any blank lines immediately following it, only if the very next non-blank line
does not start with "set -e".

import re 

pattern = (
    r'^(?P<run_line>\s*run:\s*\|\n)'      # run header line ending with newline
    r'(?P<blank_lines>(?:[ \t]*\n)*)'       # any following blank lines (if present)
    r'(?![ \t]*set\s+-e)'                  # negative lookahead: next line must NOT start with "set -e"
)
flags = re.MULTILINE  # multiline mode so that ^ matches the beginning of each line


def keys_that_need_editing(text_store):
    for k, original_text in text_store.items():
        modified_text = (
            FindReplaceTool(original_text, line_mode=True, flags=flags)
            .analyze(pattern)
            .replace_all(insert_set_e)
            .get_modified_text()
        )
        if modified_text != original_text:
            yield k

keys = list(keys_that_need_editing(s))
len(keys)

10

Define a replacement callback that inserts "set -e" as the first command in the run block as well as helper functions to do what we want to do.

def insert_set_e(match_data):
    # Retrieve the captured groups.
    run_line = match_data["groups"].get("run_line", "")
    blank_lines = match_data["groups"].get("blank_lines", "")
    # Extract the indentation from the run header.
    header_indent_match = re.match(r'^(\s*)', run_line)
    header_indent = header_indent_match.group(1) if header_indent_match else ""
    # Assume that content inside the run block is indented 2 more spaces.
    content_indent = header_indent + "  "
    # Build the replacement text by inserting the "set -e" line immediately after the header.
    replacement_text = run_line + blank_lines + content_indent + "set -e" + "\n"
    return replacement_text

get_tool = lambda text: FindReplaceTool(text, line_mode=True, flags=flags)
see_matches = lambda string: get_tool(string).find_and_print_matches(pattern)
find_and_replace = lambda string: get_tool(string).find_and_replace(pattern, insert_set_e)

Look through and edit the files one by one

key_iterator = enumerate(keys)  # make (or reset) the keys iterator

# get the next key
i, key = next(key_iterator)
string = s[key]
print(f"{i}: {key}")

1: pylint-validation/action.yml

Have a look at the matches

see_matches(string)

Match 0 at line 32:
  30      - name: Validate Source Code
  31        shell: bash
  32        run: |
      ^^^^^^^^^^^^^
  33          if [ -n "${{ inputs.ignore }}" ]; then
  34            ignore="--ignore ${{ inputs.ignore }}"
----------------------------------------

See what the new string would look like

new_string = find_and_replace(string)
print(new_string)

    name: "Pylint Validation"
    description: "Validate source code running pylint"
    
...
        - name: Validate Source Code
          shell: bash
          run: |
            set -e
            if [ -n "${{ inputs.ignore }}" ]; then
              ignore="--ignore ${{ inputs.ignore }}"
            fi
...

And if you are good with that, replace the file with those contents:

s[key] = new_string

Some details

Let's try to test the ideas on a single string.

import re
# Assume that FindReplaceTool is already imported from your module.
# from find_replace_tool import FindReplaceTool

# Sample CI YAML input
ci_yaml = """\
    - name: Install APT Packages
      if: ${{ inputs.apt-packages }}
      shell: bash
      run: |
        sudo apt-get update
        sudo apt-get install -y ${{ inputs.apt-packages }}

    - name: Install NPM Packages
      if: ${{ inputs.npm-packages }}
      shell: bash
      run: |
        set -e
        npm install -g ${{ inputs.npm-packages }}
"""

Define a regex pattern that matches a "run: |" block only if the first non-blank line
does not begin with "set -e". The pattern captures:

• the run header (with any indentation)
• any blank lines following the header
• ensures via negative lookahead that the next non-blank line is not "set -e".

pattern = (
    r'^(?P<run_line>\s*run:\s*\|\n)'      # run header line ending with newline
    r'(?P<blank_lines>(?:[ \t]*\n)*)'       # any following blank lines (if present)
    r'(?![ \t]*set\s+-e)'                  # negative lookahead: next non-blank line must NOT start with "set -e"
)
flags = re.MULTILINE

Create an instance of FindReplaceTool in line_mode (to work with YAML lines)

from lkj import FindReplaceTool
tool = FindReplaceTool(ci_yaml, line_mode=True)

Analyze the YAML text using the pattern

tool.analyze(pattern, flags=flags)

Display found matches along with their context.

tool.view_matches()

    Match 0 at line 4:
       2        if: ${{ inputs.apt-packages }}
       3        shell: bash
       4        run: |
          ^^^^^^^^^^^^^
       5          sudo apt-get update
       6          sudo apt-get install -y ${{ inputs.apt-packages }}
    ----------------------------------------

Define a callback function that inserts "set -e" into the matched run block.

def insert_set_e(match_data):
    # Get the captured run header and blank lines.
    run_header = match_data["groups"]["run_line"]
    blank_lines = match_data["groups"]["blank_lines"]
    # Determine the indentation from the run header.
    indent_match = re.match(r'^(\s*)', run_header)
    indent = indent_match.group(1) if indent_match else ""
    # Assume that the content inside the run block is indented 2 spaces more than the header.
    content_indent = indent + "  "
    # Build and return the replacement text: header + blank lines + inserted "set -e" line.
    return run_header + blank_lines + content_indent + "set -e\n"

# Replace all matched run blocks by inserting "set -e" where needed.
tool.replace_all(insert_set_e)

# Print the modified CI YAML
print("Modified CI YAML:")
print(tool.get_modified_text())

  Modified CI YAML:
      - name: Install APT Packages
        if: ${{ inputs.apt-packages }}
        shell: bash
        run: |
          set -e
          sudo apt-get update
          sudo apt-get install -y ${{ inputs.apt-packages }}
  
      - name: Install NPM Packages
        if: ${{ inputs.npm-packages }}
        shell: bash
        run: |
          set -e
          npm install -g ${{ inputs.npm-packages }}

Explanation

1.	Pattern Definition:

The regex pattern uses named groups:
• run_line matches the line that begins with run: | (with any indentation).
• blank_lines captures any blank lines immediately following the header.
• The negative lookahead (?![ \t]*set\s+-e) ensures that the first non-blank line after the header does not start with set -e.
2. Replacement Callback:
The callback insert_set_e constructs the new text for a run block by inserting a properly indented set -e line immediately after the header and any blank lines.
3. Tool Usage:
The tool is initialized in line mode so that it processes the file by lines, which is convenient for YAML files. The analysis is performed using the defined pattern, then replace_all is used with the callback to update all matches.

Running the above code will output the modified CI YAML where any run block missing the set -e line will have it inserted automatically.