Fixing Facade Error on commit counting #3123

sgoggins · 2025-04-08T15:22:41Z

Description

Various flavors of this error occur if there is any issue with the commit history:

Traceback (most recent call last):
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/ai.chaoss/augur/tasks/git/facade_tasks.py", line 366, in git_update_commit_count_weight
    commit_count = get_repo_commit_count(logger, facade_helper, repo_git)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/ai.chaoss/augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py", line 169, in get_repo_commit_count
    raise e
  File "/home/sean/github/ai.chaoss/augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py", line 164, in get_repo_commit_count
    check_commit_count_cmd = check_output(
                             ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', '--git-dir', '/mnt/external/ai-repos/mnt/repos/repos/hostedaugur/hosted/139113-github.com-spring-projects-spring-data-commons/spring-data-commons/.git', 'rev-list', '--count', 'HEAD']' returned non-zero exit status 128.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/redis.py", line 520, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/redis.py", line 520, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/redis.py", line 426, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 123fb7d4-2b7d-4ad7-975c-17379f9d1e89 raised CalledProcessError(128, ['git', '--git-dir', '/mnt/external/ai-repos/mnt/repos/repos/hostedaugur/hosted/139113-github.com-spring-projects-spring-data-commons/spring-data-commons/.git', 'rev-list', '--count', 'HEAD'])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/app/trace.py", line 468, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/app/trace.py", line 379, in on_error
    R = I.handle_error_state(
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/app/trace.py", line 178, in handle_error_state
    return {
           ^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/app/trace.py", line 225, in handle_failure
    task.backend.mark_as_failure(
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 217, in mark_as_failure
    self.on_chord_part_return(chain_elem_ctx, state, exc)
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/redis.py", line 539, in on_chord_part_return
    return self.chord_error_from_stack(callback, exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 309, in chord_error_from_stack
    return backend.fail_from_current_stack(callback.id, exc=exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 316, in fail_from_current_stack
    self.mark_as_failure(task_id, exc, exception_info.traceback)
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 172, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 528, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 956, in _store_result
    current_meta = self._get_task_meta_for(task_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 978, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/ai/lib/python3.11/site-packages/celery/backends/base.py", line 856, in get_key_for_task
    return key_t('').join([
           ^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected a bytes-like object, NoneType found

This PR fixes that issue.

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py:192

The new implementation of get_repo_commit_count is being commented out, leaving an older version active. Consider removing the old, commented-out code to avoid potential confusion and ensure that the intended improvements are actually in effect.

"""

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins · 2025-04-08T16:09:32Z

Latest update is fixing some issues and the task is usually succeeding:

(ai) sean@linda:~/github/ai.chaoss$ cat nohup.out | grep "Retry of 'git rev-list --count HEAD' failed" | wc -l
58
(ai) sean@linda:~/github/ai.chaoss$ cat nohup.out | grep "Error running 'git fsck':" | wc -l
65

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Copilot · 2025-04-08T16:32:18Z

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

+    "invalid index-pack" errors.
+    """
+    pack_dir = os.path.join(repo_loc, "objects", "pack")
+    pack_files = glob(os.path.join(pack_dir, "pack-*.pack"))


Ensure that the glob module is imported to avoid runtime errors in the remove_corrupted_pack_files function.

Copilot · 2025-04-08T16:32:18Z

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

+                    f"Retry of git rev-list failed in {repo_loc} with return code "
+                    f"{retry_error.returncode}: {retry_stderr}"
+                )
+                return 0


Returning 0 on retry failure may mask underlying repository issues; consider propagating the error instead for clearer error handling.

Suggested change

return 0

raise retry_error

A reasonable point. investigating the issues logged here right now.

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

Copilot · 2025-04-08T20:06:56Z

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

+            # For other return codes, re-raise the error.
+            raise initial_error
+
+    except Exception as e:


[nitpick] Catching a broad Exception may mask specific error conditions. It is recommended to catch more specific exceptions to handle unexpected errors more accurately.

Suggested change

except Exception as e:

except (OSError, ValueError) as e:

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

augur/tasks/git/util/facade_worker/facade_worker/_old_method.py:1

[nitpick] The entire file is commented out legacy code. If it’s no longer needed, consider removing it to prevent confusion and reduce maintenance overhead.

'''

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

Copilot · 2025-04-08T21:22:05Z

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

+    pack_files = glob(os.path.join(pack_dir, "pack-*.pack"))
+    idx_files = glob(os.path.join(pack_dir, "pack-*.idx"))


Similarly, use 'glob.glob' instead of 'glob' to correctly obtain the index files.

Suggested change

pack_files = glob(os.path.join(pack_dir, "pack-*.pack"))

idx_files = glob(os.path.join(pack_dir, "pack-*.idx"))

pack_files = glob.glob(os.path.join(pack_dir, "pack-*.pack"))

idx_files = glob.glob(os.path.join(pack_dir, "pack-*.idx"))

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py:147

The glob function is being used directly despite importing the entire 'glob' module. To avoid runtime errors, either import the glob function using 'from glob import glob' or call it with the module reference as 'glob.glob(...)'.

pack_files = glob(os.path.join(pack_dir, "pack-*.pack"))

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Copilot · 2025-04-08T21:26:36Z

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py

+            remove_corrupted_pack_files(repo_loc, logger)
+
+            # If the error message indicates a bad remote reference, attempt to fix it.
+            bad_ref = "refs/remotes/origin/dependabot/docker/test/IntegrationTests/docker/postgres-16.4"


The bad_ref value is hard-coded. Consider making this value configurable or documenting why this specific reference is targeted to ensure future maintainability.

Suggested change

bad_ref = "refs/remotes/origin/dependabot/docker/test/IntegrationTests/docker/postgres-16.4"

bad_ref = get_bad_ref()

…s.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py:230

Ensure that 'check_output' and 'CalledProcessError' are explicitly imported from the subprocess module to avoid runtime errors if they are not imported elsewhere.

output = check_output(

augur/tasks/git/util/facade_worker/facade_worker/_old_method.py:1

[nitpick] Consider removing or archiving this legacy _old_method file if it is no longer needed to reduce code clutter and maintenance overhead.

'''

Signed-off-by: Sean P. Goggins <s@goggins.com>

Signed-off-by: Sean Goggins <outdoors@acm.org>

sgoggins added 5 commits April 8, 2025 09:20

bug fix check

4073865

Signed-off-by: Sean P. Goggins <s@goggins.com>

updated

b9d0ff0

Signed-off-by: Sean P. Goggins <s@goggins.com>

maybe this?

0ce6ec4

Signed-off-by: Sean P. Goggins <s@goggins.com>

more error checking

290d291

Signed-off-by: Sean P. Goggins <s@goggins.com>

fixing mistake

b55914f

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins requested review from ABrain7710 and Copilot April 8, 2025 15:22

Copilot AI reviewed Apr 8, 2025

View reviewed changes

sgoggins added server Related to the Augur server workers Related to data workers bug-fix Fixes a bug python Pull requests that update Python code labels Apr 8, 2025

fixing comparison

36d3e31

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins requested a review from Copilot April 8, 2025 15:28

Copilot AI reviewed Apr 8, 2025

View reviewed changes

sgoggins added 2 commits April 8, 2025 10:43

updated to try and reset the repository clone if there's an error

fbf2791

Signed-off-by: Sean P. Goggins <s@goggins.com>

more retry logic

9c26c58

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins added 2 commits April 8, 2025 11:15

updated for more attempts to fix

ff451fd

Signed-off-by: Sean P. Goggins <s@goggins.com>

handling git pack errors

d9353d4

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins requested a review from Copilot April 8, 2025 16:31

Copilot AI reviewed Apr 8, 2025

View reviewed changes

sgoggins added 2 commits April 8, 2025 11:38

catching additional errors

5f906ff

Signed-off-by: Sean P. Goggins <s@goggins.com>

added missing import

8db709e

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins requested a review from Copilot April 8, 2025 20:06

Copilot AI reviewed Apr 8, 2025

View reviewed changes

updated with additional error checking that worked manually

f16b361

Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins requested a review from Copilot April 8, 2025 21:21

Copilot AI reviewed Apr 8, 2025

View reviewed changes

sgoggins requested a review from Copilot April 8, 2025 21:25

updated import

c3bbcb7

Signed-off-by: Sean P. Goggins <s@goggins.com>

Copilot AI reviewed Apr 8, 2025

View reviewed changes

sgoggins requested a review from Copilot April 8, 2025 21:26

Copilot AI reviewed Apr 8, 2025

View reviewed changes

Update augur/tasks/git/util/facade_worker/facade_worker/utilitymethod…

cf425ee

…s.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sean P. Goggins <s@goggins.com>

sgoggins requested a review from Copilot April 8, 2025 21:27

Copilot AI reviewed Apr 8, 2025

View reviewed changes

sgoggins and others added 5 commits April 8, 2025 16:37

fixing hang

1e485c6

Signed-off-by: Sean P. Goggins <s@goggins.com>

resetting to working version

68bded4

Signed-off-by: Sean P. Goggins <s@goggins.com>

Addressing issue with PR commit URL's being non present

fad372e

Signed-off-by: Sean P. Goggins <s@goggins.com>

update

08bb67a

Signed-off-by: Sean Goggins <outdoors@acm.org>

fixing dm table script

f5a8220

Signed-off-by: Sean Goggins <outdoors@acm.org>

sgoggins marked this pull request as draft April 30, 2025 21:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing Facade Error on commit counting #3123

Fixing Facade Error on commit counting #3123

sgoggins commented Apr 8, 2025

Copilot AI left a comment

Copilot AI left a comment

sgoggins commented Apr 8, 2025

Copilot AI left a comment

Copilot AI Apr 8, 2025

Copilot AI Apr 8, 2025

sgoggins Apr 8, 2025

Copilot AI left a comment

Copilot AI Apr 8, 2025

Copilot AI left a comment

Copilot AI Apr 8, 2025

Copilot AI left a comment

Copilot AI left a comment

Copilot AI Apr 8, 2025

Copilot AI left a comment

		pack_files = glob(os.path.join(pack_dir, "pack-*.pack"))
		idx_files = glob(os.path.join(pack_dir, "pack-*.idx"))

	bad_ref = "refs/remotes/origin/dependabot/docker/test/IntegrationTests/docker/postgres-16.4"
	bad_ref = get_bad_ref()

Fixing Facade Error on commit counting #3123

Are you sure you want to change the base?

Fixing Facade Error on commit counting #3123

Conversation

sgoggins commented Apr 8, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

sgoggins commented Apr 8, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

sgoggins Apr 8, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment