From c6b2cb5e4957e675a84f06e643b7fa64e63f2da6 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:20:37 -0600 Subject: [PATCH 01/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index 1cf74e11e6..d237d9e372 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -50,6 +50,154 @@ The aims of the project are as follows: - (Stretch Goal) Improve the operation of the overall machine learning insights pipeline in CHAOSS/augur, and generalize these capabilities. + +## IDEA: Implement Conversion Rate Metric in CHAOSS Software + +**Hours: 350** + +[Micro-tasks and place for questions](https://github.com/chaoss/community/issues/305) + +### Conversion Rate + +Question: What are the rates at which new contributors become more sustained contributors? + +### Description + +The conversion rate metric is primarily aimed at identifying how new community members become more sustained contributors over time. However, the conversion rate metric can also help understand the changing roles of contributors, how a community is growing or declining, and paths to maintainership within an open source community. + +### Objectives (why) + - Observe if new members are becoming more involved with an open source project + - Observe if new members are taking on leadership roles within an open source project + - Observe if outreach efforts are generating new contributors to an open source project + - Observe if outreach efforts are impacting roles of existing community members + - Observe if community conflict results in changing roles within an open source community + - Identify casual, regular, and core contributors + +### Implementation + +This project could be implemented using either the CHAOSS/Augur, or CHAOSS/Grimoirelab (including stack components noted in references) technology stacks. + +The aims of the project are as follows: + - Implement the Conversion Rate Metric in CHAOSS Software + - After discussion, consider which CHAOSS Software Stack you wish to work with + - In collaboration with mentors, define the technology framework, and initial path to a "hello world" version of the metric + - Iterative development of the metric + - Assist in the deployment of this metric for a pre-determined collection of repositories in a publicly viewable website linked to the CHAOSS project. + - Advance the work of the [chaoss metrics models working group](https://github.com/chaoss/wg-metrics-models). + +* _Difficulty:_ Medium +* _Requirements:_ Knowledge of Python is desired. Some knowledge of Javascript or twitter/bootstrap is also desired. Key requirement is a keenness to dig into this challenge! +* _Recommended:_ Python experience. +* _Mentors:_ Sean Goggins, Daniel Izquerdo + +#### Filters (optional) + - Commits + - Issue creation + - Issue comments + - Change request creation + - Change request comments + - Merged change requests + - Code Reviews + - Code Review Comments + - Reactions (emoji) + - Chat platform messages + - Maillist messages + - Meetup attendance + +#### Visualizations + +![](./images/gsoc-1.png) + +Source: https://chaoss.github.io/grimoirelab-sigils/assets/images/screenshots/sigils/overall-community-structure.png + +![](./images/gsoc-2.png) + +Source: https://opensource.com/sites/default/files/uploads/2021-09-15-developer-level-02.png + +#### Tools Providing the Metric + - GrimoireLab + - Augur + - openEuler Infra + +#### Data Collection Strategies + +The following is an example from the [openEuler](https://www.openeuler.org/en/) community: + - A group of people who attended an offline event A held by the community, can be identified as Group A. Demographic information of Group A could be fetched from an on-line survey when people register for the event. To identify the conversation rate of these participants: + - Some people from Group A started watching and forking the repos, indicating they have shown some interest in this community. We marked them as subgroup D0 (Developer Level 0) as a subset of Group A. + - Conversion rate from the total number of people in Group A to the number of people in subgroup D0 is: D0/Group A + - Some people from subgroup D0 make more contributions beyond just watching or forking, including creating issues, making comments on an issue, or performed a code review. We marked them as subgroup D1 (Developer Level 1) as a subset of D0. + - Conversion rate from the total number of people in Subgroup D0 to the number of people in subgroup D1 is: D1/D0. + - Some people from subgroup D1 continue to make more contributions, like code contributions, to the project. This could include creating merge requests and merging new project code. We marked them as subgroup D2 (Developer Level 2) as a subset of D1. + - Conversion rate from the total number of people in subgroup D1 to the number of people in subgroup D2 is: D2/D1. + +![](./images/gsoc-3.png) + + Definition: + - Developer Level 0 (D0) example: Contributors who have given the project a star, or are watching or have forked the repository + - Developer Level 1 (D1): Contributors who have created issues, made comments on an issue, or performed a code review + - Developer Level 2 (D2): Contributors who have created a merge request and successfully merged code + - Conversion Rate (Group A -> D0): CR (Group A -> D2) = D0/Group A + - Conversion Rate (D0 -> D1): CR (D0 -> D1) = D1/D0 + - Conversion Rate (D1 -> D2): CR (D1 -> D2) = D2/D1 + +### References + - https://opensource.com/article/21/11/data-open-source-contributors + - https://github.com/chaoss/augur + - https://gitee.com/openeuler/website-v2/blob/master/web-ui/docs/en/blog/zhongjun/2021-09-15-developer-level.md + - https://chaoss.github.io/grimoirelab-sigils/common/onion_analysis/ + - https://mikemcquaid.com/2018/08/14/the-open-source-contributor-funnel-why-people-dont-contribute-to-your-open-source-project/ +### Contributors + - Yehui Wang + - Clement Li + - zhongjun + - Xiaoya Xia + - Matt Germonprez + - Sean Goggins + - King Gao + + + +## IDEA: Open Source Software Health Metrics Visualization Exploration + +**Hours: 300** + +[Micro-tasks and place for questions](https://github.com/chaoss/augur-community-reports/issues/34) + +The CHAOSS Community currently delivers pre-packaged visualizations of open source software health data through Augur APIs (https://github.com/chaoss/augur/blob/main/augur/routes/pull_request_reports.py and https://github.com/chaoss/augur/blob/main/augur/routes/contributor_reports.py), and the https://github.com/chaoss/augur-community-reports repository. This project seeks to expand, refine, and standardize the visualization of different classes of community health metrics data. Specifically, some analyses are temporal, others are anomaly driven, and in some cases contrasts across repositories and communities are required. In each case, the visualization of data is an essential component for metrics, and what we are now referring to as metrics models (https://github.com/chaoss/wg-metrics-models). + +Additional resources include: http://new.augurlabs.io/ && https://github.com/augurlabs/augur_view which demonsrate the updated twitter/bootstrap Augur frontend. + +The aims of the project are as follows: + - Experiment with standard metrics visualizations using direct Augur database connections, or through the Augur API. + - Refine metrics, and metrics model visualizations using Jupyter Notebooks are similar technology. + - Transform visualizations, as they are completed, into Augur API endpoints, following the pull request, and contributor reports examples. + +* _Difficulty:_ Medium +* _Requirements:_ Strong interest in data visualization. +* _Recommended:_ Experience with Python is desirable, and experience designing, or developing visualizations is desirable. +* _Mentors:_ Isaac Milarsky, Andrew Brain + +## Idea: Build Knowledgebase Application on CHAOSS Website + +**Hours: 350** + +Micro-tasks and place for questions: https://github.com/chaoss/website/issues/708 + +While much of the work of CHAOSS is done in GitHub, the CHAOSS website is often the first place people visit to get information [https://chaoss.community/](https://chaoss.community/). The goal of the website is to create clear paths for new members who want to contribute, metrics users who want information about metrics, and existing members who need information about project operations. As project grows, there is a need for alternative display and categorization options for knowledgebase topics to reduce the burden on website visitors in finding the information that they need. + +Work on this project would require the student to work closely with the mentors and the community to come up with different display and categorization options for CHAOSS Knowledge Base topics. CHAOSS knowledgebase topics that are currently under consideration for this application are released metrics, metrics models, and contributor handbook information. Information about these topics are captured and stored in GitHub repositories by the relevant working groups. The application will need to pull information from github markdown documents to display on the website knowledge base application (we have existing code that does this). + +The aims of the project are as follows: +* Use Wordpress to implement a knowledge application (example knowledgebase plugin up for consideration - https://wordpress.org/plugins/basepress/) +* Research and ideate different display options and categorizations for knowledgebase topics. +* Build web pages to display different knowledgebase topics. + +The aims will require working with front-end web development technologies and WordPress to build a knowledgebase application that can display information about knowledgebase topics. + +* _Difficulty:_ Low +* _Requirements:_ Interest in front-end web development +* _Recommended:_ Experience with Wordpress, HTML, CSS, JavaScript, and GitHub Markdown +* _Mentors:_ John McGinness, Sean Goggins * _Difficulty:_ Medium * _Requirements:_ Interest in software analytics. Python programming. Conceptual understanding of machine learning, and an eagerness to learn maching learning, and SQL knowledge. * _Recommended:_ Experience with Python From f98b5dbcab8864ac21e83d9e840639259bba7e6f Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:23:12 -0600 Subject: [PATCH 02/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index d237d9e372..19e91c5f1a 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -55,7 +55,7 @@ The aims of the project are as follows: **Hours: 350** -[Micro-tasks and place for questions](https://github.com/chaoss/community/issues/305) +[Micro-tasks and place for questions]([https://github.com/chaoss/community/issues/305](https://github.com/chaoss/augur/issues/2992)) ### Conversion Rate From 8b1a9396e1c104e24c698075b1991d93d094e638 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:25:52 -0600 Subject: [PATCH 03/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index 19e91c5f1a..719de62cef 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -55,7 +55,7 @@ The aims of the project are as follows: **Hours: 350** -[Micro-tasks and place for questions]([https://github.com/chaoss/community/issues/305](https://github.com/chaoss/augur/issues/2992)) +[Micro-tasks and place for questions](https://github.com/chaoss/augur/issues/2992) ### Conversion Rate @@ -161,7 +161,7 @@ The following is an example from the [openEuler](https://www.openeuler.org/en/) **Hours: 300** -[Micro-tasks and place for questions](https://github.com/chaoss/augur-community-reports/issues/34) +[Micro-tasks and place for questions]([https://github.com/chaoss/augur-community-reports/issues/34](https://github.com/chaoss/augur/issues/2993)) The CHAOSS Community currently delivers pre-packaged visualizations of open source software health data through Augur APIs (https://github.com/chaoss/augur/blob/main/augur/routes/pull_request_reports.py and https://github.com/chaoss/augur/blob/main/augur/routes/contributor_reports.py), and the https://github.com/chaoss/augur-community-reports repository. This project seeks to expand, refine, and standardize the visualization of different classes of community health metrics data. Specifically, some analyses are temporal, others are anomaly driven, and in some cases contrasts across repositories and communities are required. In each case, the visualization of data is an essential component for metrics, and what we are now referring to as metrics models (https://github.com/chaoss/wg-metrics-models). From f847358047d2857a7ac6c293a66179b71999f8f0 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:27:19 -0600 Subject: [PATCH 04/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index 719de62cef..b61faaec99 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -161,7 +161,7 @@ The following is an example from the [openEuler](https://www.openeuler.org/en/) **Hours: 300** -[Micro-tasks and place for questions]([https://github.com/chaoss/augur-community-reports/issues/34](https://github.com/chaoss/augur/issues/2993)) +[Micro-tasks and place for questions](https://github.com/chaoss/augur/issues/2993) The CHAOSS Community currently delivers pre-packaged visualizations of open source software health data through Augur APIs (https://github.com/chaoss/augur/blob/main/augur/routes/pull_request_reports.py and https://github.com/chaoss/augur/blob/main/augur/routes/contributor_reports.py), and the https://github.com/chaoss/augur-community-reports repository. This project seeks to expand, refine, and standardize the visualization of different classes of community health metrics data. Specifically, some analyses are temporal, others are anomaly driven, and in some cases contrasts across repositories and communities are required. In each case, the visualization of data is an essential component for metrics, and what we are now referring to as metrics models (https://github.com/chaoss/wg-metrics-models). @@ -176,29 +176,3 @@ The aims of the project are as follows: * _Requirements:_ Strong interest in data visualization. * _Recommended:_ Experience with Python is desirable, and experience designing, or developing visualizations is desirable. * _Mentors:_ Isaac Milarsky, Andrew Brain - -## Idea: Build Knowledgebase Application on CHAOSS Website - -**Hours: 350** - -Micro-tasks and place for questions: https://github.com/chaoss/website/issues/708 - -While much of the work of CHAOSS is done in GitHub, the CHAOSS website is often the first place people visit to get information [https://chaoss.community/](https://chaoss.community/). The goal of the website is to create clear paths for new members who want to contribute, metrics users who want information about metrics, and existing members who need information about project operations. As project grows, there is a need for alternative display and categorization options for knowledgebase topics to reduce the burden on website visitors in finding the information that they need. - -Work on this project would require the student to work closely with the mentors and the community to come up with different display and categorization options for CHAOSS Knowledge Base topics. CHAOSS knowledgebase topics that are currently under consideration for this application are released metrics, metrics models, and contributor handbook information. Information about these topics are captured and stored in GitHub repositories by the relevant working groups. The application will need to pull information from github markdown documents to display on the website knowledge base application (we have existing code that does this). - -The aims of the project are as follows: -* Use Wordpress to implement a knowledge application (example knowledgebase plugin up for consideration - https://wordpress.org/plugins/basepress/) -* Research and ideate different display options and categorizations for knowledgebase topics. -* Build web pages to display different knowledgebase topics. - -The aims will require working with front-end web development technologies and WordPress to build a knowledgebase application that can display information about knowledgebase topics. - -* _Difficulty:_ Low -* _Requirements:_ Interest in front-end web development -* _Recommended:_ Experience with Wordpress, HTML, CSS, JavaScript, and GitHub Markdown -* _Mentors:_ John McGinness, Sean Goggins -* _Difficulty:_ Medium -* _Requirements:_ Interest in software analytics. Python programming. Conceptual understanding of machine learning, and an eagerness to learn maching learning, and SQL knowledge. -* _Recommended:_ Experience with Python -* _Mentors:_ Sean Goggins, Andrew Brain, Isaac Milarsky From 1ffc551c3d5ecf93652a76a9190e644ac0585504 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:45:52 -0600 Subject: [PATCH 05/66] Update gsoc-interest.md Signed-off-by: Sean P. Goggins --- gsoc-interest.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/gsoc-interest.md b/gsoc-interest.md index 04a237581d..e737687a01 100644 --- a/gsoc-interest.md +++ b/gsoc-interest.md @@ -1,5 +1,7 @@ # Google Summer of Code 2025 Interested Candidates +[IDEAS PAGE](https://github.com/chaoss/augur/blob/main/gsoc-ideas.md) + Hi potential GSoC students, You can ask questions and meet the community on Slack here: https://join.slack.com/t/chaoss-workspace/shared_invite/zt-289zxh6tu-3oQaFlutPFY039MjKpnWcA ... look for the `wg-augur-8knot` channel. From ffec2844ff4917b5e32c2453659a7bdd27325a3c Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:50:55 -0600 Subject: [PATCH 06/66] Update README.md Signed-off-by: Sean P. Goggins --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index f65deb7535..b7baa57845 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,6 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -<<<<<<< HEAD Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.81.0). From a653027954617ac9d3cdf96d775a65037862f2ac Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Feb 2025 15:51:47 -0600 Subject: [PATCH 07/66] Update README.md Signed-off-by: Sean P. Goggins --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b7baa57845..3b6f08fb1c 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ We strongly believe that much of what makes open source so great is the incredib ## License, Copyright, and Funding -Copyright © 2023 University of Nebraska at Omaha, University of Missouri, Brian Warner, and the CHAOSS Project. +Copyright © 2025 University of Nebraska at Omaha, University of Missouri, Brian Warner, and the CHAOSS Project. Augur is free software: you can redistribute it and/or modify it under the terms of the MIT License as published by the Open Source Initiative. See the [LICENSE](LICENSE) file for more details. From 9500e7367e0715d49f2d39cc3061f67e41697f62 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Thu, 13 Feb 2025 10:40:23 -0600 Subject: [PATCH 08/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index b61faaec99..c376908c50 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -159,7 +159,7 @@ The following is an example from the [openEuler](https://www.openeuler.org/en/) ## IDEA: Open Source Software Health Metrics Visualization Exploration -**Hours: 300** +**Hours: 350** [Micro-tasks and place for questions](https://github.com/chaoss/augur/issues/2993) From 32d9a8ce9822e10bc017547b54e1b01212ddb76f Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Thu, 13 Feb 2025 12:42:07 -0600 Subject: [PATCH 09/66] Fix key orchestrator missing in Docker build - Copy keyman directory in Dockerfile - Update graphical dockerfile to reflect the current build process Signed-off-by: Ulincsys --- docker/backend/Dockerfile | 1 + docker/backend/graphical | 91 ++++++++++++++++++++++++++++++++++----- 2 files changed, 82 insertions(+), 10 deletions(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 220bb8ff49..03dae6b01d 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -80,6 +80,7 @@ COPY ./augur/ augur/ COPY ./metadata.py . COPY ./setup.py . COPY ./scripts/ scripts/ +COPY ./keyman/ keyman/ # Add rust and cargo to PATH ENV PATH="/usr/bin/:/root/.cargo/bin:/usr/local/bin:${PATH}" diff --git a/docker/backend/graphical b/docker/backend/graphical index a1c6b95d39..b5c223befe 100644 --- a/docker/backend/graphical +++ b/docker/backend/graphical @@ -1,22 +1,75 @@ -#SPDX-License-Identifier: MIT -FROM python:3.9-slim-bullseye +# SPDX-License-Identifier: MIT +FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" -LABEL version="0.51.1" +LABEL version="0.76.6" ENV DEBIAN_FRONTEND=noninteractive +ENV PATH="/usr/bin/:/usr/local/bin:/usr/lib:${PATH}" RUN set -x \ && apt-get update \ - && apt-get -y install --no-install-recommends \ + && apt-get -y install \ git \ bash \ curl \ gcc \ - python3-pip \ + software-properties-common \ + postgresql-contrib \ + musl-dev \ + python3-dev \ + python3-distutils \ + python3-venv \ wget \ postgresql-client \ - && rm -rf /var/lib/apt/lists/* + libpq-dev \ + build-essential \ + rustc \ + cargo \ + chromium \ + tar \ + jq \ + chromium-driver \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* \ + && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y + +# Install Firefox from Debian repositories for ARM64 architecture +RUN set -x \ + && apt-get update \ + && apt-get install -y firefox-esr + +# Install Geckodriver +RUN GECKODRIVER_VERSION=$(curl -s https://api.github.com/repos/mozilla/geckodriver/releases/latest | jq -r '.tag_name' | sed 's/v//') \ + && ARCH=$(uname -m) \ + && if [ "$ARCH" = "aarch64" ]; then \ + GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/v${GECKODRIVER_VERSION}/geckodriver-v${GECKODRIVER_VERSION}-linux-aarch64.tar.gz"; \ + GECKODRIVER_FILE="geckodriver-v${GECKODRIVER_VERSION}-linux-aarch64.tar.gz"; \ + else \ + GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/v${GECKODRIVER_VERSION}/geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz"; \ + GECKODRIVER_FILE="geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz"; \ + fi \ + && wget $GECKODRIVER_URL \ + && tar -xzf $GECKODRIVER_FILE \ + && mv geckodriver /usr/local/bin/ \ + && rm $GECKODRIVER_FILE + +# Verify installations +RUN firefox --version +RUN geckodriver --version + +# Ensure Rust directories are writable +RUN mkdir -p /root/.rustup/downloads /root/.cargo/registry && \ + chmod -R 777 /root/.rustup /root/.cargo + +# Add rust and cargo to PATH +ENV PATH="/root/.cargo/bin:${PATH}" + +# Install the specific version of Rust +RUN set -x \ + && rustup install 1.78.0 +RUN set -x \ + && rustup default 1.78.0 EXPOSE 5000 @@ -27,20 +80,38 @@ COPY ./augur/ augur/ COPY ./metadata.py . COPY ./setup.py . COPY ./scripts/ scripts/ +COPY ./keyman/ keyman/ + +# Add rust and cargo to PATH +ENV PATH="/usr/bin/:/root/.cargo/bin:/usr/local/bin:${PATH}" -#COPY ./docker/backend/docker.config.json . RUN python3 -m venv /opt/venv +RUN set -x \ + && /opt/venv/bin/pip install --upgrade pip + +RUN set -x \ + && /opt/venv/bin/pip install wheel + RUN set -x \ && /opt/venv/bin/pip install . -RUN ./scripts/docker/install-workers-deps.sh +RUN set -x \ + && /opt/venv/bin/pip install --upgrade pip \ + && /opt/venv/bin/pip install wheel \ + && /opt/venv/bin/pip install . RUN ./scripts/docker/install-go.sh +ENV PATH="${PATH}:/usr/local/go/bin" +RUN ./scripts/docker/install-workers-deps.sh + # RUN ./scripts/install/workers.sh RUN mkdir -p repos/ logs/ /augur/facade/ COPY ./docker/backend/graphical.sh / -RUN chmod +x /graphical.sh -ENTRYPOINT /graphical.sh +COPY ./docker/backend/init.sh / +RUN chmod +x /entrypoint.sh /init.sh +ENTRYPOINT ["/bin/bash", "/graphical.sh"] +#ENTRYPOINT ["/entrypoint.sh"] +CMD /init.sh From 1393440af0916afb1106669ca603fb5d17c5420c Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Fri, 14 Feb 2025 03:41:45 -0600 Subject: [PATCH 10/66] Add __init__.py to fix module import error Signed-off-by: Ulincsys --- keyman/__init__.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 keyman/__init__.py diff --git a/keyman/__init__.py b/keyman/__init__.py new file mode 100644 index 0000000000..e69de29bb2 From def43f17079293793718bc73b3f0717aa2dda4f8 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Fri, 14 Feb 2025 09:07:57 -0600 Subject: [PATCH 11/66] Update metadata.py Signed-off-by: Sean P. Goggins --- metadata.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/metadata.py b/metadata.py index 2ac76c87d3..5a3d190a96 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.81.0" -__release__ = "v0.81.0 (Super Soaker)" +__version__ = "0.81.1" +__release__ = "v0.81.1 (Super Soaker)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2112" From c6646ceef1bc2a479a8471097464c393bac42c94 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Fri, 14 Feb 2025 09:09:42 -0600 Subject: [PATCH 12/66] Update README.md Signed-off-by: Sean P. Goggins --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3b6f08fb1c..f33f9e5ae7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.81.0 +# Augur NEW Release v0.81.1 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.81.0). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.81.1). - The `main` branch is a stable version of our new architecture, which features: @@ -43,9 +43,9 @@ For more information on [how to get involved on the CHAOSS website](https://chao ## Collecting Data -Augur supports ```Python3.6``` through ```Python3.9``` on all platforms. ```Python3.10``` and above do not yet work because of machine learning worker dependencies. On OSX, you can create a ```Python3.9``` environment, by running: +Augur supports ```Python3.7``` through ```Python3.8``` on all platforms. ```Python3.12``` and above do not yet work because of machine learning worker dependencies. On OSX, you can create a ```Python3.11``` environment, by running: ``` -$ python3.9 -m venv path/to/venv +$ python3.11 -m venv path/to/venv ``` Augur's main focus is to measure the overall health and sustainability of open source projects. From 03196c5354e26779fbf6bfbe16afb4445e2c1bc7 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Fri, 14 Feb 2025 09:10:18 -0600 Subject: [PATCH 13/66] Update README.md Signed-off-by: Sean P. Goggins --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f33f9e5ae7..5ef9deb953 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ For more information on [how to get involved on the CHAOSS website](https://chao ## Collecting Data -Augur supports ```Python3.7``` through ```Python3.8``` on all platforms. ```Python3.12``` and above do not yet work because of machine learning worker dependencies. On OSX, you can create a ```Python3.11``` environment, by running: +Augur supports ```Python3.7``` through ```Python3.11``` on all platforms. ```Python3.12``` and above do not yet work because of machine learning worker dependencies. On OSX, you can create a ```Python3.11``` environment, by running: ``` $ python3.11 -m venv path/to/venv ``` From 46bf0343e5f0512d16b493533f89df18be30cd53 Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Mon, 17 Feb 2025 10:52:03 -0600 Subject: [PATCH 14/66] Show tracebacks for network exceptions Signed-off-by: Ulincsys --- augur/application/cli/__init__.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/augur/application/cli/__init__.py b/augur/application/cli/__init__.py index 00f41a5534..b3bdae7797 100644 --- a/augur/application/cli/__init__.py +++ b/augur/application/cli/__init__.py @@ -6,6 +6,7 @@ import re import json import httpx +import traceback from augur.application.db.engine import DatabaseEngine from augur.application.db import get_engine, dispose_database_engine @@ -24,13 +25,16 @@ def new_func(ctx, *args, **kwargs): return ctx.invoke(function_internet_connection, *args, **kwargs) except (TimeoutError, httpx.TimeoutException): print("Request timed out.") - except httpx.NetworkError: + except httpx.NetworkError as e: print(f"Network Error: {httpx.NetworkError}") - except httpx.ProtocolError: + print(traceback.format_exc(e)) + except httpx.ProtocolError as e: print(f"Protocol Error: {httpx.ProtocolError}") + print(traceback.format_exc(e)) + print(f"\n\n{usage} command setup failed\n \ - You are not connected to the internet.\n \ - Please connect to the internet to run Augur\n \ + There was an error while testing for network connectivity.\n \ + Please check your connection to the internet to run Augur\n \ Consider setting http_proxy variables for limited access installations.") sys.exit(-1) From 1d9583d3373794240c529df23b0b4083d582150c Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Mon, 17 Feb 2025 11:27:54 -0600 Subject: [PATCH 15/66] Fix incorrect usage of format_exc Signed-off-by: Ulincsys --- augur/application/cli/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/augur/application/cli/__init__.py b/augur/application/cli/__init__.py index b3bdae7797..d20aaad5a6 100644 --- a/augur/application/cli/__init__.py +++ b/augur/application/cli/__init__.py @@ -27,10 +27,10 @@ def new_func(ctx, *args, **kwargs): print("Request timed out.") except httpx.NetworkError as e: print(f"Network Error: {httpx.NetworkError}") - print(traceback.format_exc(e)) + print(traceback.format_exc()) except httpx.ProtocolError as e: print(f"Protocol Error: {httpx.ProtocolError}") - print(traceback.format_exc(e)) + print(traceback.format_exc()) print(f"\n\n{usage} command setup failed\n \ There was an error while testing for network connectivity.\n \ From 53c3c14db9daff59262cab2d28b7560e018fe38f Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Mon, 17 Feb 2025 11:51:12 -0600 Subject: [PATCH 16/66] Have decorator pass secondary exceptions transparently Signed-off-by: Ulincsys --- augur/application/cli/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/augur/application/cli/__init__.py b/augur/application/cli/__init__.py index d20aaad5a6..998f341e23 100644 --- a/augur/application/cli/__init__.py +++ b/augur/application/cli/__init__.py @@ -21,8 +21,6 @@ def new_func(ctx, *args, **kwargs): try: _ = client.request( method="GET", url="http://chaoss.community", timeout=10, follow_redirects=True) - - return ctx.invoke(function_internet_connection, *args, **kwargs) except (TimeoutError, httpx.TimeoutException): print("Request timed out.") except httpx.NetworkError as e: @@ -36,7 +34,9 @@ def new_func(ctx, *args, **kwargs): There was an error while testing for network connectivity.\n \ Please check your connection to the internet to run Augur\n \ Consider setting http_proxy variables for limited access installations.") - sys.exit(-1) + sys.exit(-1) + + return ctx.invoke(function_internet_connection, *args, **kwargs) return update_wrapper(new_func, function_internet_connection) From 6954f9c491d81e35a9ec5d22e2aa2de34b93a2f7 Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Mon, 17 Feb 2025 12:00:53 -0600 Subject: [PATCH 17/66] Logical fixes and general improvements - Fix logical issue with try block - Use multiline string for logging Signed-off-by: Ulincsys --- augur/application/cli/__init__.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/augur/application/cli/__init__.py b/augur/application/cli/__init__.py index 998f341e23..e68af307bb 100644 --- a/augur/application/cli/__init__.py +++ b/augur/application/cli/__init__.py @@ -17,10 +17,12 @@ def test_connection(function_internet_connection): @click.pass_context def new_func(ctx, *args, **kwargs): usage = re.search(r"Usage:\s(.*)\s\[OPTIONS\]", str(ctx.get_usage())).groups()[0] + success = False with httpx.Client() as client: try: _ = client.request( method="GET", url="http://chaoss.community", timeout=10, follow_redirects=True) + success = True except (TimeoutError, httpx.TimeoutException): print("Request timed out.") except httpx.NetworkError as e: @@ -29,12 +31,16 @@ def new_func(ctx, *args, **kwargs): except httpx.ProtocolError as e: print(f"Protocol Error: {httpx.ProtocolError}") print(traceback.format_exc()) - - print(f"\n\n{usage} command setup failed\n \ - There was an error while testing for network connectivity.\n \ - Please check your connection to the internet to run Augur\n \ - Consider setting http_proxy variables for limited access installations.") - sys.exit(-1) + + if not success: + print( + f""" + \n\n{usage} command setup failed. + There was an error while testing for network connectivity + Please check your connection to the internet to run Augur + Consider setting http_proxy variables for limited access installations.""" + ) + sys.exit(-1) return ctx.invoke(function_internet_connection, *args, **kwargs) From 641f72f1655fbba998f15b1204fada0fb0177e1f Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Mon, 17 Feb 2025 13:03:27 -0600 Subject: [PATCH 18/66] Strip whitespace from keys while loading Signed-off-by: Ulincsys --- augur/tasks/github/util/github_api_key_handler.py | 2 ++ augur/tasks/gitlab/gitlab_api_key_handler.py | 2 ++ 2 files changed, 4 insertions(+) diff --git a/augur/tasks/github/util/github_api_key_handler.py b/augur/tasks/github/util/github_api_key_handler.py index 9ca777a1be..47933e67dc 100644 --- a/augur/tasks/github/util/github_api_key_handler.py +++ b/augur/tasks/github/util/github_api_key_handler.py @@ -107,6 +107,8 @@ def get_api_keys(self) -> List[str]: if len(keys) == 0: return [] + + keys = [key.strip() for key in keys] valid_keys = [] with httpx.Client() as client: diff --git a/augur/tasks/gitlab/gitlab_api_key_handler.py b/augur/tasks/gitlab/gitlab_api_key_handler.py index 03c0cf66c3..72b0ace148 100644 --- a/augur/tasks/gitlab/gitlab_api_key_handler.py +++ b/augur/tasks/gitlab/gitlab_api_key_handler.py @@ -110,6 +110,8 @@ def get_api_keys(self) -> List[str]: if len(keys) == 0: return [] + keys = [key.strip() for key in keys] + valid_keys = [] with httpx.Client() as client: From e6940c6b99cb60298cb51a76e4c35dd1c0c7afe3 Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Tue, 18 Feb 2025 12:51:26 -0600 Subject: [PATCH 19/66] Filter worker_oauth keys by platform in cli Signed-off-by: Ulincsys --- augur/application/cli/github.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/augur/application/cli/github.py b/augur/application/cli/github.py index cad13be793..4896bf05fb 100644 --- a/augur/application/cli/github.py +++ b/augur/application/cli/github.py @@ -32,7 +32,7 @@ def update_api_key(): """ SELECT value as github_key from config Where section_name='Keys' AND setting_name='github_api_key' UNION All - SELECT access_token as github_key from worker_oauth ORDER BY github_key DESC; + SELECT access_token as github_key from worker_oauth where platform='github' ORDER BY github_key DESC; """ ) From 9a4b8660e5d45cd7dd1dbf37459b98507a03d7ad Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Fri, 21 Feb 2025 13:30:01 -0600 Subject: [PATCH 20/66] Fix comparison with NoneType logical error Signed-off-by: Ulincsys --- augur/tasks/github/pull_requests/tasks.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/augur/tasks/github/pull_requests/tasks.py b/augur/tasks/github/pull_requests/tasks.py index b65da7f4f3..d88a503949 100644 --- a/augur/tasks/github/pull_requests/tasks.py +++ b/augur/tasks/github/pull_requests/tasks.py @@ -231,9 +231,13 @@ def collect_pull_request_review_comments(repo_git: str, full_collection: bool) - repo_id = get_repo_by_repo_git(repo_git).repo_id if not full_collection: - # subtract 2 days to ensure all data is collected - core_data_last_collected = (get_core_data_last_collected(repo_id) - timedelta(days=2)).replace(tzinfo=timezone.utc) - review_msg_url += f"?since={core_data_last_collected.isoformat()}" + last_collected_date = get_core_data_last_collected(repo_id) + + # the full_collection parameter is hardcoded to False, so we must check if there actually is a last_collected date + if last_collected_date: + # subtract 2 days to ensure all data is collected + core_data_last_collected = (last_collected_date - timedelta(days=2)).replace(tzinfo=timezone.utc) + review_msg_url += f"?since={core_data_last_collected.isoformat()}" pr_reviews = get_pull_request_reviews_by_repo_id(repo_id) From 1e417caeb39acaa678745c1ff09093712b978f73 Mon Sep 17 00:00:00 2001 From: Ulincsys Date: Fri, 21 Feb 2025 18:16:01 -0600 Subject: [PATCH 21/66] Secondary task to use get_secondary_data_last_collected Signed-off-by: Ulincsys --- augur/tasks/github/pull_requests/tasks.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/augur/tasks/github/pull_requests/tasks.py b/augur/tasks/github/pull_requests/tasks.py index d88a503949..21ef9d8eba 100644 --- a/augur/tasks/github/pull_requests/tasks.py +++ b/augur/tasks/github/pull_requests/tasks.py @@ -231,13 +231,14 @@ def collect_pull_request_review_comments(repo_git: str, full_collection: bool) - repo_id = get_repo_by_repo_git(repo_git).repo_id if not full_collection: - last_collected_date = get_core_data_last_collected(repo_id) + last_collected_date = get_secondary_data_last_collected(repo_id) - # the full_collection parameter is hardcoded to False, so we must check if there actually is a last_collected date if last_collected_date: # subtract 2 days to ensure all data is collected core_data_last_collected = (last_collected_date - timedelta(days=2)).replace(tzinfo=timezone.utc) - review_msg_url += f"?since={core_data_last_collected.isoformat()}" + review_msg_url += f"?since={core_data_last_collected.isoformat()}" + else: + logger.warning(f"core_data_last_collected is NULL for recollection on repo: {repo_git}") pr_reviews = get_pull_request_reviews_by_repo_id(repo_id) From 47a8c66e79a6c9751f09cdb35c12730e7225e0f5 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Sat, 22 Feb 2025 12:02:49 -0600 Subject: [PATCH 22/66] Update README.md Signed-off-by: Sean P. Goggins --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5ef9deb953..2f4fdb67a8 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.81.1 +# Augur NEW Release v0.81.2 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.81.1). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.81.2). - The `main` branch is a stable version of our new architecture, which features: From 474c65c444aa03c5a2779de8db384e62e1a288b8 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Sat, 22 Feb 2025 12:03:40 -0600 Subject: [PATCH 23/66] Update metadata.py Signed-off-by: Sean P. Goggins --- metadata.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/metadata.py b/metadata.py index 5a3d190a96..7c91b77f57 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.81.1" -__release__ = "v0.81.1 (Super Soaker)" +__version__ = "0.81.2" +__release__ = "v0.81.2 (Basketball Jones)" __license__ = "MIT" -__copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2112" +__copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From ec6b295b7bcc6e44c04559a166b28b16a0ea6aa6 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sat, 22 Feb 2025 15:18:02 -0500 Subject: [PATCH 24/66] move golang build step to its own build container Signed-off-by: Adrian Edwards --- docker/backend/Dockerfile | 15 ++++++++++++-- scripts/docker/install-golang-deps.sh | 26 +++++++++++++++++++++++++ scripts/docker/install-workers-deps.sh | 27 -------------------------- 3 files changed, 39 insertions(+), 29 deletions(-) create mode 100755 scripts/docker/install-golang-deps.sh diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 03dae6b01d..053fd5ae11 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -1,4 +1,14 @@ # SPDX-License-Identifier: MIT + + +FROM golang:1.23 AS golang + + +# RUN ./scripts/docker/install-go.sh +ENV PATH="${PATH}:/usr/local/go/bin" +RUN ./scripts/docker/install-golang-deps.sh + + FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" @@ -101,10 +111,11 @@ RUN set -x \ && /opt/venv/bin/pip install wheel \ && /opt/venv/bin/pip install . -RUN ./scripts/docker/install-go.sh -ENV PATH="${PATH}:/usr/local/go/bin" RUN ./scripts/docker/install-workers-deps.sh +COPY --from=golang "$HOME/scc" "$HOME/scc" +COPY --from=golang "$HOME/scorecard" "$HOME/scorecard" + # RUN ./scripts/install/workers.sh RUN mkdir -p repos/ logs/ /augur/facade/ diff --git a/scripts/docker/install-golang-deps.sh b/scripts/docker/install-golang-deps.sh new file mode 100755 index 0000000000..6661129ed8 --- /dev/null +++ b/scripts/docker/install-golang-deps.sh @@ -0,0 +1,26 @@ +#!/bin/bash +set -x + +# Note this +CURRENT_DIR=$PWD; +# 18GB +# Install scc +SCC_DIR="$HOME/scc" +echo "Cloning Sloc Cloc and Code (SCC) to generate value data ..." +git clone https://github.com/boyter/scc "$SCC_DIR" +cd $SCC_DIR +go build; +echo "scc build done" +cd $CURRENT_DIR +# 18GB + +# Install scorecard +SCORECARD_DIR="$HOME/scorecard" +echo "Cloning OSSF Scorecard to generate scorecard data ..." +git clone https://github.com/ossf/scorecard $SCORECARD_DIR +cd $SCORECARD_DIR +go build; +echo "scorecard build done" +cd $CURRENT_DIR + +# 16GB \ No newline at end of file diff --git a/scripts/docker/install-workers-deps.sh b/scripts/docker/install-workers-deps.sh index 188ec43622..c69e179486 100755 --- a/scripts/docker/install-workers-deps.sh +++ b/scripts/docker/install-workers-deps.sh @@ -8,30 +8,3 @@ do /opt/venv/bin/pip install . cd $OLD done - -# install nltk -# taken from ./scripts/install/nltk_dictionaries.sh -for i in stopwords punkt popular universal_tagset ; do - /opt/venv/bin/python -m nltk.downloader $i -done - -# Note this -CURRENT_DIR=$PWD; - -# Install scc -SCC_DIR="$HOME/scc" -echo "Cloning Sloc Cloc and Code (SCC) to generate value data ..." -git clone https://github.com/boyter/scc "$SCC_DIR" -cd $SCC_DIR -go build; -echo "scc build done" -cd $CURRENT_DIR - -# Install scorecard -SCORECARD_DIR="$HOME/scorecard" -echo "Cloning OSSF Scorecard to generate scorecard data ..." -git clone https://github.com/ossf/scorecard $SCORECARD_DIR -cd $SCORECARD_DIR -go build; -echo "scorecard build done" -cd $CURRENT_DIR From 3a075e05835a7c0cd966146d306c758cb25488cc Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sat, 22 Feb 2025 15:33:43 -0500 Subject: [PATCH 25/66] refactor rust stuff into a builder container too Signed-off-by: Adrian Edwards --- docker/backend/Dockerfile | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 053fd5ae11..83205bb3f6 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -9,6 +9,25 @@ ENV PATH="${PATH}:/usr/local/go/bin" RUN ./scripts/docker/install-golang-deps.sh +FROM rust:1.78 as rust + +# Ensure Rust directories are writable +RUN mkdir -p /root/.rustup/downloads /root/.cargo/registry && \ + chmod -R 777 /root/.rustup /root/.cargo + +# Add rust and cargo to PATH +# ENV PATH="/root/.cargo/bin:${PATH}" + +# Install the specific version of Rust +# RUN set -x \ +# && rustup install 1.78.0 +# RUN set -x \ +# && rustup default 1.78.0 + + +# Add rust and cargo to PATH +ENV PATH="/usr/bin/:/root/.cargo/bin:/usr/local/bin:${PATH}" + FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" @@ -34,15 +53,12 @@ RUN set -x \ postgresql-client \ libpq-dev \ build-essential \ - rustc \ - cargo \ chromium \ tar \ jq \ chromium-driver \ && apt-get clean \ - && rm -rf /var/lib/apt/lists/* \ - && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y + && rm -rf /var/lib/apt/lists/* # Install Firefox from Debian repositories for ARM64 architecture RUN set -x \ @@ -68,18 +84,9 @@ RUN GECKODRIVER_VERSION=$(curl -s https://api.github.com/repos/mozilla/geckodriv RUN firefox --version RUN geckodriver --version -# Ensure Rust directories are writable -RUN mkdir -p /root/.rustup/downloads /root/.cargo/registry && \ - chmod -R 777 /root/.rustup /root/.cargo -# Add rust and cargo to PATH -ENV PATH="/root/.cargo/bin:${PATH}" -# Install the specific version of Rust -RUN set -x \ - && rustup install 1.78.0 -RUN set -x \ - && rustup default 1.78.0 + EXPOSE 5000 @@ -92,8 +99,7 @@ COPY ./setup.py . COPY ./scripts/ scripts/ COPY ./keyman/ keyman/ -# Add rust and cargo to PATH -ENV PATH="/usr/bin/:/root/.cargo/bin:/usr/local/bin:${PATH}" + RUN python3 -m venv /opt/venv From c292154e7a72e9d9c7a4d949a4b6ae283afe8aa6 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sat, 22 Feb 2025 15:35:27 -0500 Subject: [PATCH 26/66] comment out rust builder as it seems unused Signed-off-by: Adrian Edwards --- docker/backend/Dockerfile | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 83205bb3f6..1ae4a44b87 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -9,24 +9,24 @@ ENV PATH="${PATH}:/usr/local/go/bin" RUN ./scripts/docker/install-golang-deps.sh -FROM rust:1.78 as rust +# FROM rust:1.78 as rust -# Ensure Rust directories are writable -RUN mkdir -p /root/.rustup/downloads /root/.cargo/registry && \ - chmod -R 777 /root/.rustup /root/.cargo +# # Ensure Rust directories are writable +# RUN mkdir -p /root/.rustup/downloads /root/.cargo/registry && \ +# chmod -R 777 /root/.rustup /root/.cargo -# Add rust and cargo to PATH -# ENV PATH="/root/.cargo/bin:${PATH}" +# # Add rust and cargo to PATH +# # ENV PATH="/root/.cargo/bin:${PATH}" -# Install the specific version of Rust -# RUN set -x \ -# && rustup install 1.78.0 -# RUN set -x \ -# && rustup default 1.78.0 +# # Install the specific version of Rust +# # RUN set -x \ +# # && rustup install 1.78.0 +# # RUN set -x \ +# # && rustup default 1.78.0 -# Add rust and cargo to PATH -ENV PATH="/usr/bin/:/root/.cargo/bin:/usr/local/bin:${PATH}" +# # Add rust and cargo to PATH +# ENV PATH="/usr/bin/:/root/.cargo/bin:/usr/local/bin:${PATH}" FROM python:3.11-slim-bullseye From 04f6e32829c684918792dc29a4b8084259cb3927 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sat, 22 Feb 2025 18:29:37 -0500 Subject: [PATCH 27/66] fix copying of binaries into the main container Signed-off-by: Adrian Edwards --- docker/backend/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 1ae4a44b87..4c92c01e85 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -119,8 +119,8 @@ RUN set -x \ RUN ./scripts/docker/install-workers-deps.sh -COPY --from=golang "$HOME/scc" "$HOME/scc" -COPY --from=golang "$HOME/scorecard" "$HOME/scorecard" +COPY --from=golang "/root/scc" "./scc" +COPY --from=golang "/root/scorecard/scorecard" "./scorecard" # RUN ./scripts/install/workers.sh From 022ac35d3fc069b96aca9f3cdb8c929e2bfc389e Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sat, 22 Feb 2025 18:30:42 -0500 Subject: [PATCH 28/66] specify versions of scc and scorecard This fixes an issue with scc where the latest version requires a golang update to build. This also prevents the same happening to scorecard in future Signed-off-by: Adrian Edwards --- scripts/docker/install-golang-deps.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/scripts/docker/install-golang-deps.sh b/scripts/docker/install-golang-deps.sh index 6661129ed8..7d255b6b2c 100755 --- a/scripts/docker/install-golang-deps.sh +++ b/scripts/docker/install-golang-deps.sh @@ -7,7 +7,9 @@ CURRENT_DIR=$PWD; # Install scc SCC_DIR="$HOME/scc" echo "Cloning Sloc Cloc and Code (SCC) to generate value data ..." -git clone https://github.com/boyter/scc "$SCC_DIR" +# this needs to be done from source. the latest version doesnt seem to exist on the package repo +# however, the latest version (v3.5.0) requires bumping the golang version in the Dockerfile +git clone --depth 1 --branch v3.4.0 https://github.com/boyter/scc "$SCC_DIR" cd $SCC_DIR go build; echo "scc build done" @@ -17,7 +19,8 @@ cd $CURRENT_DIR # Install scorecard SCORECARD_DIR="$HOME/scorecard" echo "Cloning OSSF Scorecard to generate scorecard data ..." -git clone https://github.com/ossf/scorecard $SCORECARD_DIR +# lock version to prevent future issues if the golang version is bumped +git clone --depth 1 --branch v5.1.1 https://github.com/ossf/scorecard $SCORECARD_DIR cd $SCORECARD_DIR go build; echo "scorecard build done" From ec0e1394a8a9c4e8b262bd6f180eab2998cea210 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sun, 23 Feb 2025 09:07:49 -0500 Subject: [PATCH 29/66] copy scripts into golang builder so it can find them Signed-off-by: Adrian Edwards --- docker/backend/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 4c92c01e85..bbe57b3bb7 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -6,6 +6,7 @@ FROM golang:1.23 AS golang # RUN ./scripts/docker/install-go.sh ENV PATH="${PATH}:/usr/local/go/bin" +COPY ./scripts/ scripts/ RUN ./scripts/docker/install-golang-deps.sh From 21fae403c63caf969a7d93047c7f845223f3c9be Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 25 Feb 2025 19:17:52 -0600 Subject: [PATCH 30/66] reduce dependency on create_collection_status_record tasks --- augur/application/cli/backend.py | 3 +++ augur/tasks/frontend.py | 6 +++++- augur/tasks/init/celery_app.py | 7 +++++-- augur/tasks/start_tasks.py | 4 ++-- augur/util/repo_load_controller.py | 4 +++- 5 files changed, 18 insertions(+), 6 deletions(-) diff --git a/augur/application/cli/backend.py b/augur/application/cli/backend.py index 2b5ec69042..aeddbf4634 100644 --- a/augur/application/cli/backend.py +++ b/augur/application/cli/backend.py @@ -86,6 +86,9 @@ def start(ctx, disable_collection, development, pidfile, port): worker_vmem_cap = get_value("Celery", 'worker_process_vmem_cap') + # create rabbit messages so if it failed on shutdown the queues are clean + clear_rabbitmq_messages() + gunicorn_command = f"gunicorn -c {gunicorn_location} -b {host}:{port} augur.api.server:app --log-file gunicorn.log" server = subprocess.Popen(gunicorn_command.split(" ")) diff --git a/augur/tasks/frontend.py b/augur/tasks/frontend.py index f526d9041d..cd2d3e32ba 100644 --- a/augur/tasks/frontend.py +++ b/augur/tasks/frontend.py @@ -9,7 +9,7 @@ from augur.tasks.github.util.github_graphql_data_access import GithubGraphQlDataAccess from augur.application.db.lib import get_group_by_name, get_repo_by_repo_git, get_github_repo_by_src_id, get_gitlab_repo_by_src_id from augur.tasks.github.util.util import get_owner_repo -from augur.application.db.models.augur_operations import retrieve_owner_repos, FRONTEND_REPO_GROUP_NAME, RepoGroup +from augur.application.db.models.augur_operations import retrieve_owner_repos, FRONTEND_REPO_GROUP_NAME, RepoGroup, CollectionStatus from augur.tasks.github.util.github_paginator import hit_api from augur.application.db.models import UserRepo, Repo @@ -235,6 +235,8 @@ def add_github_repo(logger, session, url, repo_group_id, group_id, repo_type, re logger.error(f"Error while adding repo: Failed to insert user repo record. A record with a repo_id of {repo_id} and a group id of {group_id} needs to be added to the user repo table so that this repo shows up in the users group") return + CollectionStatus.insert(session, logger, repo_id) + def get_gitlab_repo_data(gl_session, url: str, logger) -> bool: @@ -281,6 +283,8 @@ def add_gitlab_repo(logger, session, url, repo_group_id, group_id, repo_src_id): if not result: logger.error(f"Error while adding repo: Failed to insert user repo record. A record with a repo_id of {repo_id} and a group id of {group_id} needs to be added to the user repo table so that this repo shows up in the users group") return + + CollectionStatus.insert(session, logger, repo_id) # @celery.task # def add_org_repo_list(user_id, group_name, urls): diff --git a/augur/tasks/init/celery_app.py b/augur/tasks/init/celery_app.py index 1be45b1f00..5026244f81 100644 --- a/augur/tasks/init/celery_app.py +++ b/augur/tasks/init/celery_app.py @@ -201,8 +201,8 @@ def setup_periodic_tasks(sender, **kwargs): The tasks so that they are grouped by the module they are defined in """ from celery.schedules import crontab - from augur.tasks.start_tasks import augur_collection_monitor, augur_collection_update_weights - from augur.tasks.start_tasks import non_repo_domain_tasks, retry_errored_repos + from augur.tasks.start_tasks import augur_collection_monitor + from augur.tasks.start_tasks import non_repo_domain_tasks, retry_errored_repos, create_collection_status_records from augur.tasks.git.facade_tasks import clone_repos from augur.tasks.db.refresh_materialized_views import refresh_materialized_views from augur.tasks.data_analysis.contributor_breadth_worker.contributor_breadth_worker import contributor_breadth_model @@ -232,6 +232,9 @@ def setup_periodic_tasks(sender, **kwargs): logger.info(f"Setting 404 repos to be marked for retry on midnight each day") sender.add_periodic_task(crontab(hour=0, minute=0),retry_errored_repos.s()) + one_day_in_seconds = 24*60*60 + sender.add_periodic_task(one_day_in_seconds, create_collection_status_records.s()) + @after_setup_logger.connect def setup_loggers(*args,**kwargs): """Override Celery loggers with our own.""" diff --git a/augur/tasks/start_tasks.py b/augur/tasks/start_tasks.py index 3ba30ed707..2a697a0ea1 100644 --- a/augur/tasks/start_tasks.py +++ b/augur/tasks/start_tasks.py @@ -378,5 +378,5 @@ def create_collection_status_records(self): CollectionStatus.insert(session, logger, repo[0]) repo = execute_sql(query).first() - #Check for new repos every seven minutes to be out of step with the clone_repos task - create_collection_status_records.si().apply_async(countdown=60*7) + # no longer recursively run this task because collection status records are added when repos are inserted + #create_collection_status_records.si().apply_async(countdown=60*7) diff --git a/augur/util/repo_load_controller.py b/augur/util/repo_load_controller.py index 7021a215fb..af46ce3260 100644 --- a/augur/util/repo_load_controller.py +++ b/augur/util/repo_load_controller.py @@ -5,7 +5,7 @@ from typing import Any, Dict from augur.application.db.engine import DatabaseEngine -from augur.application.db.models import Repo, UserRepo, RepoGroup, UserGroup, User +from augur.application.db.models import Repo, UserRepo, RepoGroup, UserGroup, User, CollectionStatus from augur.application.db.models.augur_operations import retrieve_owner_repos from augur.application.db.util import execute_session_query @@ -67,8 +67,10 @@ def add_cli_repo(self, repo_data: Dict[str, Any], from_org_list=False, repo_type # if the repo doesn't exist it adds it if "gitlab" in url: repo_id = Repo.insert_gitlab_repo(self.session, url, repo_group_id, "CLI") + CollectionStatus.insert(self.session, logger, repo_id) else: repo_id = Repo.insert_github_repo(self.session, url, repo_group_id, "CLI", repo_type) + CollectionStatus.insert(self.session, logger, repo_id) if not repo_id: logger.warning(f"Invalid repo group id specified for {url}, skipping.") From 9ca06bd2103a94d88d0e1ef06d762a5f869ab9ab Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 25 Feb 2025 19:25:00 -0600 Subject: [PATCH 31/66] cleanup everything --- augur/application/cli/backend.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/augur/application/cli/backend.py b/augur/application/cli/backend.py index aeddbf4634..69f93aa6ea 100644 --- a/augur/application/cli/backend.py +++ b/augur/application/cli/backend.py @@ -87,7 +87,7 @@ def start(ctx, disable_collection, development, pidfile, port): worker_vmem_cap = get_value("Celery", 'worker_process_vmem_cap') # create rabbit messages so if it failed on shutdown the queues are clean - clear_rabbitmq_messages() + cleanup_collection_status_and_rabbit(logger, ctx.obj.engine) gunicorn_command = f"gunicorn -c {gunicorn_location} -b {host}:{port} augur.api.server:app --log-file gunicorn.log" server = subprocess.Popen(gunicorn_command.split(" ")) @@ -183,7 +183,7 @@ def start(ctx, disable_collection, development, pidfile, port): try: keypub.shutdown() - cleanup_after_collection_halt(logger, ctx.obj.engine) + cleanup_collection_status_and_rabbit(logger, ctx.obj.engine) except RedisConnectionError: pass @@ -305,7 +305,7 @@ def stop_collection(ctx): logger.info(f"Waiting on [{', '.join(str(p.pid for p in alive))}]") time.sleep(0.5) - cleanup_after_collection_halt(logger, ctx.obj.engine) + cleanup_collection_status_and_rabbit(logger, ctx.obj.engine) @cli.command('kill') @test_connection @@ -333,10 +333,10 @@ def augur_stop(signal, logger, engine): _broadcast_signal_to_processes(augur_processes, broadcast_signal=signal, given_logger=logger) if "celery" in process_names: - cleanup_after_collection_halt(logger, engine) + cleanup_collection_status_and_rabbit(logger, engine) -def cleanup_after_collection_halt(logger, engine): +def cleanup_collection_status_and_rabbit(logger, engine): clear_redis_caches() connection_string = get_value("RabbitMQ", "connection_string") From 4e8db7871444bc509756b257572ba85e5916ab21 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 26 Feb 2025 10:22:05 -0600 Subject: [PATCH 32/66] Release Bump Signed-off-by: Sean P. Goggins --- README.md | 4 ++-- augur/tasks/github/augur-notes.code-workspace | 14 ++++++++++++++ metadata.py | 4 ++-- 3 files changed, 18 insertions(+), 4 deletions(-) create mode 100644 augur/tasks/github/augur-notes.code-workspace diff --git a/README.md b/README.md index 2f4fdb67a8..8722ccca48 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.81.2 +# Augur NEW Release v0.85.0 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.81.2). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.0). - The `main` branch is a stable version of our new architecture, which features: diff --git a/augur/tasks/github/augur-notes.code-workspace b/augur/tasks/github/augur-notes.code-workspace new file mode 100644 index 0000000000..f99c46d484 --- /dev/null +++ b/augur/tasks/github/augur-notes.code-workspace @@ -0,0 +1,14 @@ +{ + "folders": [ + { + "path": "../../../../../augurlabs/augur-notes" + }, + { + "path": "../../.." + }, + { + "path": "../../../../../sociallycompute/project2025" + } + ], + "settings": {} +} \ No newline at end of file diff --git a/metadata.py b/metadata.py index 7c91b77f57..c53ee334a1 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.81.2" -__release__ = "v0.81.2 (Basketball Jones)" +__version__ = "0.85.0" +__release__ = "v0.85.0 (Basketball Jones)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From 6f5ef29bc60cc1053c8567faa09effc7decb1112 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 26 Feb 2025 10:23:46 -0600 Subject: [PATCH 33/66] Release Versions Signed-off-by: Sean P. Goggins --- docker/backend/Dockerfile | 2 +- docker/database/Dockerfile | 2 +- docker/rabbitmq/Dockerfile | 2 +- metadata.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 03dae6b01d..af68d26aa8 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -2,7 +2,7 @@ FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" -LABEL version="0.76.6" +LABEL version="0.85.0" ENV DEBIAN_FRONTEND=noninteractive ENV PATH="/usr/bin/:/usr/local/bin:/usr/lib:${PATH}" diff --git a/docker/database/Dockerfile b/docker/database/Dockerfile index c1d1098875..d7870c7223 100644 --- a/docker/database/Dockerfile +++ b/docker/database/Dockerfile @@ -2,7 +2,7 @@ FROM postgres:16 LABEL maintainer="outdoors@acm.org" -LABEL version="0.76.6" +LABEL version="0.85.0" ENV POSTGRES_DB "test" ENV POSTGRES_USER "augur" diff --git a/docker/rabbitmq/Dockerfile b/docker/rabbitmq/Dockerfile index d0ea2dea4a..71177bd98f 100644 --- a/docker/rabbitmq/Dockerfile +++ b/docker/rabbitmq/Dockerfile @@ -1,7 +1,7 @@ FROM rabbitmq:3.12-management-alpine LABEL maintainer="574/augur@simplelogin.com" -LABEL version="0.76.6" +LABEL version="0.85.0" ARG RABBIT_MQ_DEFAULT_USER=augur ARG RABBIT_MQ_DEFAULT_PASSWORD=password123 diff --git a/metadata.py b/metadata.py index c53ee334a1..4814c495a5 100644 --- a/metadata.py +++ b/metadata.py @@ -6,7 +6,7 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" __version__ = "0.85.0" -__release__ = "v0.85.0 (Basketball Jones)" +__release__ = "v0.85.0 (Eurovision)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From f5ff0f30f842c4711ee63ba445115ccf0e4b27fe Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Sat, 1 Mar 2025 14:26:36 -0600 Subject: [PATCH 34/66] only include since date in query if it exists --- augur/tasks/github/events.py | 47 ++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/augur/tasks/github/events.py b/augur/tasks/github/events.py index cf7df57582..5c79691c26 100644 --- a/augur/tasks/github/events.py +++ b/augur/tasks/github/events.py @@ -281,14 +281,23 @@ def _collect_and_process_issue_events(self, owner, repo, repo_id, key_auth, sinc with engine.connect() as connection: - # TODO: Remove src id if it ends up not being needed - query = text(f""" - select issue_id as issue_id, gh_issue_number as issue_number, gh_issue_id as gh_src_id - from issues - where repo_id={repo_id} - and updated_at > timestamptz(timestamp '{since}') - order by created_at desc; - """) + if since: + # TODO: Remove src id if it ends up not being needed + query = text(f""" + select issue_id as issue_id, gh_issue_number as issue_number, gh_issue_id as gh_src_id + from issues + where repo_id={repo_id} + and updated_at > timestamptz(timestamp '{since}') + order by created_at desc; + """) + else: + # TODO: Remove src id if it ends up not being needed + query = text(f""" + select issue_id as issue_id, gh_issue_number as issue_number, gh_issue_id as gh_src_id + from issues + where repo_id={repo_id} + order by created_at desc; + """) issue_result = connection.execute(query).fetchall() @@ -335,13 +344,21 @@ def _collect_and_process_pr_events(self, owner, repo, repo_id, key_auth, since): with engine.connect() as connection: - query = text(f""" - select pull_request_id, pr_src_number as gh_pr_number, pr_src_id - from pull_requests - where repo_id={repo_id} - and pr_updated_at > timestamptz(timestamp '{since}') - order by pr_created_at desc; - """) + if since: + query = text(f""" + select pull_request_id, pr_src_number as gh_pr_number, pr_src_id + from pull_requests + where repo_id={repo_id} + and pr_updated_at > timestamptz(timestamp '{since}') + order by pr_created_at desc; + """) + else: + query = text(f""" + select pull_request_id, pr_src_number as gh_pr_number, pr_src_id + from pull_requests + where repo_id={repo_id} + order by pr_created_at desc; + """) pr_result = connection.execute(query).fetchall() From 9da26caa826280a3f467d9ce8828fa2b5c659e34 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Sat, 1 Mar 2025 15:21:32 -0600 Subject: [PATCH 35/66] version bumpt'y dump Signed-off-by: Sean P. Goggins --- README.md | 4 ++-- docker/backend/Dockerfile | 2 +- docker/database/Dockerfile | 2 +- docker/rabbitmq/Dockerfile | 2 +- metadata.py | 4 ++-- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 8722ccca48..50db605109 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.85.0 +# Augur NEW Release v0.85.1 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.0). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.1). - The `main` branch is a stable version of our new architecture, which features: diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index af68d26aa8..367526cb18 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -2,7 +2,7 @@ FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" -LABEL version="0.85.0" +LABEL version="0.85.1" ENV DEBIAN_FRONTEND=noninteractive ENV PATH="/usr/bin/:/usr/local/bin:/usr/lib:${PATH}" diff --git a/docker/database/Dockerfile b/docker/database/Dockerfile index d7870c7223..a1a59b0c89 100644 --- a/docker/database/Dockerfile +++ b/docker/database/Dockerfile @@ -2,7 +2,7 @@ FROM postgres:16 LABEL maintainer="outdoors@acm.org" -LABEL version="0.85.0" +LABEL version="0.85.1" ENV POSTGRES_DB "test" ENV POSTGRES_USER "augur" diff --git a/docker/rabbitmq/Dockerfile b/docker/rabbitmq/Dockerfile index 71177bd98f..b4ec1948ba 100644 --- a/docker/rabbitmq/Dockerfile +++ b/docker/rabbitmq/Dockerfile @@ -1,7 +1,7 @@ FROM rabbitmq:3.12-management-alpine LABEL maintainer="574/augur@simplelogin.com" -LABEL version="0.85.0" +LABEL version="0.85.1" ARG RABBIT_MQ_DEFAULT_USER=augur ARG RABBIT_MQ_DEFAULT_PASSWORD=password123 diff --git a/metadata.py b/metadata.py index 4814c495a5..184caa4f70 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.85.0" -__release__ = "v0.85.0 (Eurovision)" +__version__ = "0.85.1" +__release__ = "v0.85.1 (Eurovision: Fire Saga)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From 6debf5f33082b5b8581a9e77d03a14275e0291e3 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 4 Mar 2025 11:11:46 -0600 Subject: [PATCH 36/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index c376908c50..aa927897c8 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -147,15 +147,7 @@ The following is an example from the [openEuler](https://www.openeuler.org/en/) - https://chaoss.github.io/grimoirelab-sigils/common/onion_analysis/ - https://mikemcquaid.com/2018/08/14/the-open-source-contributor-funnel-why-people-dont-contribute-to-your-open-source-project/ ### Contributors - - Yehui Wang - - Clement Li - - zhongjun - - Xiaoya Xia - - Matt Germonprez - Sean Goggins - - King Gao - - ## IDEA: Open Source Software Health Metrics Visualization Exploration From e68adcc88b5a82e4810ad0a5495188c6221077b3 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 4 Mar 2025 11:12:11 -0600 Subject: [PATCH 37/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 1 - 1 file changed, 1 deletion(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index aa927897c8..85a86ea881 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -115,7 +115,6 @@ Source: https://chaoss.github.io/grimoirelab-sigils/assets/images/screenshots/si Source: https://opensource.com/sites/default/files/uploads/2021-09-15-developer-level-02.png #### Tools Providing the Metric - - GrimoireLab - Augur - openEuler Infra From b2707b9575c87f31bbedbab38e28a7ccc83da2c1 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 4 Mar 2025 11:12:52 -0600 Subject: [PATCH 38/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index 85a86ea881..57cc2d582e 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -88,7 +88,7 @@ The aims of the project are as follows: * _Difficulty:_ Medium * _Requirements:_ Knowledge of Python is desired. Some knowledge of Javascript or twitter/bootstrap is also desired. Key requirement is a keenness to dig into this challenge! * _Recommended:_ Python experience. -* _Mentors:_ Sean Goggins, Daniel Izquerdo +* _Mentors:_ Sean Goggins #### Filters (optional) - Commits From 6c039e3c4f2acf45cc50e38ba88ae0aed679913a Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 4 Mar 2025 11:13:49 -0600 Subject: [PATCH 39/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index 57cc2d582e..cea5dff355 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -146,7 +146,9 @@ The following is an example from the [openEuler](https://www.openeuler.org/en/) - https://chaoss.github.io/grimoirelab-sigils/common/onion_analysis/ - https://mikemcquaid.com/2018/08/14/the-open-source-contributor-funnel-why-people-dont-contribute-to-your-open-source-project/ ### Contributors - - Sean Goggins + - Sean Goggins + - Andrew Brain + - John McGinness ## IDEA: Open Source Software Health Metrics Visualization Exploration From 9e9380490b85ec3c096e74e36a6ab37ce5b331d4 Mon Sep 17 00:00:00 2001 From: Sean Goggins Date: Thu, 6 Mar 2025 11:16:02 -0600 Subject: [PATCH 40/66] test Signed-off-by: Sean Goggins --- test.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test.md b/test.md index 9daeafb986..753e9975dd 100644 --- a/test.md +++ b/test.md @@ -1 +1 @@ -test +teh test From 247b3bfd0a643f85ea4c1d76c9bcafa9437aa159 Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Sat, 8 Mar 2025 14:08:51 -0600 Subject: [PATCH 41/66] fix pr events uniqueness --- augur/application/db/data_parse.py | 2 +- .../versions/31_update_pr_events_unique.py | 38 +++++++++++++++++++ augur/tasks/github/events.py | 2 +- 3 files changed, 40 insertions(+), 2 deletions(-) create mode 100644 augur/application/schema/alembic/versions/31_update_pr_events_unique.py diff --git a/augur/application/db/data_parse.py b/augur/application/db/data_parse.py index 71bde0aa21..6f20fa35a7 100644 --- a/augur/application/db/data_parse.py +++ b/augur/application/db/data_parse.py @@ -291,7 +291,7 @@ def extract_pr_event_data(event: dict, pr_id: int, gh_src_id: int, platform_id: 'action': event['event'], 'action_commit_hash': None, 'created_at': event['created_at'], - 'issue_event_src_id': gh_src_id, + 'issue_event_src_id': event["id"], 'node_id': event['node_id'], 'node_url': event['url'], 'tool_source': tool_source, diff --git a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py new file mode 100644 index 0000000000..a148a04ef3 --- /dev/null +++ b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py @@ -0,0 +1,38 @@ +"""Update pr events unique + +Revision ID: 31 +Revises: 30 +Create Date: 2025-03-08 + +""" +from alembic import op +import sqlalchemy as sa +from sqlalchemy import text + + +# revision identifiers, used by Alembic. +revision = '30' +down_revision = '29' +branch_labels = None +depends_on = None + + +def upgrade(): + conn = op.get_bind() + conn.execute(text(""" + UPDATE pull_request_events + SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT; + """)) + + op.create_unique_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', ['repo_id', 'issue_event_src_id'], schema='augur_data') + + +def downgrade(): + op.drop_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', schema='augur_data', type_='unique') + + conn = op.get_bind() + conn.execute(text(""" + UPDATE pull_request_events + SET issue_event_src_id = pr_platform_event_id; + """)) + diff --git a/augur/tasks/github/events.py b/augur/tasks/github/events.py index 5c79691c26..654dc68c6a 100644 --- a/augur/tasks/github/events.py +++ b/augur/tasks/github/events.py @@ -78,7 +78,7 @@ def _insert_issue_events(self, events): bulk_insert_dicts(self._logger, events, IssueEvent, issue_event_natural_keys) def _insert_pr_events(self, events): - pr_event_natural_keys = ["node_id"] + pr_event_natural_keys = ["repo_id", "issue_event_src_id"] bulk_insert_dicts(self._logger, events, PullRequestEvent, pr_event_natural_keys) def _insert_contributors(self, contributors): From 4ed1a26ea760acfe56c77a50456a20d10e148a23 Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Sat, 8 Mar 2025 14:17:41 -0600 Subject: [PATCH 42/66] update revision number --- .../schema/alembic/versions/31_update_pr_events_unique.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py index a148a04ef3..dbf4f9c599 100644 --- a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py +++ b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py @@ -11,8 +11,8 @@ # revision identifiers, used by Alembic. -revision = '30' -down_revision = '29' +revision = '31' +down_revision = '30' branch_labels = None depends_on = None From cff1bdc7f3e81486ece5863d45450d8c2c3c9892 Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 11 Mar 2025 19:41:57 -0500 Subject: [PATCH 43/66] incrementally update database --- .../versions/31_update_pr_events_unique.py | 62 ++++++++++++++++--- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py index dbf4f9c599..c04754a2bd 100644 --- a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py +++ b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py @@ -8,6 +8,7 @@ from alembic import op import sqlalchemy as sa from sqlalchemy import text +from augur.application.db import create_database_engine, get_database_string # revision identifiers, used by Alembic. @@ -17,22 +18,65 @@ depends_on = None + # conn = op.get_bind() + # conn.execute(text(""" + # UPDATE pull_request_events + # SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT; + # """)) + + def upgrade(): - conn = op.get_bind() - conn.execute(text(""" - UPDATE pull_request_events - SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT; - """)) + + connection_string = get_database_string() + engine = create_database_engine(connection_string) + + with engine.connect() as conn: + + result = conn.execute(text("SELECT COUNT(*) FROM pull_request_events WHERE issue_event_src_id=pr_platform_event_id")) + total_rows = result.scalar() + print(f"Rows needing updated: {total_rows}") + print(f"0.0% complete") + total_updated = 0 + + while True: + result = conn.execute(text(""" + WITH cte AS ( + SELECT pr_event_id + FROM pull_request_events + WHERE issue_event_src_id=pr_platform_event_id + LIMIT 250000 + ) + UPDATE pull_request_events + SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT + FROM cte + WHERE pull_request_events.pr_event_id = cte.pr_event_id + RETURNING 1; + """)) + + conn.commit() + + rows_updated = result.rowcount + total_updated += rows_updated + + if rows_updated == 0: + print(f"Update complete") + break + + percentage_updated = (total_updated / total_rows) * 100 + + print(f"{percentage_updated:.1f}% complete ({total_rows-total_updated} rows left)") + op.create_unique_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', ['repo_id', 'issue_event_src_id'], schema='augur_data') - + def downgrade(): op.drop_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', schema='augur_data', type_='unique') + print("Please run in background. This downgrade will take a very *very* long time") conn = op.get_bind() conn.execute(text(""" UPDATE pull_request_events - SET issue_event_src_id = pr_platform_event_id; - """)) - + SET issue_event_src_id = pr_platform_event_id + WHERE issue_event_src_id <> pr_platform_event_id; + """)) \ No newline at end of file From 025192b2e4262b215763e48c723183c396d0057e Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 11 Mar 2025 19:44:52 -0500 Subject: [PATCH 44/66] add index log --- .../schema/alembic/versions/31_update_pr_events_unique.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py index c04754a2bd..5e3c60167c 100644 --- a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py +++ b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py @@ -66,7 +66,7 @@ def upgrade(): print(f"{percentage_updated:.1f}% complete ({total_rows-total_updated} rows left)") - + print("Creating (repo_id, issue_event_src_id) index") op.create_unique_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', ['repo_id', 'issue_event_src_id'], schema='augur_data') From e00d1726485857e92ea6858278660ce4f1fffe47 Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 11 Mar 2025 19:51:51 -0500 Subject: [PATCH 45/66] only print if there are rows that need updated --- .../versions/31_update_pr_events_unique.py | 67 ++++++++++--------- 1 file changed, 34 insertions(+), 33 deletions(-) diff --git a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py index 5e3c60167c..b55b60a09a 100644 --- a/augur/application/schema/alembic/versions/31_update_pr_events_unique.py +++ b/augur/application/schema/alembic/versions/31_update_pr_events_unique.py @@ -34,39 +34,40 @@ def upgrade(): result = conn.execute(text("SELECT COUNT(*) FROM pull_request_events WHERE issue_event_src_id=pr_platform_event_id")) total_rows = result.scalar() - print(f"Rows needing updated: {total_rows}") - print(f"0.0% complete") - total_updated = 0 - - while True: - result = conn.execute(text(""" - WITH cte AS ( - SELECT pr_event_id - FROM pull_request_events - WHERE issue_event_src_id=pr_platform_event_id - LIMIT 250000 - ) - UPDATE pull_request_events - SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT - FROM cte - WHERE pull_request_events.pr_event_id = cte.pr_event_id - RETURNING 1; - """)) - - conn.commit() - - rows_updated = result.rowcount - total_updated += rows_updated - - if rows_updated == 0: - print(f"Update complete") - break - - percentage_updated = (total_updated / total_rows) * 100 - - print(f"{percentage_updated:.1f}% complete ({total_rows-total_updated} rows left)") - - print("Creating (repo_id, issue_event_src_id) index") + if total_rows != 0: + print(f"Rows needing updated: {total_rows}") + print(f"0.0% complete") + total_updated = 0 + + while True: + result = conn.execute(text(""" + WITH cte AS ( + SELECT pr_event_id + FROM pull_request_events + WHERE issue_event_src_id=pr_platform_event_id + LIMIT 250000 + ) + UPDATE pull_request_events + SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT + FROM cte + WHERE pull_request_events.pr_event_id = cte.pr_event_id + RETURNING 1; + """)) + + conn.commit() + + rows_updated = result.rowcount + total_updated += rows_updated + + if rows_updated == 0: + print(f"Update complete") + break + + percentage_updated = (total_updated / total_rows) * 100 + + print(f"{percentage_updated:.1f}% complete ({total_rows-total_updated} rows left)") + + print("Creating (repo_id, issue_event_src_id) index") op.create_unique_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', ['repo_id', 'issue_event_src_id'], schema='augur_data') From 53748c4f70643defeabdd466d952a4ecb1b05d7e Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Mar 2025 10:10:20 -0500 Subject: [PATCH 46/66] Update README.md Signed-off-by: Sean P. Goggins --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 50db605109..26b1db7dc9 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.85.1 +# Augur NEW Release v0.85.2 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.1). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.2). - The `main` branch is a stable version of our new architecture, which features: From e67cf0cd382ccbc18fc5e2c37fe8ac41cf04ca05 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 12 Mar 2025 10:10:43 -0500 Subject: [PATCH 47/66] Update metadata.py Signed-off-by: Sean P. Goggins --- metadata.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/metadata.py b/metadata.py index 184caa4f70..4933b258b2 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.85.1" -__release__ = "v0.85.1 (Eurovision: Fire Saga)" +__version__ = "0.85.2" +__release__ = "v0.85.2 (Eurovision: Fire Saga)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From 94ebe3e425a9f932ebfe0b12b4baaddb3269d60a Mon Sep 17 00:00:00 2001 From: Somesh Ghosh Date: Sun, 16 Mar 2025 16:51:59 +0530 Subject: [PATCH 48/66] Corrected GitLab Public Access Token URL Signed-off-by: Somesh Ghosh --- docs/dev-osx-install.md | 2 +- docs/new-install-ubuntu-python-3.10.md | 2 +- docs/new-install.md | 2 +- docs/source/getting-started/dev-osx-install.rst | 3 +-- docs/source/getting-started/new-install.rst | 3 +-- 5 files changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/dev-osx-install.md b/docs/dev-osx-install.md index c70bac7714..aa5b67b7fa 100644 --- a/docs/dev-osx-install.md +++ b/docs/dev-osx-install.md @@ -31,7 +31,7 @@ export PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig" ## Git Platform Requirements (Things to have setup prior to initiating installation.) 1. Obtain a GitHub Access Token: https://github.com/settings/tokens -2. Obtain a GitLab Access Token: https://gitlab.com/-/profile/personal_access_tokens +2. Obtain a GitLab Access Token: https://gitlab.com/-/user_settings/personal_access_tokens ### Fork and Clone Augur 1. Fork https://github.com/chaoss/augur diff --git a/docs/new-install-ubuntu-python-3.10.md b/docs/new-install-ubuntu-python-3.10.md index 3cd7a5a38a..6fa2ee78fc 100644 --- a/docs/new-install-ubuntu-python-3.10.md +++ b/docs/new-install-ubuntu-python-3.10.md @@ -5,7 +5,7 @@ We default to this version of Ubuntu for the moment because Augur does not yet s ## Git Platform Requirements (Things to have setup prior to initiating installation.) 1. Obtain a GitHub Access Token: https://github.com/settings/tokens -2. Obtain a GitLab Access Token: https://gitlab.com/-/profile/personal_access_tokens +2. Obtain a GitLab Access Token: https://gitlab.com/-/user_settings/personal_access_tokens ### Fork and Clone Augur 1. Fork https://github.com/chaoss/augur diff --git a/docs/new-install.md b/docs/new-install.md index 9ec8960857..017cb41322 100644 --- a/docs/new-install.md +++ b/docs/new-install.md @@ -5,7 +5,7 @@ We default to this version of Ubuntu for the moment because Augur does not yet s ## Git Platform Requirements (Things to have setup prior to initiating installation.) 1. Obtain a GitHub Access Token: https://github.com/settings/tokens -2. Obtain a GitLab Access Token: https://gitlab.com/-/profile/personal_access_tokens +2. Obtain a GitLab Access Token: https://gitlab.com/-/user_settings/personal_access_tokens ### Fork and Clone Augur 1. Fork https://github.com/chaoss/augur diff --git a/docs/source/getting-started/dev-osx-install.rst b/docs/source/getting-started/dev-osx-install.rst index 05285da207..05d16c29be 100644 --- a/docs/source/getting-started/dev-osx-install.rst +++ b/docs/source/getting-started/dev-osx-install.rst @@ -47,8 +47,7 @@ Git Platform Requirements (Things to have setup prior to initiating installation ---------------------------------------------------------------------------------- 1. Obtain a GitHub Access Token: https://github.com/settings/tokens -2. Obtain a GitLab Access Token: - https://gitlab.com/-/profile/personal_access_tokens +2. Obtain a GitLab Access Token: https://gitlab.com/-/user_settings/personal_access_tokens Fork and Clone Augur ~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/getting-started/new-install.rst b/docs/source/getting-started/new-install.rst index 3fd45f4a97..b10aa0b41e 100644 --- a/docs/source/getting-started/new-install.rst +++ b/docs/source/getting-started/new-install.rst @@ -9,8 +9,7 @@ Git Platform Requirements (Things to have setup prior to initiating installation ---------------------------------------------------------------------------------- 1. Obtain a GitHub Access Token: https://github.com/settings/tokens -2. Obtain a GitLab Access Token: - https://gitlab.com/-/profile/personal_access_tokens +2. Obtain a GitLab Access Token: https://gitlab.com/-/user_settings/personal_access_tokens Fork and Clone Augur ~~~~~~~~~~~~~~~~~~~~ From 1b4b6cdf309653fb940e0713e7b327a5fcdcca40 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sun, 16 Mar 2025 15:41:51 -0400 Subject: [PATCH 49/66] remove podman compose it seems to be broken and not kept up to date with the docker compose file Signed-off-by: Adrian Edwards --- podman-compose.yml | 68 ---------------------------------------------- 1 file changed, 68 deletions(-) delete mode 100644 podman-compose.yml diff --git a/podman-compose.yml b/podman-compose.yml deleted file mode 100644 index 9970f81325..0000000000 --- a/podman-compose.yml +++ /dev/null @@ -1,68 +0,0 @@ -#SPDX-License-Identifier: MIT -version: '3' -services: - augur-db: - image: postgres:14 - restart: unless-stopped - environment: - - "POSTGRES_DB=augur" - - "POSTGRES_USER=${AUGUR_DB_USER:-augur}" - - "POSTGRES_PASSWORD=${AUGUR_DB_PASSWORD:-augur}" - - "PGDATA=/var/lib/postgresql/data/pgdata" - ports: - - "${AUGUR_DB_PORT:-5432}:5432" - volumes: - - augurpostgres:/var/lib/postgresql/data - - redis: - image: "redis:alpine" - ports: - - 6379:6379 - - rabbitmq: - image: augur-rabbitmq - build: - context: . - dockerfile: ./docker/rabbitmq/Dockerfile - args: - - RABBIT_MQ_DEFAULT_USER=${AUGUR_RABBITMQ_USERNAME:-augur} - - RABBIT_MQ_DEFAULT_PASSWORD=${AUGUR_RABBITMQ_PASSWORD:-password123} - - RABBIT_MQ_DEFAULT_VHOST=${AUGUR_RABBITMQ_VHOST:-augur_vhost} - # ports for amqp connections / management api - ports: - - 5671:5671 - - 5672:5672 - - 15671:15671 - - 15672:15672 - - augur: - image: augur-new:latest - build: - context: . - dockerfile: ./docker/backend/Dockerfile - volumes: - - facade:/augur/facade - restart: unless-stopped - ports: - - 5002:5000 - environment: - - "AUGUR_DB=postgresql+psycopg2://${AUGUR_DB_USER:-augur}:${AUGUR_DB_PASSWORD:-augur}@augur-db:5432/augur" - - "AUGUR_DB_SCHEMA_BUILD=1" - - "AUGUR_GITHUB_API_KEY=${AUGUR_GITHUB_API_KEY}" - - "AUGUR_GITLAB_API_KEY=${AUGUR_GITLAB_API_KEY}" - - "AUGUR_GITHUB_USERNAME=${AUGUR_GITHUB_USERNAME}" - - "AUGUR_GITLAB_USERNAME=${AUGUR_GITLAB_USERNAME}" - - REDIS_CONN_STRING=redis://redis:6379 - - RABBITMQ_CONN_STRING=amqp://${AUGUR_RABBITMQ_USERNAME:-augur}:${AUGUR_RABBITMQ_PASSWORD:-password123}@rabbitmq:5672/${AUGUR_RABBITMQ_VHOST:-augur_vhost} - depends_on: - - augur-db - - redis - - rabbitmq - -volumes: - facade: - driver: local - augurpostgres: - driver: local - - From 2b5981c6620c130e1b313911c52033886d71a868 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Sun, 16 Mar 2025 19:13:14 -0400 Subject: [PATCH 50/66] Delete Vagrantfile Signed-off-by: Adrian Edwards --- Vagrantfile | 108 ---------------------------------------------------- 1 file changed, 108 deletions(-) delete mode 100644 Vagrantfile diff --git a/Vagrantfile b/Vagrantfile deleted file mode 100644 index 95de27ce02..0000000000 --- a/Vagrantfile +++ /dev/null @@ -1,108 +0,0 @@ -$script = <<-'SCRIPT' -set -euxo pipefail - -sudo apt-get -y update -sudo apt-get -y install --no-install-recommends \ - build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev libffi-dev curl libbz2-dev \ - git gcc gfortran \ - python3 python3-pip python3.8-venv \ - postgresql postgresql-contrib \ - libomp-dev \ - golang libgomp1 -sudo pg_ctlcluster 12 main start - -go get -u github.com/boyter/scc/ - -# # install Go -# installGo() ( -# cd "$(mktemp -d)" -# wget https://golang.org/dl/go1.16.5.linux-amd64.tar.gz -# rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.5.linux-amd64.tar.gz -# ) -# sudo installGo -# export PATH=$PATH:/usr/local/go/bin - - -########################################################################################## -# see: https://oss-augur.readthedocs.io/en/master/getting-started/database.html -cat < /tmp/init.psql -CREATE DATABASE augur; -CREATE USER augur WITH ENCRYPTED PASSWORD 'password'; -GRANT ALL PRIVILEGES ON DATABASE augur TO augur; -EOF -sudo -u postgres psql -U postgres -f /tmp/init.psql - - -########################################################################################## -# see: https://oss-augur.readthedocs.io/en/master/getting-started/installation.html - -mkdir -p "$HOME/augur/" "$HOME/augur/logs/" "$HOME/augur/repos/" -cat < "$HOME/augur/config.json" -{ - "Database": { - "host": "localhost", - "password": "password" - }, - "Server": { - "host": "0.0.0.0" - }, - "Logging": { - "logs_directory": "$HOME/augur/logs/", - "log_level": "INFO", - "verbose": 0, - "quiet": 0, - "debug": 1 - }, - "Workers": { - "facade_worker": { - "repo_directory": "$HOME/augur/repos/", - "switch": 1 - }, - "github_worker": { - "switch": 1 - }, - "insight_worker": { - "switch": 1 - }, - "linux_badge_worker": { - "switch": 1 - }, - "pull_request_worker": { - "switch": 1 - }, - "repo_info_worker": { - "switch": 1 - }, - "release_worker": { - "switch": 1 - } - } -} -EOF - - -python3 -m venv $HOME/.virtualenvs/augur_env -source $HOME/.virtualenvs/augur_env/bin/activate -pip install wheel - -cd /vagrant -python setup.py bdist_wheel -make clean -make install-dev - -augur config init --rc-config-file "$HOME/config.json" -augur db create-schema -augur backend start" - -SCRIPT - -Vagrant.configure("2") do |config| - config.vm.box = "ubuntu/focal64" - - config.vm.provider "virtualbox" do |v| - v.memory = 20480 - v.cpus = 4 - end - - config.vm.provision "shell", privileged: false, inline: $script -end From 3ae2c200faee49530965491c55d1524c837dd131 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Mon, 17 Mar 2025 09:38:09 -0500 Subject: [PATCH 51/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index cea5dff355..af9eb2f7ae 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -5,6 +5,34 @@ [Micro-tasks and place for questions](https://github.com/chaoss/augur/issues/1640) +## Microtasks + +For becoming familiar with Augur, you can start by reading some documentation. You can find useful information at in the links, below. Grimoirelab also has a set of installation instructions and documentation here: https://chaoss.github.io/grimoirelab-tutorial/ + +#### GSoC Students (Primarily, though contributions welcome from Outreachy): +Once you're familiar with Augur, you can have a look at the following microtasks. + +Microtask 0: + Download and configure Augur, creating a dev environment using the general cautions noted here: +Augur + https://oss-augur.readthedocs.io/en/dev/getting-started/installation.html and the full documentation here: + https://oss-augur.readthedocs.io/en/dev/development-guide/toc.html +Grimoirelab + https://chaoss.github.io/grimoirelab-tutorial/ + +Microstask 1: + Work on any Augur or Grimoirelab Issue that's Open + +Microtask 2: + Identify new issues you encounter during installation. + +Microstask 3: + Explore data presently captured, develop an experimental visualization using tools of your choice. If Jupyter Notebooks against an Augur database/API endpoint collection, use https://github.com/chaoss/augur-community-reports for development. + +Microtask 4: + Anything you want to show us. Even if you find bugs in our documentation and want to issue a PR for those! + + This project will add GenSIM logic, and other capabilities to the Clustering Worker inside of Augur Software, and be extended into a generalized Open Source Software Conversational Topic Modeling Instrument. CHOASS/augur has several workers that store machine learning information derived from computational linguistic analysis of data in the `message` table. The message table includes messages from issue, pull request, pull request review, and email messages. They are related to their origin with bridge tables like `pull_request_message_ref`. The ML/CL workers are all run against all the messages, regardless of origin. From 7deb0f10a500f0f3139549e2b5b8b9a5e873665f Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Mon, 17 Mar 2025 09:38:27 -0500 Subject: [PATCH 52/66] Update gsoc-ideas.md Signed-off-by: Sean P. Goggins --- gsoc-ideas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gsoc-ideas.md b/gsoc-ideas.md index af9eb2f7ae..3bc2748811 100644 --- a/gsoc-ideas.md +++ b/gsoc-ideas.md @@ -9,7 +9,7 @@ For becoming familiar with Augur, you can start by reading some documentation. You can find useful information at in the links, below. Grimoirelab also has a set of installation instructions and documentation here: https://chaoss.github.io/grimoirelab-tutorial/ -#### GSoC Students (Primarily, though contributions welcome from Outreachy): +#### GSoC Students : Once you're familiar with Augur, you can have a look at the following microtasks. Microtask 0: From d8d40a21616a3b4b155f46a3c783f7990e3f284b Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Tue, 18 Mar 2025 09:36:08 -0400 Subject: [PATCH 53/66] add flower section to the docker compose Signed-off-by: Adrian Edwards --- docker-compose.yml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docker-compose.yml b/docker-compose.yml index e077260ed7..bdd1127931 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -62,6 +62,26 @@ services: - redis - rabbitmq + # Flower is a UI that helps more easily monitor running tasks for celery workers. + # This wont show you every error from the entire augur system, but it should show most of them. + # uncomment the section below to use flower + # flower: + # image: augur-new:latest + # restart: unless-stopped + # command: + # [ "celery", "-A", "augur.tasks.init.celery_app.celery_app", "flower", "--max-tasks=1000000" ] + # ports: + # - 5555:5555 + # environment: + # - "AUGUR_DB=postgresql+psycopg2://${AUGUR_DB_USER:-augur}:${AUGUR_DB_PASSWORD:-augur}@augur-db:5432/augur" + # - REDIS_CONN_STRING=redis://redis:6379 + # - RABBITMQ_CONN_STRING=amqp://${AUGUR_RABBITMQ_USERNAME:-augur}:${AUGUR_RABBITMQ_PASSWORD:-password123}@rabbitmq:5672/${AUGUR_RABBITMQ_VHOST:-augur_vhost} + # depends_on: + # - augur + # - augur-db + # - redis + # - rabbitmq + volumes: facade: driver: local From a96b838ca24775dce0bb15212736a21d1323cac3 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 18 Mar 2025 09:07:36 -0500 Subject: [PATCH 54/66] Update CONTRIBUTING.md Signed-off-by: Sean P. Goggins --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6368d58bf5..6fed03731b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -92,7 +92,7 @@ git push origin master ### Augur - [Stable documentation (`main` branch)](https://oss-augur.readthedocs.io/en/main/) - [Nightly/developer build documentation (`dev` branch)](https://oss-augur.readthedocs.io/en/dev/) (warning: this is should be considered an unstable branch and should not be used for production) -- [Live Augur demo](http://zephyr.osshealth.io/) +- [Live Augur demo](https://ai.chaoss.io) ### CHAOSS - [Website](https://chaoss.community/) From b47510560943a4642f97e7305e4b3e5374f66dfb Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 18 Mar 2025 09:17:00 -0500 Subject: [PATCH 55/66] fixing install-dev command by removal Signed-off-by: Sean P. Goggins --- docs/source/getting-started/installation.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst index 5c2e2f62f7..4e367958c1 100644 --- a/docs/source/getting-started/installation.rst +++ b/docs/source/getting-started/installation.rst @@ -216,11 +216,6 @@ your installation of Python 3: on most systems, this is ``python3``, but yours m # run the install script $ make install -.. code-block:: bash - - # If you want to develop with Augur, use this command instead - $ make install-dev - If you think something went wrong, check the log files in ``logs/``. If you want to try again, you can use ``make clean`` to delete any build files before running ``make install`` again. MacOS users: From 64cc5e1b562ce0c03d0dd51f109aacd6ade135eb Mon Sep 17 00:00:00 2001 From: Isaac Milarsky Date: Tue, 18 Mar 2025 18:57:10 -0500 Subject: [PATCH 56/66] log stderr in called process for facade commit count Signed-off-by: Isaac Milarsky --- .../facade_worker/facade_worker/analyzecommit.py | 1 + .../facade_worker/facade_worker/utilitymethods.py | 12 ++++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/augur/tasks/git/util/facade_worker/facade_worker/analyzecommit.py b/augur/tasks/git/util/facade_worker/facade_worker/analyzecommit.py index 1e2e04ac12..5f8bcd5772 100644 --- a/augur/tasks/git/util/facade_worker/facade_worker/analyzecommit.py +++ b/augur/tasks/git/util/facade_worker/facade_worker/analyzecommit.py @@ -32,6 +32,7 @@ import sqlalchemy as s from augur.application.db.lib import execute_sql, fetchall_data_from_sql_text +from augur.tasks.init import get_rabbitmq_conn_string def analyze_commit(logger, repo_id, repo_loc, commit): diff --git a/augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py b/augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py index 40f3a29e0e..caae6c02ba 100644 --- a/augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py +++ b/augur/tasks/git/util/facade_worker/facade_worker/utilitymethods.py @@ -26,7 +26,7 @@ # repos. It also rebuilds analysis data, checks any changed affiliations and # aliases, and caches data for display. import subprocess -from subprocess import check_output +from subprocess import check_output, CalledProcessError import os import sqlalchemy as s from augur.application.db.models import * @@ -154,7 +154,15 @@ def get_repo_commit_count(logger, facade_helper, repo_git): if count_branches(repo_loc) == 0: return 0 - check_commit_count_cmd = check_output(["git", "--git-dir", repo_loc, "rev-list", "--count", "HEAD"]) + try: + check_commit_count_cmd = check_output( + ["git", "--git-dir", repo_loc, "rev-list", "--count", "HEAD"], + stderr=subprocess.PIPE) + except CalledProcessError as e: + logger.error(f"Ran into {e}: {e.output} {e.stderr} \n With return code {e.returncode}") + raise e + + commit_count = int(check_commit_count_cmd) return commit_count From f4dad21846326dea7c552482d162e898ebb19a36 Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 18 Mar 2025 19:38:24 -0500 Subject: [PATCH 57/66] stop using keys once they have 100 requests remaining --- augur/tasks/github/util/github_data_access.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/augur/tasks/github/util/github_data_access.py b/augur/tasks/github/util/github_data_access.py index a648f990c6..648301778b 100644 --- a/augur/tasks/github/util/github_data_access.py +++ b/augur/tasks/github/util/github_data_access.py @@ -5,6 +5,8 @@ from urllib.parse import urlparse, parse_qs, urlencode from keyman.KeyClient import KeyClient +GITHUB_RATELIMIT_REMAINING_CAP = 100 + class RatelimitException(Exception): @@ -110,6 +112,13 @@ def make_request(self, url, method="GET", timeout=100): response.raise_for_status() + try: + if "X-RateLimit-Remaining" in response.headers and int(response.headers["X-RateLimit-Remaining"]) < GITHUB_RATELIMIT_REMAINING_CAP: + raise RatelimitException(response) + except ValueError: + self.logger.warning(f"X-RateLimit-Remaining was not an integer. Value: {response.headers["X-RateLimit-Remaining"]}") + + return response def make_request_with_retries(self, url, method="GET", timeout=100): @@ -148,7 +157,7 @@ def __handle_github_ratelimit_response(self, response): f'\n\n\n\nSleeping for {retry_after} seconds due to secondary rate limit issue.\n\n\n\n') time.sleep(retry_after) - elif "X-RateLimit-Remaining" in headers and int(headers["X-RateLimit-Remaining"]) == 0: + elif "X-RateLimit-Remaining" in headers and int(headers["X-RateLimit-Remaining"]) < GITHUB_RATELIMIT_REMAINING_CAP: current_epoch = int(time.time()) epoch_when_key_resets = int(headers["X-RateLimit-Reset"]) key_reset_time = epoch_when_key_resets - current_epoch From 192f87d178c5b73fbd9dc4ae0c478bde3545813e Mon Sep 17 00:00:00 2001 From: Andrew Brain Date: Tue, 18 Mar 2025 19:40:28 -0500 Subject: [PATCH 58/66] set to 50 --- augur/tasks/github/util/github_data_access.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/augur/tasks/github/util/github_data_access.py b/augur/tasks/github/util/github_data_access.py index 648301778b..9610e2a839 100644 --- a/augur/tasks/github/util/github_data_access.py +++ b/augur/tasks/github/util/github_data_access.py @@ -5,7 +5,7 @@ from urllib.parse import urlparse, parse_qs, urlencode from keyman.KeyClient import KeyClient -GITHUB_RATELIMIT_REMAINING_CAP = 100 +GITHUB_RATELIMIT_REMAINING_CAP = 50 class RatelimitException(Exception): From 73a4c6ad11942bf757697dd0c9e8c3de9da30df2 Mon Sep 17 00:00:00 2001 From: Adrian Edwards Date: Thu, 13 Mar 2025 20:24:58 -0400 Subject: [PATCH 59/66] adjust path that scc gets copied to Signed-off-by: Adrian Edwards --- docker/backend/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index df6e952b05..78b2666b77 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -120,7 +120,7 @@ RUN set -x \ RUN ./scripts/docker/install-workers-deps.sh -COPY --from=golang "/root/scc" "./scc" +COPY --from=golang "/root/scc" "/root/scc" COPY --from=golang "/root/scorecard/scorecard" "./scorecard" # RUN ./scripts/install/workers.sh From a414ab1e1737093097269af94217a2f84b8740a4 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Wed, 19 Mar 2025 12:20:50 -0500 Subject: [PATCH 60/66] version update Signed-off-by: Sean P. Goggins --- README.md | 4 ++-- docker/backend/Dockerfile | 2 +- docker/database/Dockerfile | 2 +- docker/rabbitmq/Dockerfile | 2 +- metadata.py | 4 ++-- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 26b1db7dc9..a50d94037e 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.85.2 +# Augur NEW Release v0.85.3 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.2). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.3). - The `main` branch is a stable version of our new architecture, which features: diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 78b2666b77..6142ef71c4 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -32,7 +32,7 @@ RUN ./scripts/docker/install-golang-deps.sh FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" -LABEL version="0.85.1" +LABEL version="0.85.3" ENV DEBIAN_FRONTEND=noninteractive ENV PATH="/usr/bin/:/usr/local/bin:/usr/lib:${PATH}" diff --git a/docker/database/Dockerfile b/docker/database/Dockerfile index a1a59b0c89..aa7dee1603 100644 --- a/docker/database/Dockerfile +++ b/docker/database/Dockerfile @@ -2,7 +2,7 @@ FROM postgres:16 LABEL maintainer="outdoors@acm.org" -LABEL version="0.85.1" +LABEL version="0.85.3" ENV POSTGRES_DB "test" ENV POSTGRES_USER "augur" diff --git a/docker/rabbitmq/Dockerfile b/docker/rabbitmq/Dockerfile index b4ec1948ba..d7c47ebbaa 100644 --- a/docker/rabbitmq/Dockerfile +++ b/docker/rabbitmq/Dockerfile @@ -1,7 +1,7 @@ FROM rabbitmq:3.12-management-alpine LABEL maintainer="574/augur@simplelogin.com" -LABEL version="0.85.1" +LABEL version="0.85.3" ARG RABBIT_MQ_DEFAULT_USER=augur ARG RABBIT_MQ_DEFAULT_PASSWORD=password123 diff --git a/metadata.py b/metadata.py index 4933b258b2..a4136cac86 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.85.2" -__release__ = "v0.85.2 (Eurovision: Fire Saga)" +__version__ = "0.85.3" +__release__ = "v0.85.3 (Eurovision: Fire Saga)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From a71fde963e5d779b6ae1afc66f9fd509f29caa2b Mon Sep 17 00:00:00 2001 From: Marcel Beyer Date: Fri, 21 Mar 2025 11:50:10 +0100 Subject: [PATCH 61/66] fix syntax error Signed-off-by: Marcel Beyer --- augur/tasks/github/util/github_data_access.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/augur/tasks/github/util/github_data_access.py b/augur/tasks/github/util/github_data_access.py index 9610e2a839..0eda1c8462 100644 --- a/augur/tasks/github/util/github_data_access.py +++ b/augur/tasks/github/util/github_data_access.py @@ -116,7 +116,7 @@ def make_request(self, url, method="GET", timeout=100): if "X-RateLimit-Remaining" in response.headers and int(response.headers["X-RateLimit-Remaining"]) < GITHUB_RATELIMIT_REMAINING_CAP: raise RatelimitException(response) except ValueError: - self.logger.warning(f"X-RateLimit-Remaining was not an integer. Value: {response.headers["X-RateLimit-Remaining"]}") + self.logger.warning(f"X-RateLimit-Remaining was not an integer. Value: {response.headers['X-RateLimit-Remaining']}") return response From 5e1fa515b6395dec6ea6316d78b8f3e560ed81a7 Mon Sep 17 00:00:00 2001 From: Cali Dolfi <47504866+cdolfi@users.noreply.github.com> Date: Wed, 26 Mar 2025 18:52:40 -0400 Subject: [PATCH 62/66] docker simplification and version upgrade --- .../data_analysis/clustering_worker/setup.py | 2 +- .../contributor_breadth_worker/setup.py | 2 +- .../data_analysis/discourse_analysis/setup.py | 2 +- .../data_analysis/insight_worker/setup.py | 2 +- .../data_analysis/message_insights/setup.py | 2 +- .../pull_request_analysis_worker/setup.py | 2 +- augur/tasks/git/util/facade_worker/setup.py | 2 +- docker-compose.yml | 25 ++++++++++--------- docker/backend/Dockerfile | 2 +- setup.py | 2 +- 10 files changed, 22 insertions(+), 21 deletions(-) diff --git a/augur/tasks/data_analysis/clustering_worker/setup.py b/augur/tasks/data_analysis/clustering_worker/setup.py index a197b21568..409dc4aeba 100644 --- a/augur/tasks/data_analysis/clustering_worker/setup.py +++ b/augur/tasks/data_analysis/clustering_worker/setup.py @@ -26,7 +26,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3', + 'psycopg2-binary==2.9.6', #'sklearn==0.0.0', 'scikit-learn==1.5.0', 'numpy==1.26.0', diff --git a/augur/tasks/data_analysis/contributor_breadth_worker/setup.py b/augur/tasks/data_analysis/contributor_breadth_worker/setup.py index 805edfb36b..536f12c48f 100644 --- a/augur/tasks/data_analysis/contributor_breadth_worker/setup.py +++ b/augur/tasks/data_analysis/contributor_breadth_worker/setup.py @@ -27,7 +27,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3' + 'psycopg2-binary==2.9.6' ], entry_points={ 'console_scripts': [ diff --git a/augur/tasks/data_analysis/discourse_analysis/setup.py b/augur/tasks/data_analysis/discourse_analysis/setup.py index ca936a6000..af0db8aad3 100644 --- a/augur/tasks/data_analysis/discourse_analysis/setup.py +++ b/augur/tasks/data_analysis/discourse_analysis/setup.py @@ -26,7 +26,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3', + 'psycopg2-binary==2.9.6', 'click==8.0.3', 'scipy>=1.10.0', 'nltk==3.6.6', diff --git a/augur/tasks/data_analysis/insight_worker/setup.py b/augur/tasks/data_analysis/insight_worker/setup.py index 92d663e3ae..524eaa1e9c 100644 --- a/augur/tasks/data_analysis/insight_worker/setup.py +++ b/augur/tasks/data_analysis/insight_worker/setup.py @@ -27,7 +27,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3', + 'psycopg2-binary==2.9.6', 'click==8.0.3', 'scipy>=1.10.0', 'sklearn==0.0', diff --git a/augur/tasks/data_analysis/message_insights/setup.py b/augur/tasks/data_analysis/message_insights/setup.py index 2f86701619..e5630d0cc7 100644 --- a/augur/tasks/data_analysis/message_insights/setup.py +++ b/augur/tasks/data_analysis/message_insights/setup.py @@ -28,7 +28,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3', + 'psycopg2-binary==2.9.6', 'click==8.0.3', 'scipy>=1.10.0', 'scikit-learn==1.5.0', #0.24.2', diff --git a/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py b/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py index 63ccbec1de..f7f61af562 100644 --- a/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py +++ b/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py @@ -26,7 +26,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3', + 'psycopg2-binary==2.9.6', 'sklearn==0.0', 'nltk==3.6.6', 'numpy==1.26.0', diff --git a/augur/tasks/git/util/facade_worker/setup.py b/augur/tasks/git/util/facade_worker/setup.py index e2a1af8b75..b20f7dd9cb 100644 --- a/augur/tasks/git/util/facade_worker/setup.py +++ b/augur/tasks/git/util/facade_worker/setup.py @@ -27,7 +27,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.3', + 'psycopg2-binary==2.9.6', 'click==8.0.3', 'XlsxWriter==1.3.7' ], diff --git a/docker-compose.yml b/docker-compose.yml index bdd1127931..737d59f8b3 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,14 +10,16 @@ services: - "POSTGRES_PASSWORD=${AUGUR_DB_PASSWORD:-augur}" - "PGDATA=/var/lib/postgresql/data/pgdata" ports: - - "127.0.0.1:${AUGUR_DB_PORT:-5432}:5432" + - 5432:5432 volumes: - augurpostgres:/var/lib/postgresql/data + networks: + - augur redis: image: "redis:alpine" - ports: - - 6379:6379 + networks: + - augur rabbitmq: image: augur-rabbitmq @@ -28,12 +30,8 @@ services: - RABBIT_MQ_DEFAULT_USER=${AUGUR_RABBITMQ_USERNAME:-augur} - RABBIT_MQ_DEFAULT_PASSWORD=${AUGUR_RABBITMQ_PASSWORD:-password123} - RABBIT_MQ_DEFAULT_VHOST=${AUGUR_RABBITMQ_VHOST:-augur_vhost} - # ports for amqp connections / management api - ports: - - 5671:5671 - - 5672:5672 - - 15671:15671 - - 15672:15672 + networks: + - augur augur: image: augur-new:latest @@ -45,8 +43,8 @@ services: restart: unless-stopped ports: - 5002:5000 - extra_hosts: - - "host.docker.internal:host-gateway" #Be able to ping services on the local machine + #extra_hosts: + # - "host.docker.internal:host-gateway" #Be able to ping services on the local machine environment: - "AUGUR_DB=postgresql+psycopg2://${AUGUR_DB_USER:-augur}:${AUGUR_DB_PASSWORD:-augur}@augur-db:5432/augur" - "AUGUR_DB_SCHEMA_BUILD=1" @@ -61,6 +59,8 @@ services: - augur-db - redis - rabbitmq + networks: + - augur # Flower is a UI that helps more easily monitor running tasks for celery workers. # This wont show you every error from the entire augur system, but it should show most of them. @@ -88,4 +88,5 @@ volumes: augurpostgres: driver: local - +networks: + augur: \ No newline at end of file diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 6142ef71c4..772c0e60ca 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -121,7 +121,7 @@ RUN set -x \ RUN ./scripts/docker/install-workers-deps.sh COPY --from=golang "/root/scc" "/root/scc" -COPY --from=golang "/root/scorecard/scorecard" "./scorecard" +COPY --from=golang "/root/scorecard/scorecard" "/root/scorecard" # RUN ./scripts/install/workers.sh diff --git a/setup.py b/setup.py index 8591f483e2..9e9d952839 100644 --- a/setup.py +++ b/setup.py @@ -52,7 +52,7 @@ "pandas==1.5.3", # 1.4.3 "numpy==1.26.0", # 1.23.2 "requests==2.32.0", # 2.28.1 - "psycopg2-binary==2.9.3", #2.9.3 what is pscopg-binary 3.0.16 + "psycopg2-binary==2.9.6", #2.9.3 what is pscopg-binary 3.0.16 "click==8.0.3", # 8.1.3 "psutil==5.8.0", # 5.9.1 "gunicorn==22.0.0", # 20.1.0 From ec1f374ca7263ae4528c8726c980e92970cbb86f Mon Sep 17 00:00:00 2001 From: John Strunk Date: Mon, 31 Mar 2025 15:30:30 -0400 Subject: [PATCH 63/66] Rework CI container builds - Move from custom scripting to re-usable actions - Add workflow_dispatch so build can be triggered manually Signed-off-by: John Strunk --- .github/workflows/build_docker.yml | 78 +++++++++++++++++++----------- .github/workflows/docker-image.yml | 20 -------- 2 files changed, 50 insertions(+), 48 deletions(-) delete mode 100644 .github/workflows/docker-image.yml diff --git a/.github/workflows/build_docker.yml b/.github/workflows/build_docker.yml index b29ab2ed89..a47a9374ca 100644 --- a/.github/workflows/build_docker.yml +++ b/.github/workflows/build_docker.yml @@ -11,45 +11,67 @@ on: release: types: - published + workflow_dispatch: permissions: {} jobs: build: + name: Build image permissions: contents: read # to fetch code (actions/checkout) packages: write # to push docker image - - name: Build image + strategy: + matrix: + image: + - backend + - database + - rabbitmq runs-on: ubuntu-latest steps: - - name: Checkout main - uses: actions/checkout@v2 - - name: Run the build - run: | - set -ex - # use that here since the variable are not present before start, so can't be in env - export LOGIN=$GITHUB_REPOSITORY_OWNER + - name: Checkout repository + uses: actions/checkout@v4 - echo $PASSWORD | docker login $REGISTRY -u $LOGIN --password-stdin + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + id: setup-buildx - for i in docker/* ; do - CONTAINER=$(basename $i) - echo "Building $CONTAINER" - export IMAGE=$LOGIN/augur_$CONTAINER - DOCKERFILE=${i}/Dockerfile + - name: Login to GitHub Container Registry + uses: docker/login-action@v3 + if: github.event_name != 'pull_request' + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} - docker build . -f $DOCKERFILE --tag $REGISTRY/$IMAGE:latest - if [[ $GITHUB_EVENT_NAME == 'release' ]]; then - TAG=$(basename $GITHUB_REF) - docker tag $REGISTRY/$IMAGE:latest $REGISTRY/$IMAGE:$TAG - docker push $REGISTRY/$IMAGE:latest - docker push $REGISTRY/$IMAGE:$TAG - elif [[ $GITHUB_EVENT_NAME == 'push' ]]; then - docker tag $REGISTRY/$IMAGE:latest $REGISTRY/$IMAGE:devel-latest - docker push $REGISTRY/$IMAGE:devel-latest - fi - done + - name: Set container metadata + uses: docker/metadata-action@v5 + id: meta env: - REGISTRY: ghcr.io - PASSWORD: ${{ secrets.GITHUB_TOKEN }} + DOCKER_METADATA_ANNOTATIONS_LEVELS: index,manifest + with: + annotations: | + org.opencontainers.image.title=augur_${{ matrix.image}} + labels: | + org.opencontainers.image.title=augur_${{ matrix.image}} + images: ghcr.io/${{ github.repository_owner }}/augur_${{ matrix.image }} + # Pushes to the dev branch update the *:devel-latest tag + # Releases update the *:latest tag and the *: tag + tags: | + type=raw,value=devel-latest,enable=${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/dev' }} + type=raw,value=latest,enable=${{ github.event_name == 'release' }} + type=raw,value=${{ github.event.release.tag_name }},enable=${{ github.event_name == 'release' }} + + - name: Build and push + id: push + uses: docker/build-push-action@v6 + with: + annotations: ${{ steps.meta.outputs.annotations }} + context: . + file: ./docker/${{ matrix.image }}/Dockerfile + labels: ${{ steps.meta.outputs.labels }} + platforms: linux/amd64 + push: ${{ github.event_name != 'pull_request' }} + tags: ${{ steps.meta.outputs.tags }} + cache-from: type=gha,scope=container-${{ matrix.image }} + cache-to: type=gha,scope=container-${{ matrix.image }},mode=max diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml deleted file mode 100644 index 3f19c12343..0000000000 --- a/.github/workflows/docker-image.yml +++ /dev/null @@ -1,20 +0,0 @@ -name: Docker Image CI - -on: - push: - branches: [ "main" ] - pull_request: - branches: [ "main" ] - -jobs: - - build: - - runs-on: ubuntu-latest - - steps: - - uses: actions/checkout@v3 - - name: Build the Docker image - run: | - ls -l - docker build . --file docker/backend/Dockerfile --tag my-image-name:$(date +%s) From f0b8def50fa5cacb0e0a8f6c31c20547ac6c536f Mon Sep 17 00:00:00 2001 From: Cali Dolfi <47504866+cdolfi@users.noreply.github.com> Date: Mon, 31 Mar 2025 17:30:47 -0400 Subject: [PATCH 64/66] update version and compose --- .../data_analysis/clustering_worker/setup.py | 2 +- .../contributor_breadth_worker/setup.py | 2 +- .../data_analysis/discourse_analysis/setup.py | 2 +- .../data_analysis/insight_worker/setup.py | 2 +- .../data_analysis/message_insights/setup.py | 2 +- .../pull_request_analysis_worker/setup.py | 2 +- augur/tasks/git/util/facade_worker/setup.py | 2 +- docker-compose.yml | 36 ++++++++++--------- setup.py | 2 +- 9 files changed, 27 insertions(+), 25 deletions(-) diff --git a/augur/tasks/data_analysis/clustering_worker/setup.py b/augur/tasks/data_analysis/clustering_worker/setup.py index 409dc4aeba..bd591554c3 100644 --- a/augur/tasks/data_analysis/clustering_worker/setup.py +++ b/augur/tasks/data_analysis/clustering_worker/setup.py @@ -26,7 +26,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6', + 'psycopg2-binary==2.9.9', #'sklearn==0.0.0', 'scikit-learn==1.5.0', 'numpy==1.26.0', diff --git a/augur/tasks/data_analysis/contributor_breadth_worker/setup.py b/augur/tasks/data_analysis/contributor_breadth_worker/setup.py index 536f12c48f..70a4cd6312 100644 --- a/augur/tasks/data_analysis/contributor_breadth_worker/setup.py +++ b/augur/tasks/data_analysis/contributor_breadth_worker/setup.py @@ -27,7 +27,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6' + 'psycopg2-binary==2.9.9' ], entry_points={ 'console_scripts': [ diff --git a/augur/tasks/data_analysis/discourse_analysis/setup.py b/augur/tasks/data_analysis/discourse_analysis/setup.py index af0db8aad3..67cb4dd7e7 100644 --- a/augur/tasks/data_analysis/discourse_analysis/setup.py +++ b/augur/tasks/data_analysis/discourse_analysis/setup.py @@ -26,7 +26,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6', + 'psycopg2-binary==2.9.9', 'click==8.0.3', 'scipy>=1.10.0', 'nltk==3.6.6', diff --git a/augur/tasks/data_analysis/insight_worker/setup.py b/augur/tasks/data_analysis/insight_worker/setup.py index 524eaa1e9c..b28b4bf5ab 100644 --- a/augur/tasks/data_analysis/insight_worker/setup.py +++ b/augur/tasks/data_analysis/insight_worker/setup.py @@ -27,7 +27,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6', + 'psycopg2-binary==2.9.9', 'click==8.0.3', 'scipy>=1.10.0', 'sklearn==0.0', diff --git a/augur/tasks/data_analysis/message_insights/setup.py b/augur/tasks/data_analysis/message_insights/setup.py index e5630d0cc7..bd136fdfec 100644 --- a/augur/tasks/data_analysis/message_insights/setup.py +++ b/augur/tasks/data_analysis/message_insights/setup.py @@ -28,7 +28,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6', + 'psycopg2-binary==2.9.9', 'click==8.0.3', 'scipy>=1.10.0', 'scikit-learn==1.5.0', #0.24.2', diff --git a/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py b/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py index f7f61af562..391b368f32 100644 --- a/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py +++ b/augur/tasks/data_analysis/pull_request_analysis_worker/setup.py @@ -26,7 +26,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6', + 'psycopg2-binary==2.9.9', 'sklearn==0.0', 'nltk==3.6.6', 'numpy==1.26.0', diff --git a/augur/tasks/git/util/facade_worker/setup.py b/augur/tasks/git/util/facade_worker/setup.py index b20f7dd9cb..ef65f223a2 100644 --- a/augur/tasks/git/util/facade_worker/setup.py +++ b/augur/tasks/git/util/facade_worker/setup.py @@ -27,7 +27,7 @@ def read(filename): 'Flask-Login==0.5.0', 'Flask-WTF==1.0.0', 'requests==2.32.0', - 'psycopg2-binary==2.9.6', + 'psycopg2-binary==2.9.9', 'click==8.0.3', 'XlsxWriter==1.3.7' ], diff --git a/docker-compose.yml b/docker-compose.yml index 737d59f8b3..81452bd09d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,7 +10,7 @@ services: - "POSTGRES_PASSWORD=${AUGUR_DB_PASSWORD:-augur}" - "PGDATA=/var/lib/postgresql/data/pgdata" ports: - - 5432:5432 + - "${AUGUR_DB_PORT:-5432}:5432" volumes: - augurpostgres:/var/lib/postgresql/data networks: @@ -65,22 +65,24 @@ services: # Flower is a UI that helps more easily monitor running tasks for celery workers. # This wont show you every error from the entire augur system, but it should show most of them. # uncomment the section below to use flower - # flower: - # image: augur-new:latest - # restart: unless-stopped - # command: - # [ "celery", "-A", "augur.tasks.init.celery_app.celery_app", "flower", "--max-tasks=1000000" ] - # ports: - # - 5555:5555 - # environment: - # - "AUGUR_DB=postgresql+psycopg2://${AUGUR_DB_USER:-augur}:${AUGUR_DB_PASSWORD:-augur}@augur-db:5432/augur" - # - REDIS_CONN_STRING=redis://redis:6379 - # - RABBITMQ_CONN_STRING=amqp://${AUGUR_RABBITMQ_USERNAME:-augur}:${AUGUR_RABBITMQ_PASSWORD:-password123}@rabbitmq:5672/${AUGUR_RABBITMQ_VHOST:-augur_vhost} - # depends_on: - # - augur - # - augur-db - # - redis - # - rabbitmq + #flower: + # image: augur-new:latest + # restart: unless-stopped + # command: + # [ "celery", "-A", "augur.tasks.init.celery_app.celery_app", "flower", "--max-tasks=1000000" ] + # ports: + # - 5555:5555 + # environment: + # - "AUGUR_DB=postgresql+psycopg2://${AUGUR_DB_USER:-augur}:${AUGUR_DB_PASSWORD:-augur}@augur-db:5432/augur" + # - REDIS_CONN_STRING=redis://redis:6379 + # - RABBITMQ_CONN_STRING=amqp://${AUGUR_RABBITMQ_USERNAME:-augur}:${AUGUR_RABBITMQ_PASSWORD:-password123}@rabbitmq:5672/${AUGUR_RABBITMQ_VHOST:-augur_vhost} + # depends_on: + # - augur + # - augur-db + # - redis + # - rabbitmq + # networks: + # - augur volumes: facade: diff --git a/setup.py b/setup.py index 9e9d952839..a356fef858 100644 --- a/setup.py +++ b/setup.py @@ -52,7 +52,7 @@ "pandas==1.5.3", # 1.4.3 "numpy==1.26.0", # 1.23.2 "requests==2.32.0", # 2.28.1 - "psycopg2-binary==2.9.6", #2.9.3 what is pscopg-binary 3.0.16 + "psycopg2-binary==2.9.9", #2.9.3 what is pscopg-binary 3.0.16 "click==8.0.3", # 8.1.3 "psutil==5.8.0", # 5.9.1 "gunicorn==22.0.0", # 20.1.0 From 3f9123cd80199ad9811ed0862a87fb2602879e64 Mon Sep 17 00:00:00 2001 From: "Sean P. Goggins" Date: Tue, 1 Apr 2025 09:20:25 -0500 Subject: [PATCH 65/66] version bump Signed-off-by: Sean P. Goggins --- README.md | 4 ++-- docker/backend/Dockerfile | 2 +- docker/database/Dockerfile | 2 +- docker/rabbitmq/Dockerfile | 2 +- metadata.py | 4 ++-- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index a50d94037e..33e41bcf15 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Augur NEW Release v0.85.3 +# Augur NEW Release v0.86.0 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). @@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o ## NEW RELEASE ALERT! **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**. -Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.3). +Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.86.0). - The `main` branch is a stable version of our new architecture, which features: diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 772c0e60ca..959d3952f5 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -32,7 +32,7 @@ RUN ./scripts/docker/install-golang-deps.sh FROM python:3.11-slim-bullseye LABEL maintainer="outdoors@acm.org" -LABEL version="0.85.3" +LABEL version="0.86.0" ENV DEBIAN_FRONTEND=noninteractive ENV PATH="/usr/bin/:/usr/local/bin:/usr/lib:${PATH}" diff --git a/docker/database/Dockerfile b/docker/database/Dockerfile index aa7dee1603..e5825ca7bf 100644 --- a/docker/database/Dockerfile +++ b/docker/database/Dockerfile @@ -2,7 +2,7 @@ FROM postgres:16 LABEL maintainer="outdoors@acm.org" -LABEL version="0.85.3" +LABEL version="0.86.0" ENV POSTGRES_DB "test" ENV POSTGRES_USER "augur" diff --git a/docker/rabbitmq/Dockerfile b/docker/rabbitmq/Dockerfile index d7c47ebbaa..9b0b3c98c3 100644 --- a/docker/rabbitmq/Dockerfile +++ b/docker/rabbitmq/Dockerfile @@ -1,7 +1,7 @@ FROM rabbitmq:3.12-management-alpine LABEL maintainer="574/augur@simplelogin.com" -LABEL version="0.85.3" +LABEL version="0.86.0" ARG RABBIT_MQ_DEFAULT_USER=augur ARG RABBIT_MQ_DEFAULT_PASSWORD=password123 diff --git a/metadata.py b/metadata.py index a4136cac86..740db52f3d 100644 --- a/metadata.py +++ b/metadata.py @@ -5,8 +5,8 @@ __short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection" -__version__ = "0.85.3" -__release__ = "v0.85.3 (Eurovision: Fire Saga)" +__version__ = "0.86.0" +__release__ = "v0.86.0 (Pod People)" __license__ = "MIT" __copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Sean Goggins, Brian Warner & Augurlabs 2025" From 3b542a4a4ccfbe3ae3453aee03f6fd5d2b18fc4b Mon Sep 17 00:00:00 2001 From: John Strunk Date: Tue, 1 Apr 2025 15:43:18 -0400 Subject: [PATCH 66/66] Fix failing CI when not tagging a docker image Signed-off-by: John Strunk --- .github/workflows/build_docker.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build_docker.yml b/.github/workflows/build_docker.yml index a47a9374ca..fbce968697 100644 --- a/.github/workflows/build_docker.yml +++ b/.github/workflows/build_docker.yml @@ -71,7 +71,8 @@ jobs: file: ./docker/${{ matrix.image }}/Dockerfile labels: ${{ steps.meta.outputs.labels }} platforms: linux/amd64 - push: ${{ github.event_name != 'pull_request' }} + # Only push if we've tagged the image in the metadata step + push: ${{ github.event_name != 'pull_request' && steps.meta.outputs.tags != '' }} tags: ${{ steps.meta.outputs.tags }} cache-from: type=gha,scope=container-${{ matrix.image }} cache-to: type=gha,scope=container-${{ matrix.image }},mode=max