Skip to content

Commit afce512

Browse files
z64555z64555
authored
Repair Automated Forum and Nebula Scripts (#4052)
* Document things * Maybe use a format that doxygen can understand * Use dict comprehension to clean up main.groups a bit * More docstrings * Move FileGroup into module its supposed to be in * Remove nightly-trigger.yml * Rename workflow files to better describe their actions * Re-add post build capabilities * Post-build may be triggered by build-release.yaml or manually * minor doc update to file_list.get_ftp_files() * Maybe fix yaml errors * "You have a syntax error on L345" Gee, thanks github debugger. Could you tell me what exactly that error may be? * More fixes * More fixes * fix typo * fix typo * Allow manual trigger to specify releaseTag * Try omitting the default for string * Use correct .inputs member * try environment instead of env * Try elevating scope of env * Comment out stuff to directly test env passing * See if echo is working correctly * Test if top level env is in scope * Try github.event instead of github.events * Uncomment lines, use correct context signature * Pass ref_name to post-build-release workflow * Fix arguments to git Stash linux package since it can fail, too * str and repr cannot be compared * Get repo metadata differently to allow tests on forks * echo repository var * config["templates"] does not have a member ["installer"] so... we'll use ["release"] instead * debug render_installer_config. file groups malformed? * Print out both keys of groups dict * debug get_release_files. print binary_file list * Revert file_list debug print Try sorting binary_files by ReleaseFile.name * Add checks before running mako to improve debugging QoL * revert change to installer.py * maybe fix call to render_nebula_release * type the files argument to get IDE hints * Revert typing.... :( * missing parenthesis * Disabled installer mako rendering * Error check in ForumAPI to allow tests to continue * debug print hlp forum key stuff * Instead of none, check ""
1 parent 9b16413 commit afce512

File tree

14 files changed

+1082
-1
lines changed

14 files changed

+1082
-1
lines changed

‎.github/workflows/build-release.yaml‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,12 @@ jobs:
181181
working-directory: ./builds
182182
run: $GITHUB_WORKSPACE/ci/linux/create_dist_pack.sh Linux
183183
- name: Upload result package
184+
# Stash the result to artifact filespace
185+
uses: actions/upload-artifact@v2
186+
with:
187+
name: ${{ steps.generate_package.outputs.package_name }}
188+
path: ${{ steps.generate_package.outputs.package_path }}
189+
- name: Publish result package
184190
# publish the zipped result
185191
uses: softprops/action-gh-release@v1
186192
env:
@@ -335,4 +341,13 @@ jobs:
335341
with:
336342
files: |
337343
${{ steps.generate_package.outputs.package_path }}
338-
${{ steps.generate_package.outputs.debug_path }}
344+
${{ steps.generate_package.outputs.debug_path }}
345+
346+
post_build:
347+
name: Post builds to Nebula and Forum
348+
needs: [linux_zip, windows_zip]
349+
uses: ./.github/workflows/post-build-release.yaml
350+
with:
351+
linux_result: ${{ needs.build_linux.result }}
352+
windows_result: ${{ needs.build_windows.result }}
353+
releaseTag: ${{ github.ref_name }}

.github/workflows/standalone-image-workflow.yaml renamed to .github/workflows/build-standalone.yaml

File renamed without changes.

.github/workflows/master-build.yaml renamed to .github/workflows/cache-master.yaml

File renamed without changes.

‎.github/workflows/post-build-release.yaml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
name: Post Release builds
2+
3+
on:
4+
workflow_call:
5+
# Trigger when called by other workflows
6+
inputs:
7+
linux_result: # job.result; either success, failure, canceled or skipped
8+
required: true
9+
type: boolean
10+
windows_result: # job.result
11+
required: true
12+
type: boolean
13+
releaseTag:
14+
required: true
15+
type: string
16+
17+
workflow_dispatch:
18+
# Trigger manually from Github
19+
inputs:
20+
releaseTag:
21+
description: Release Tag
22+
required: true
23+
type: string
24+
25+
env:
26+
GITHUB_REPO: ${{ github.repository }} # repo this workflow was called from
27+
INDIEGAMES_USER: ${{ secrets.INDIEGAMES_USER }}
28+
INDIEGAMES_SSHPASS: ${{ secrets.INDIEGAMES_PASSWORD }}
29+
NEBULA_USER: ${{ secrets.NEBULA_USER }}
30+
NEBULA_PASSWORD: ${{ secrets.NEBULA_PASSWORD }}
31+
HLP_API: ${{ secrets.HLP_API }}
32+
HLP_KEY: ${{ secrets.HLP_KEY }}
33+
RELEASE_TAG: ${{ github.event.inputs.releaseTag }}
34+
35+
jobs:
36+
post_builds:
37+
name: Post builds on Nebula and the forums
38+
runs-on: ubuntu-latest
39+
steps:
40+
- name: Checkout
41+
uses: actions/checkout@v1
42+
with:
43+
fetch-depth: '0'
44+
- name: Install Python dependencies
45+
run: pip install -r ci/post/requirements.txt
46+
- name: Post Builds (Auto trigger)
47+
if: ${{ github.event_name == 'workflow_call' }}
48+
env:
49+
LINUX_RESULT: ${{ github.event.inputs.linux_result }}
50+
WINDOWS_RESULT: ${{ github.event.inputs.windows_result }}
51+
run: python ci/post/main.py release
52+
53+
- name: Post Builds (Manual trigger)
54+
if: ${{ github.event_name == 'workflow_dispatch' }}
55+
# assume user knows what they're doing...
56+
env:
57+
LINUX_RESULT: success
58+
WINDOWS_RESULT: success
59+
run: |
60+
echo "releaseTag= ${{ github.event.inputs.releaseTag }}"
61+
echo "repository= ${{ github.repository }}"
62+
python ci/post/main.py release

.github/workflows/pr-workflow.yaml renamed to .github/workflows/test-pull_request.yaml

File renamed without changes.

‎ci/post/file_list.py‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
import re # regex module
2+
from ftplib import FTP, error_perm
3+
from itertools import groupby
4+
from typing import List, Tuple, Dict
5+
6+
import requests # HTTP requests module
7+
8+
from util import retry_multi, GLOBAL_TIMEOUT # from util.py
9+
10+
11+
class ReleaseFile:
12+
"""! Class representing a Released file on Nebula
13+
14+
`name`: str
15+
Mod (or build) name,
16+
`url`: str
17+
Primary host URL,
18+
`group`: str
19+
Mod group string,
20+
`subgroup`: str
21+
Mod subgroup string,
22+
`mirrors`: List[str]
23+
List of URL's of FTP mirrors
24+
"""
25+
def __init__(self, name, url, group, subgroup=None, mirrors=None):
26+
if mirrors is None:
27+
mirrors = []
28+
self.mirrors = mirrors
29+
self.subgroup = subgroup
30+
self.group = group
31+
self.url = url
32+
self.name = name
33+
34+
self.base_url = "/".join(url.split('/')[0:-1]) + "/"
35+
self.filename = url.split('/')[-1]
36+
37+
# A list of tuples of (filename, hash)
38+
self.content_hashes = None
39+
40+
self.hash = None
41+
self.size = 0
42+
43+
def __repr__(self):
44+
return repr((self.name))
45+
46+
47+
class SourceFile:
48+
"""! Class represeting a source file
49+
50+
`name`: str
51+
File name,
52+
`url`: str
53+
FTP URL,
54+
`group`
55+
<unknown>
56+
57+
@details More details
58+
"""
59+
def __init__(self, name, url, group):
60+
self.group = group
61+
self.url = url
62+
self.name = name
63+
64+
65+
class FileGroup:
66+
"""! Represents a file group
67+
68+
`name`: str
69+
Name of this group
70+
`files`: List[ReleaseFile]
71+
List of files within this group
72+
`mainFile`: str
73+
If this FileGroup has a subgroup, `mainFile` is the head of that group
74+
`subFiles`: List[ReleaseFile]
75+
Files within a subgroup
76+
"""
77+
78+
def __init__(self, name, files: List[ReleaseFile]):
79+
self.files = files
80+
self.name = name
81+
82+
if len(files) == 1:
83+
self.mainFile = files[0]
84+
self.subFiles = {}
85+
else:
86+
self.mainFile = None
87+
subFiles = []
88+
for file in files:
89+
# We only have subcategories for Windows where SSE2 is the main group
90+
if file.subgroup == "SSE2":
91+
self.mainFile = file
92+
else:
93+
subFiles.append(file)
94+
95+
self.subFiles = dict(((x[0], next(x[1])) for x in groupby(subFiles, lambda f: f.subgroup)))
96+
97+
98+
def get_release_files(tag_name, config) -> Tuple[List[ReleaseFile], Dict[str, SourceFile]]:
99+
"""! Brief Gets the binary and source files from the Github Release server
100+
101+
@param[in] `tag_name` Git tag of the current release
102+
@param[in] `config` confi metadata set in main.py
103+
104+
@returns `List[ReleaseFile]` List of release files
105+
@returns `Dict[str, SourceFile]` Dictionary of source files
106+
107+
@details Sends an `HTTP GET` request to github using their REST API to retrieve metadata. The files are not
108+
actually downloaded here, just their metadata is gathered and organized in their respective container for later
109+
use.
110+
"""
111+
112+
@retry_multi(5) # retry at most 5 times
113+
def execute_request(path):
114+
"""!
115+
@brief Performs a GET request with the given path. To be used with Github's REST API.
116+
@returns If successful, returns a .JSON object
117+
"""
118+
headers = {
119+
"Accept": "application/vnd.github.v3+json"
120+
}
121+
url = "https://api.github.com" + path
122+
123+
# GET https://api.github.com/<path> Accept: "application/vnd.github.v3+json"
124+
125+
response = requests.get(url, headers=headers, timeout=GLOBAL_TIMEOUT)
126+
127+
response.raise_for_status() # Raise a RequestException if we failed, and trigger retry
128+
129+
return response.json()
130+
131+
build_group_regex = re.compile("fs2_open_.*-builds-([^.-]*)(-([^.]*))?.*") # regex for matching binary .zip's and .7z's
132+
source_file_regex = re.compile("fs2_open_.*-source-([^.]*)?.*") # regex for matching source .zip's and .7z's
133+
134+
# Get the github release metadata of the given tag name
135+
response = execute_request(
136+
"/repos/{}/releases/tags/{}".format(config["github"]["repo"], tag_name))
137+
138+
# Extract the binary and source files from the response["asset"] metadata
139+
binary_files = []
140+
source_files = {}
141+
for asset in response["assets"]:
142+
url = asset["browser_download_url"]
143+
name = asset["name"]
144+
145+
group_match = build_group_regex.match(name)
146+
147+
if group_match is not None:
148+
platform = group_match.group(1)
149+
# x64 is the Visual Studio name but for consistency we need Win64
150+
if platform == "x64":
151+
platform = "Win64"
152+
153+
binary_files.append(ReleaseFile(name, url, platform, group_match.group(3)))
154+
else:
155+
group_match = source_file_regex.match(name)
156+
157+
if group_match is None:
158+
continue
159+
160+
group = group_match.group(1)
161+
162+
source_files[group] = SourceFile(name, url, group)
163+
164+
binary_files.sort(key=lambda ReleaseFile: ReleaseFile.name)
165+
166+
return binary_files, source_files
167+
168+
169+
def get_ftp_files(build_type, tag_name, config) -> List[ReleaseFile] :
170+
"""!
171+
@brief Gets file metadata for nightlies hosted on FTP, as determined by config["ftp"] attributes
172+
173+
@param [in] `build_type` Unknown str
174+
@param [in] `tag_name` Github tag name of the release
175+
@param [in] `config` config metadata set in main.py
176+
"""
177+
178+
tag_regex = re.compile("nightly_(.*)")
179+
build_group_regex = re.compile("nightly_.*-builds-([^.]+).*")
180+
181+
files = []
182+
try:
183+
with FTP(config["ftp"]["host"], config["ftp"]["user"], config["ftp"]["pass"]) as ftp:
184+
# extract version
185+
version_str = tag_regex.match(tag_name).group(1)
186+
187+
# extract filepath w/ version
188+
# then list all ftp hits with that path
189+
path_template = config["ftp"]["path"]
190+
path = path_template.format(type=build_type, version=version_str)
191+
file_entries = list(ftp.mlsd(path, ["type"]))
192+
193+
# get all ftp hits of type file
194+
for entry in file_entries:
195+
if entry[1]["type"] == "file":
196+
files.append(entry[0])
197+
except error_perm:
198+
print("Received permanent FTP error!")
199+
return []
200+
201+
out_data = []
202+
for file in files:
203+
# from the file list, extract only nightly files
204+
file_match = build_group_regex.match(file)
205+
if file_match is None:
206+
print("Ignoring non nightly file '{}'".format(file))
207+
continue
208+
209+
group_match = file_match.group(1)
210+
primary_url = None
211+
mirrors = []
212+
213+
# x64 is the name Visual Studio uses but Win64 works better for us since that gets displayed in the nightly post
214+
if "x64" in group_match:
215+
group_match = group_match.replace("x64", "Win64")
216+
217+
# construct the download URL list for all mirrors. The first listed ftp location is taken as the Primary
218+
for mirror in config["ftp"]["mirrors"]:
219+
download_url = mirror.format(type=build_type, version=version_str, file=file)
220+
if primary_url is None:
221+
primary_url = download_url
222+
else:
223+
mirrors.append(download_url)
224+
225+
# Form the List[ReleaseFile] list with the download URL links
226+
out_data.append(ReleaseFile(file, primary_url, group_match, None, mirrors))
227+
228+
return out_data

‎ci/post/forum.py‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
from typing import List
2+
3+
import requests
4+
import semantic_version
5+
from mako.template import Template
6+
7+
from file_list import ReleaseFile
8+
9+
10+
class ForumAPI:
11+
def __init__(self, config):
12+
self.config = config
13+
14+
def create_post(self, title, content, board):
15+
print(self.config["hlp"]["api"])
16+
print(self.config["hlp"]["key"])
17+
if (self.config["hlp"]["api"] == "") or (self.config["hlp"]["key"] == ""):
18+
print("Post failed! No API or API_KEY given!")
19+
return
20+
21+
resp = requests.post(self.config["hlp"]["api"], data={
22+
"api_key": self.config["hlp"]["key"],
23+
"board": str(board),
24+
"subject": title,
25+
"body": content
26+
})
27+
28+
if resp.text != "OK":
29+
print("Post failed! Response: %s" %resp.text)
30+
31+
def post_nightly(self, date, revision, files, log, success):
32+
print("Posting nightly thread...")
33+
34+
title = "Nightly: {} - Revision {}".format(date, revision)
35+
36+
template = Template(filename=self.config["templates"]["nightly"])
37+
rendered = template.render_unicode(**{
38+
"date": date,
39+
"revision": revision,
40+
"files": files,
41+
"log": log,
42+
"success": success
43+
})
44+
45+
print("Creating post...")
46+
self.create_post(title, rendered, self.config["nightly"]["hlp_board"])
47+
48+
def post_release(self, date, version: semantic_version.Version, groups, sources):
49+
print("Posting release thread...")
50+
51+
title = "Release: {}".format(version)
52+
53+
template = Template(
54+
filename=self.config["templates"]["release"].format(major=version.major, minor=version.minor),
55+
module_directory='/tmp/mako_modules')
56+
rendered = template.render_unicode(**{
57+
"date": date,
58+
"version": version,
59+
"groups": groups,
60+
"sources": sources
61+
}).strip("\n")
62+
63+
print("Creating post...")
64+
65+
self.create_post(title, rendered, self.config["release"]["hlp_board"])

0 commit comments

Comments
 (0)