A repository for automation scripts to sync OGD logs from MySQL to BigQuery.
Setup:
- Install python3 for your environment
- Install python dependencies: "pip3 install -r requirements.txt"
- Copy
config.py.templatetoconfig.pyset server/authentication data and the source & destination db tables - Download the authentication key needed for the BigQuery project. Save it as a .json file in the
configdirectory and ensure the file path is defined inconfig.py
usage: <python> main.py <game> --max-days <count>
<python> is your python command.
<game> is the game whose data you wish to move to BigQuery
<count> is the max number of days-worth of data you wish to moveThese processes are also set up to run automatically in GitHub actions. Current workflows are configured to run at the following times:
| Time (UTC) | Time (Central) | Time (Central, Daylight) | Frequency | Game | Status |
|---|---|---|---|---|---|
| 8:40 | 2:40 (AM) | 3:40 (AM) | Daily | ANSWER_CAMPUS |  |
| 9:00 | 3:00 (AM) | 4:00 (AM) | Daily | AQUALAB |  |
| 8:55 | 2:55 (AM) | 3:55 (AM) | Daily | AYCET |  |
| 9:05 | 3:05 (AM) | 4:05 (AM) | Daily | BACTERIA |  |
| 10:45 | 4:45 (AM) | 5:45 (AM) | Daily | BALLOON |  |
| 9:10 | 3:10 (AM) | 4:10 (AM) | Daily | BLOOM |  |
| 8:50 | 2:50 (AM) | 3:50 (AM) | Daily | CRUSHSTATIONS |  |
| 9:15 | 3:15 (AM) | 4:15 (AM) | Daily | CRYSTAL |  |
| 9:20 | 3:20 (AM) | 4:20 (AM) | Daily | CYCLE_CARBON |  |
| 9:25 | 3:25 (AM) | 4:25 (AM) | Daily | CYCLE_NITROGEN |  |
| 9:30 | 3:30 (AM) | 4:30 (AM) | Daily | CYCLE_WATER |  |
| 9:35 | 3:35 (AM) | 4:35 (AM) | Daily | EARTHQUAKE |  |
| 11:15 | 5:15 (AM) | 6:15 (AM) | Daily | ASTRO |  |
| 8:45 | 2:45 (AM) | 3:45 (AM) | Daily | GWAKKAMOLE |  |
| 9:40 | 3:40 (AM) | 4:40 (AM) | Daily | ICECUBE |  |
| 9:45 | 3:45 (AM) | 4:45 (AM) | Daily | JOURNALISM |  |
| 9:50 | 3:50 (AM) | 4:50 (AM) | Daily | JOWILDER |  |
| 9:55 | 3:55 (AM) | 4:55 (AM) | Daily | LAKELAND |  |
| 10:00 | 4:00 (AM) | 5:00 (AM) | Daily | MAGNET |  |
| 10:05 | 4:05 (AM) | 5:05 (AM) | Daily | MASHOPOLIS |  |
| 11:05 | 5:05 (AM) | 6:05 (AM) | Weekly | MATCH |  |
| 10:10 | 4:10 (AM) | 5:10 (AM) | Daily | PENGUINS |  |
| 11:10 | 5:10 (AM) | 6:10 (AM) | Daily | PENNYCOOK |  |
| 10:15 | 4:15 (AM) | 5:15 (AM) | Daily | SHADOWSPECT |  |
| 10:20 | 4:20 (AM) | 5:20 (AM) | Daily | SHIPWRECKS |  |
| 10:55 | 4:55 (AM) | 5:55 (AM) | Weekly | SLIDE |  |
| 11:00 | 5:00 (AM) | 6:00 (AM) | Weekly | STACK |  |
| 10:25 | 4:25 (AM) | 5:25 (AM) | Daily | THERMOLAB |  |
| 10:30 | 4:30 (AM) | 5:30 (AM) | Daily | TRANSFORMATION_QUEST |  |
| 11:20 | 5:20 (AM) | 6:20 (AM) | Daily | TE_BACTERIA |  |
| 10:35 | 4:35 (AM) | 5:35 (AM) | Daily | WAVES |  |
| 10:50 | 4:50 (AM) | 5:50 (AM) | Daily | WEATHER_STATION |  |
| 10:40 | 4:40 (AM) | 5:40 (AM) | Daily | WIND |  |
Current latest run: 11:20 UTC/5:20 CST
Last keep-alive on 01/01/26