Skip to content

Added changes to the makefile#105

Merged
ediakatos merged 2 commits intomainfrom
feature/makefile/auto
Jun 17, 2025
Merged

Added changes to the makefile#105
ediakatos merged 2 commits intomainfrom
feature/makefile/auto

Conversation

@ediakatos
Copy link
Copy Markdown
Collaborator

This pull request introduces a new Makefile target for running all download scripts together and updates the Selenium configuration in the data_acquisition_scrape.py script to streamline the scraping process. Below are the most important changes grouped by theme:

New Makefile Target:

  • Added a run_all_download target in the Makefile to execute multiple download scripts (run_glide_download, run_cerf_download, run_disaster_charter_download, and run_idus_download) in a single command. This simplifies running all downloads at once.

Selenium Configuration Updates:

  • Updated the GECKODRIVER_PATH in src/glide/data_acquisition_scrape.py to point to /usr/local/bin/geckodriver for better compatibility.
  • Restored and uncommented the PROFILE_PATH pointing to a Linux-based Firefox profile, replacing the macOS-specific path, to improve cross-platform functionality.
  • Reactivated the use of the -profile argument with the PROFILE_PATH in the scrape_with_selenium function to ensure the correct Firefox profile is used during scraping.

@ediakatos ediakatos merged commit 17303e2 into main Jun 17, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant