Skip to content

Conversation

@mehallhm
Copy link
Collaborator

@mehallhm mehallhm commented Nov 30, 2025

the goal of the scraper rework is to make it:

  • more accurate to Banner
  • more resilient

this rework is nearly a complete rewrite of the scraper, building on everything learned from the first couple attempts. many of the larger changes are documented below, but it is non-comprehensive.

  • runs most network requests through a unified "engine" which handles retries, throttling, backoff, etc. in order to deal with Banner's rate limiting and unreliable api
  • checks and enforces schemas on returned data from Banner. this is probably the most significant change in terms of accuracy as it ensures the model we expect is returned, giving a very detailed understanding of the exact shape of returned information.
  • more to come

@vercel
Copy link

vercel bot commented Nov 30, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
searchneu Error Error Dec 8, 2025 7:54pm

@mehallhm
Copy link
Collaborator Author

merging work into #205

@mehallhm mehallhm closed this Dec 11, 2025
@mehallhm mehallhm deleted the feat/scraper-upgrades branch December 11, 2025 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants