Skip to content

Comments

chore: improve dedicated scraping#15

Merged
hazedav merged 1 commit intomasterfrom
dedicated-scrapers
Feb 1, 2026
Merged

chore: improve dedicated scraping#15
hazedav merged 1 commit intomasterfrom
dedicated-scrapers

Conversation

@hazedav
Copy link
Owner

@hazedav hazedav commented Feb 1, 2026

Summary

This PR makes manufacturer filtering in FilamentProfilesScraper programmatic by introducing a manufacturer property on the base FilamentScraper class. Dedicated scrapers now self-identify, eliminating the need for hardcoded manufacturer lists.

Key Changes

  • Base class enhancement: Added optional manufacturer property to FilamentScraper that returns None by default
  • Dedicated scraper updates: PolymakerScraper, SunluScraper, and OvertureScraper now return their manufacturer name
  • Dynamic filtering: FilamentProfilesScraper uses get_dedicated_manufacturers() instead of a hardcoded tuple
  • New helper function: get_dedicated_manufacturers() in __init__.py introspects all scraper classes to build the exclusion set

Benefits

  • Extensibility: Adding a new dedicated scraper (e.g., HatchboxScraper with manufacturer = "Hatchbox") automatically filters that manufacturer from the aggregate scraper
  • Single source of truth: SCRAPER_CLASSES registry eliminates duplicate maintenance
  • Type safety: The manufacturer property is properly typed as str | None

Files Changed

File Change
filament_sites/base.py Added manufacturer property with docstring
filament_sites/__init__.py Added SCRAPER_CLASSES registry and get_dedicated_manufacturers()
filament_sites/filamentprofiles.py Replaced hardcoded tuple with dynamic lookup
filament_sites/polymaker.py Added manufacturer property returning "Polymaker"
filament_sites/sunlu.py Added manufacturer property returning "SUNLU"
filament_sites/overture.py Added manufacturer property returning "Overture"
filament_sites/test_filamentprofiles.py Added TestDynamicManufacturerFiltering test class

Test Plan

  • pytest teamtone/fetch/filament_sites/ passes (115 tests)
  • New tests verify get_dedicated_manufacturers() returns lowercase set
  • New tests confirm known scrapers (polymaker, sunlu, overture) are in the set
  • New tests ensure aggregate scraper doesn't pollute the set

🤖 Generated with Claude Code

@hazedav hazedav merged commit e185445 into master Feb 1, 2026
1 check passed
@hazedav hazedav deleted the dedicated-scrapers branch February 1, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant