scraper-component for declarative scraping

[Niedzwiedzw]

Hey, so here's something I've been working on over the weekend for one of my personal usecases:
https://github.com/Niedzwiedzw/scraper-component

It has couple nice features like compile time checked selectors (during macro resolution) and yeah I think it's easier to maintain (error messages are very verbose and informative). Of course it has some rough edges but yeah I figured someone here might find that useful

[adamreichold]

I think creating a static LazyLock<Selector> for every selector is a bit inefficient even if convenient. I would suggest add something like fn extractor() -> FooExtractor to the implementation of a component Foo which yields a struct containing all necessary selectors. Calling can then manage this itself and you could still provide a convenience wrapper which uses a static LayzLock<FooExtractor> internally.

[Niedzwiedzw]

hmm well it probably does add some overhead on first call but after that it's just a pointer dereference right? it should be opt-in though, just like compile-time selector checking (very convenient!) but I was thinking of doing it behind a feature flag

[adamreichold]

but after that it's just a pointer dereference right?

Not really, it still needs to check in synchronised manner whether initialisation has actually happened. It will also initialise a futex to implement the blocking behaviour for the chance of concurrent access.

It isn't much in the uncontested fast path, but it can add up especially if you have a lot of fine-grained LazyLocks.

[Niedzwiedzw]

so is it actually cheaper to re-parse the selectors every time? or it just depends and should be up to the user?

[Niedzwiedzw]

it will also initialise a futex

so maybe thread_local! {} would be more appropriate?

[adamreichold]

so is it actually cheaper to re-parse the selectors every time?

No, the LazyLock will definitely be cheaper. But my point is that programs often already have a lifecycle for auxiliary data like this so there is no need hide it behind their back.

You give user code the ability to manage a struct containing the necessary selectors and such as it sees fit. If you want, you can provide a LazyLock around that single struct as a convenience and still reduce the overhead compared to separate LazyLocks for each selector.

[adamreichold]

I am not really sure why the Single wrappers are necessary? Why not implement TryFromElement directly for Element?

[Niedzwiedzw]

it is a bit of a WIP, but essentially there is not really a way to distinguish between collections and single elements, and .select() yields an iterator. But of course it would be better if it was solved at macro level (#[component(single)]) cause it gets tiresome to unwrap those single-element arrays every time, I think I'm gonna do that today

[adamreichold]

but after that it's just a pointer dereference right?

So overlapping trait impls? Isn't more a problem as to how you cut your trait hierarchy? You do own the relevant traits...

[Niedzwiedzw]

well this is my best idea to solve this cleanly - API is already a bit too implicit for my taste so I think it's better to annotate fields which expect a collection of results. but yeah, Single being a bit too invasive, I changed it to #[component(many)] for fields which are gonna collect into multiple values (no #[component(many)] implies [T; 1] and unpacks it automatically).

Niedzwiedzw/scraper-component@82ac3ab