Standardize cronjob HTTP fetchers with a shared client wrapper

[rootp1]

At the moment, our backend cronjobs and feed fetchers rely on adhoc 'requests' calls, with each file handling timeouts and errors in its own way.
In this issue, I am wanting to propose gradually migrating to httpx, starting with a small set of fetchers and expanding over time rather than attempting a large, one shot rewrite.

httpx provides a modern, flexible API with built-in support for both sync and async usage, allowing us do this without changing behavior today.

It offers more structured timeout handling and a clearer exception hierarchy.
Additionally, it makes it easier to build a shared HTTP client wrapper, reducing duplication and improving scalability of our cronjobs.

[Demiserular]

are you working on this issue?

[Prasad8830]

Hi @Demiserular, @rootp1, I’d like to start working on it. Could you suggest a couple of low-risk fetchers or modules I can begin with?

[Demiserular]

@Prasad8830 maybe you can go with greedybear/cronjobs/tor_exit_nodes.py -- its a low risk like No auth, no JSON schema, no pagination, no multi-step flows. Single simple GET to a stable plaintext endpoint, then regex parse (r.text) and iterate.

[Prasad8830]

@Demiserular, Ok I will proceed with it.

[Prasad8830]

Hello @Demiserular,

I have migrated the endpoint you mentioned (tor_exit_nodes). Could you please review it?

For the next step, I’m considering a low-risk migration for whatsmyip (/greedybear/cronjobs/whatsmyip.py). Should I proceed with this?

[Demiserular]

I am not maintainer

[Prasad8830]

@Demiserular, ok!

[Prasad8830]

@Demiserular, @rootp1, so, should I proceed with the next migration?

[Demiserular]

@regulartim please review this pr

[regulartim]

Hey @rootp1 ! Thanks for bringing this up.

At the moment, our backend cronjobs and feed fetchers rely on adhoc 'requests' calls, with each file handling timeouts and errors in its own way.

I agree that this could be improved, with a wrapper or a common base class.

Additionally, it makes it easier to build a shared HTTP client wrapper, reducing duplication and improving scalability of our cronjobs.

Why would httpx make that easier?

[rootp1]

Hey @rootp1 ! Thanks for bringing this up.

Thankyou @regulartim .

I agree that this could be improved, with a wrapper or a common base class.

Yes, Tim.

Why would httpx make that easier?

I agree the common base class is the main improvement here, regardless of the library.

My thinking with httpx is that it fits more naturally with that approach since it’s client side first, making it easier to implement shared config like timeouts, headers, and later extend to async if needed.

But even if we stick with requests, introducing a base class would already solve most of the inconsistency, httpx just makes that pattern cleaner and more scalable.

[regulartim]

But even if we stick with requests, introducing a base class would already solve most of the inconsistency, httpx just makes that pattern cleaner and more scalable.

Yeah, I would prefer to stick to requests for now.