Responsible crawling enforcement

[zigmars-ancveirs]

Goal

Make responsible crawling policy enforceable (not just a doc).

Deliverables

• Rate limiting + robots handling where appropriate
• User-agent policy + backoff
• Logging of fetch attempts (aggregate)

Done when

• A crawler run shows policy adherence in logs