Python script to crawl WARC data and find what sites disallow Googlebot
Feel free to submit a pull request with any improvements
Source for crawl data: https://commoncrawl.org/the-data/get-started/
tcope25/warc-robots-analysis
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|