This repository was archived by the owner on Oct 11, 2025. It is now read-only.
This repository was archived by the owner on Oct 11, 2025. It is now read-only.
抓取网页出现HTTP ERROR处理问题 #52
Description
1.http error 404 没有丢弃url
2.其他错误在爬虫执行完后，继续尝试，但不能无限次尝试（有的时候会出现爬虫任务根本停不下来）