修改代码仍然重复爬取

[Leo062410]

按照#66修改了search.py代码情况下，仍然出现anaconda prompt控制台中旧数据被反复爬取好几次后才爬到新数据，导致效率很低。感谢作者，希望能解答一下
DOWNLOAD_DELAY = 12

WEIBO_TYPE = 0

CONTAIN_TYPE = 0

START_DATE = '2020-06-08'

END_DATE = '2020-06-28'

FURTHER_THRESHOLD = 35

LIMIT_RESULT = 0

[dataabc]

删除crawls文件夹看看

[Leo062410]

删除crawls文件夹看看

试过了是一样的。之前对照微博看过，相当于控制台会重复爬取第一页数据多次后才爬取到第二页，然后再重复第二页多次再第三页

[dataabc]

删除crawls文件夹，并把search.py的start_requests方法修改为：

    def start_requests(self):
        start_date = datetime.strptime(self.start_date, '%Y-%m-%d')
        end_date = datetime.strptime(self.end_date,
                                     '%Y-%m-%d') + timedelta(days=1)
        start_str = start_date.strftime('%Y-%m-%d') + '-0'
        end_str = end_date.strftime('%Y-%m-%d') + '-0'
        for keyword in self.keyword_list:
            base_url = 'https://s.weibo.com/weibo?q=%s' % keyword
            url = base_url + self.weibo_type
            url += self.contain_type
            url += '&timescope=custom:{}:{}'.format(start_str, end_str)
            yield scrapy.Request(url=url,
                                    callback=self.parse,
                                    meta={
                                        'base_url': base_url,
                                        'keyword': keyword
                                    })

[Leo062410]

删除crawls文件夹，并把search.py的start_requests方法修改为：

    def start_requests(self):
        start_date = datetime.strptime(self.start_date, '%Y-%m-%d')
        end_date = datetime.strptime(self.end_date,
                                     '%Y-%m-%d') + timedelta(days=1)
        start_str = start_date.strftime('%Y-%m-%d') + '-0'
        end_str = end_date.strftime('%Y-%m-%d') + '-0'
        for keyword in self.keyword_list:
            base_url = 'https://s.weibo.com/weibo?q=%s' % keyword
            url = base_url + self.weibo_type
            url += self.contain_type
            url += '&timescope=custom:{}:{}'.format(start_str, end_str)
            yield scrapy.Request(url=url,
                                    callback=self.parse,
                                    meta={
                                        'base_url': base_url,
                                        'keyword': keyword
                                    })

解决了，非常感谢！！！