python网络爬虫——CrawlSpider
发布时间:2020-12-20 10:29:51 所属栏目:Python 来源:网络整理
导读:- CrawlSpider - 作用:用于进行全站数据爬取 - CrawlSpider就是Spider的一个子类 - 如何新建一个基于CrawlSpider的爬虫文件 - scrapy genspider -t crawl xxx www.xxx.com - 例:choutiPro - LinkExtractor连接提取器:根据指定规则(正则)进行连接的提取
- CrawlSpider - 分布式(通常用不到,爬取数据量级巨大、时间少时用分布式) 抽屉: # spider文件 import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider,Rule class ChoutiSpider(CrawlSpider): name = ‘chouti‘ # allowed_domains = [‘www.xxx.com‘] start_urls = [‘https://dig.chouti.com/1‘] # 连接提取器:从起始url对应的页面中提取符合规则的所有连接;allow=正则表达式 # 正则为空的话,提取页面中所有连接 link = LinkExtractor(allow=r‘d+‘) rules = ( # 规则解析器:将连接提取器提取到的连接对应的页面源码进行指定规则的解析 # Rule自动发送对应链接的请求 Rule(link,callback=‘parse_item‘,follow=True),# follow:True 将连接提取器 继续 作用到 连接提取器提取出来的连接 对应的页面源码中 ) def parse_item(self,response): item = {} #item[‘domain_id‘] = response.xpath(‘//input[@id="sid"]/@value‘).get() #item[‘name‘] = response.xpath(‘//div[@id="name"]‘).get() #item[‘description‘] = response.xpath(‘//div[@id="description"]‘).get() return item ? 阳光热线网 # 1.spider文件 import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider,Rule from sunLineCrawl.items import SunlinecrawlItem,ContentItem class SunSpider(CrawlSpider): name = ‘sun‘ # allowed_domains = [‘www.xxx.com‘] start_urls = [‘http://wz.sun0769.com/index.php/question/questionType?type=4&page=‘] link = LinkExtractor(allow=r‘type=4&page=d+‘) # 提取页码连接 link1 = LinkExtractor(allow=r‘question/2019d+/d+.shtml‘) # 提取详情页连接 rules = ( Rule(link,callback=‘parse_item‘,follow=False),Rule(link1,callback=‘parse_detail‘),) # 解析出标题和网友名称数据 def parse_item(self,response): tr_list = response.xpath(‘//*[@id="morelist"]/div/table[2]//tr/td/table//tr‘) for tr in tr_list: title = tr.xpath(‘./td[2]/a[2]/text()‘).extract_first() net_friend = tr.xpath(‘./td[4]/text()‘).extract_first() item = SunlinecrawlItem() item[‘title‘] = title item[‘net_friend‘] = net_friend yield item # 解析出新闻的内容 def parse_detail(self,response): content = response.xpath(‘/html/body/div[9]/table[2]//tr[1]/td/div[2]//text()‘).extract() content = ‘‘.join(content) item = ContentItem() item[‘content‘] = content yield item -------------------------------------------------------------------------------- # 2.items文件 import scrapy class SunlinecrawlItem(scrapy.Item): title = scrapy.Field() net_friend = scrapy.Field() class ContentItem(scrapy.Item): content = scrapy.Field() -------------------------------------------------------------------------------- # 3.pipelines文件 class SunlinecrawlPipeline(object): def process_item(self,item,spider): # 确定接受到的item是什么类型(Content/Sunlinecrawl) if item.__class__.__name__ == ‘SunlinecrawlItem‘: print(item[‘title‘],item[‘net_friend‘]) else: print(item[‘content‘]) return item -------------------------------------------------------------------------------- # 4.setting文件 BOT_NAME = ‘sunLineCrawl‘ SPIDER_MODULES = [‘sunLineCrawl.spiders‘] NEWSPIDER_MODULE = ‘sunLineCrawl.spiders‘ LOG_LEVEL = ‘ERROR‘ USER_AGENT = ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/76.0.3809.132 Safari/537.36‘ ROBOTSTXT_OBEY = False ITEM_PIPELINES = { ‘sunLineCrawl.pipelines.SunlinecrawlPipeline‘: 300,} (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |