|
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,Arial,'Hiragino Sans GB','Microsoft YaHei',simsun;vertical-align:baseline;color:rgb(93,93,93);background-color:rgb(255,255,255);"> 
<table style="margin:0px auto 10px;padding:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,simsun;vertical-align:baseline;border-spacing:0px;color:rgb(93,255);"><tbody style="margin:0px;padding:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><tr class="firstRow" style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Project</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Language</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Star</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Watch</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Fork</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Nutch</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">Java</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">1111</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">195</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">808</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">webmagic</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">Java</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">4216</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">618</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">2306</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">WebCollector</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">1222</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">255</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">958</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">heritrix3</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">773</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">141</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">428</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">crawler4j</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">1831</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">242</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">1136</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Pyspider</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">Python</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">8581</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">687</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">2273</td></tr><tr style="margin:0px;padding:0px;border-top:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;"><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:700;line-height:inherit;font-family:inherit;vertical-align:baseline;">Scrapy</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">Python</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">19642</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">1405</td><td style="margin:0px;padding:5px;border-color:rgb(218,218);font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:#FFFFFF;">5261</td></tr></table><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">看到了吗？星星数排名第一的Scrapy比其他所有的加起来都要多，我仿佛听到他这样说：<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">优点：<ul class="list-paddingleft-2" style="margin-top:1em;margin-bottom:0px;padding-left:30px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,simsun;vertical-align:baseline;list-style-position:outside;color:rgb(93,255);"><li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">极其灵活的定制化爬取。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">社区人数多、文档完善。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">URL去重采用布隆过滤器方案。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">可以处理不完整的HTML，Scrapy已经提供了selectors（一个在lxml的基础上提供了更高级的接口），可以高效地处理不完整的HTML代码。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">缺点：<ul class="list-paddingleft-2" style="margin-top:1em;margin-bottom:0px;padding-left:30px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">不支持分布式部署。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">原生不支持抓取JavaScript的页面。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">全命令行操作，对用户不友好，需要一定学习周期。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">结论<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">篇幅有限，就先选择这三个最有代表性的框架进行PK。他们都有远超别人的优点，比如：Nutch天生的搜索引擎解决方案、Pyspider产品级的WebUI、Scrapy最灵活的定制化爬取。也都各自致命的缺点，比如Scrapy不支持分布式部署，Pyspider不够灵活，Nutch和搜索绑定。究竟该怎么选择呢？<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">我们的目标是做纯粹的爬虫，不是搜索引擎，所以先把Nutch排除掉，剩下人性化的Pyspider和高可定制的Scrapy。Scrapy的灵活性几乎能够让我们完成任何苛刻的抓取需求，它的“难用”也让我们不知不觉的研究爬虫技术。现在还不是享受Pyspider的时候，目前的当务之急是打好基础，应该学习最接近爬虫本质的框架，了解它的原理，所以把Pyspider也排除掉。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">最终，理性的从个人的需求角度对比，还是Scrapy胜出！其实Scrapy还有更多优点：<ul class="list-paddingleft-2" style="margin-top:1em;margin-bottom:0px;padding-left:30px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">HTML,XML源数据选择及提取的原生支持。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">提供了一系列在spider之间共享的可复用的过滤器(即 Item Loaders)，对智能处理爬取数据提供了内置支持。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">通过 feed导出 提供了多格式(JSON、CSV、XML)，多存储后端(FTP、S3、本地文件系统)的内置支持。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">提供了media pipeline，可以 自动下载 爬取到的数据中的图片(或者其他资源)。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">高扩展性。您可以通过使用 signals ，设计好的API(中间件,extensions,pipelines)来定制实现您的功能。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">内置的中间件及扩展为下列功能提供了支持:<ul class="list-paddingleft-2" style="margin-top:1em;margin-bottom:0px;padding-left:30px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">cookies and session 处理<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">HTTP 压缩<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">HTTP 认证<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">HTTP 缓存<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">user-agent模拟<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">robots.txt<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">爬取深度限制<ul class="list-paddingleft-2" style="margin-top:1em;margin-bottom:0px;padding-left:30px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">针对非英语语系中不标准或者错误的编码声明,提供了自动检测以及健壮的编码支持。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">支持根据模板生成爬虫。在加速爬虫创建的同时，保持在大型项目中的代码更为一致。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">针对多爬虫下性能评估、失败检测，提供了可扩展的 状态收集工具 。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">提供 交互式shell终端,为您测试XPath表达式，编写和调试爬虫提供了极大的方便。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">提供 System service,简化在生产环境的部署及运行。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">内置 Telnet终端 ，通过在Scrapy进程中钩入Python终端，使您可以查看并且调试爬虫。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">Logging 为您在爬取过程中捕捉错误提供了方便。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">支持 Sitemaps 爬取。<li style="margin-top:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;clear:both;"><p style="margin-bottom:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">具有缓存的DNS解析器。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">下一步<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">吹了半天的Scrapy，时间也到了，如果大家能够喜欢上它，学习的效率一定会成倍提升！下次我会为大家带来满满的干货，并完成更具挑战性的爬虫任务，我们下期再见！<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">原文链接：http://www.cnblogs.com/tuohai666/p/8861422.html<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">欢迎关注我的博客：https://home.cnblogs.com/u/sm123456/ （编辑：李大同）
【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!
|
