|
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,Arial,'Hiragino Sans GB','Microsoft YaHei',simsun;vertical-align:baseline;color:rgb(93,93,93);background-color:rgb(255,255,255);"> 
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">只有MD5的超链接可以点，点进去看看，<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">我写这个的爬虫的目的是下载样本，只关心样本的下载地址，并不关心其他的信息。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">现在我们可以理清下思路：<pre style="margin-bottom:0px;padding-right:0px;padding-left:0px;border:0px;font-size:18px;line-height:inherit;font-family:inherit;vertical-align:baseline;color:rgb(93,255);"><pre class="source-code" style="margin-bottom:0px;padding:15px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:rgb(243,244,245);"><code style="margin:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">获取每一页的网页源码
解析当前页的每一个<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;color:rgb(0,134,179);">md5对应的详细信息链接
在详细信息页面解析下载地址。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">提取规则<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">先从第一页开始爬起<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">scrapy支持xss选择器和css选择器，一般是用哪个比较方便就用哪个。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">对应链接的xpath选择器为?<code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">//td/a/@href?,<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">下一页的链接的xpath选择器为?<code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">//a[contains(text(),'Next')]/@href<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">登陆<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">之前写爬虫的时候，是可以直接通过post登陆的，而我开始写这篇文章的时候开始加上了google的ReCaptcha验证码。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">只能换种方式使用selenium来登陆Malwr.得到登陆后的cookie后，我们就可以用这个cookie开始下载样本了。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">FilesPipeline是从file_urls中拿到下载地址然后开始下载文件，因此我们还需要Item.py定义一个file_urls.<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">解析下载地址，将解析到的下载链接列表放入item的file_urls,返回item。pipeline会file_urls拿到链接开始调用get_media_requests下载文件<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
（编辑：李大同）
【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!
|
