加入收藏 | 设为首页 | 会员中心 | 我要投稿 李大同 (https://www.lidatong.com.cn/)- 科技、建站、经验、云计算、5G、大数据,站长网!
当前位置: 首页 > 编程开发 > Python > 正文

python – 在Windows服务器上使用scrapy进行线程阻塞

发布时间:2020-12-20 13:40:52 所属栏目:Python 来源:网络整理
导读:我在 Windows服务器上运行以下代码时出错 scrapy shell "http://www.yahoo.com" 但我对没有重定向到https的网站没有问题 我认为问题在于线程阻塞.请有人帮帮我 这是错误消息 C:Documents and Settingsmahyarscrapy shell "http://www.yahoo.com"2014-03-03
我在 Windows服务器上运行以下代码时出错

scrapy shell "http://www.yahoo.com"

但我对没有重定向到https的网站没有问题
我认为问题在于线程阻塞.请有人帮帮我

这是错误消息

C:Documents and Settingsmahyar>scrapy shell "http://www.yahoo.com"
2014-03-03 15:49:38-0600 [scrapy] INFO: Scrapy 0.22.2 started (bot: scrapybot)
2014-03-03 15:49:38-0600 [scrapy] INFO: Optional features available: ssl,http11
2014-03-03 15:49:38-0600 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL
': 0}
2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled extensions: TelnetConsole,Close
Spider,WebService,CoreStats,SpiderState
2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled downloader middlewares: HttpAuth
Middleware,DownloadTimeoutMiddleware,UserAgentMiddleware,RetryMiddleware,Def
aultHeadersMiddleware,MetaRefreshMiddleware,HttpCompressionMiddleware,Redirec
tMiddleware,CookiesMiddleware,ChunkedTransferMiddleware,DownloaderStats
2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled spider middlewares: HttpErrorMid
dleware,OffsiteMiddleware,RefererMiddleware,UrlLengthMiddleware,DepthMiddlew
are
2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled item pipelines:
2014-03-03 15:49:38-0600 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:602
3
2014-03-03 15:49:38-0600 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-03-03 15:49:38-0600 [default] INFO: Spider opened
2014-03-03 15:49:38-0600 [default] DEBUG: Redirecting (301) to <GET https://www.
yahoo.com/> from <GET http://www.yahoo.com>
Traceback (most recent call last):
  File "c:Python27librunpy.py",line 162,in _run_module_as_main
    "__main__",fname,loader,pkg_name)
  File "c:Python27librunpy.py",line 72,in _run_code
    exec code in run_globals
  File "c:Python27libsite-packagesscrapycmdline.py",line 168,in <module>
    execute()
  File "c:Python27libsite-packagesscrapycmdline.py",line 143,in execute
    _run_print_help(parser,_run_command,cmd,args,opts)
  File "c:Python27libsite-packagesscrapycmdline.py",line 89,in _run_print
_help
    func(*a,**kw)
  File "c:Python27libsite-packagesscrapycmdline.py",line 150,in _run_comm
and
    cmd.run(args,opts)
  File "c:Python27libsite-packagesscrapycommandsshell.py",line 50,in run

    shell.start(url=url,spider=spider)
  File "c:Python27libsite-packagesscrapyshell.py",line 45,in start
    self.fetch(url,spider)
  File "c:Python27libsite-packagesscrapyshell.py",line 90,in fetch
    reactor,self._schedule,request,spider)
  File "c:Python27libsite-packagestwistedinternetthreads.py",line 122,in
 blockingCallFromThread
    result.raiseException()
  File "<string>",line 2,in raiseException
OverflowError: integer 2147486719 does not fit '32-bit int'

解决方法

看起来您正在运行32位版本的Windows,而Scrapy需要64位操作系统.

(编辑:李大同)

【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容!

    推荐文章
      热点阅读