python – 在Windows服务器上使用scrapy进行线程阻塞
发布时间:2020-12-20 13:40:52 所属栏目:Python 来源:网络整理
导读:我在 Windows服务器上运行以下代码时出错 scrapy shell "http://www.yahoo.com" 但我对没有重定向到https的网站没有问题 我认为问题在于线程阻塞.请有人帮帮我 这是错误消息 C:Documents and Settingsmahyarscrapy shell "http://www.yahoo.com"2014-03-03
我在
Windows服务器上运行以下代码时出错
scrapy shell "http://www.yahoo.com" 但我对没有重定向到https的网站没有问题 这是错误消息 C:Documents and Settingsmahyar>scrapy shell "http://www.yahoo.com" 2014-03-03 15:49:38-0600 [scrapy] INFO: Scrapy 0.22.2 started (bot: scrapybot) 2014-03-03 15:49:38-0600 [scrapy] INFO: Optional features available: ssl,http11 2014-03-03 15:49:38-0600 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL ': 0} 2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled extensions: TelnetConsole,Close Spider,WebService,CoreStats,SpiderState 2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled downloader middlewares: HttpAuth Middleware,DownloadTimeoutMiddleware,UserAgentMiddleware,RetryMiddleware,Def aultHeadersMiddleware,MetaRefreshMiddleware,HttpCompressionMiddleware,Redirec tMiddleware,CookiesMiddleware,ChunkedTransferMiddleware,DownloaderStats 2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled spider middlewares: HttpErrorMid dleware,OffsiteMiddleware,RefererMiddleware,UrlLengthMiddleware,DepthMiddlew are 2014-03-03 15:49:38-0600 [scrapy] INFO: Enabled item pipelines: 2014-03-03 15:49:38-0600 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:602 3 2014-03-03 15:49:38-0600 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080 2014-03-03 15:49:38-0600 [default] INFO: Spider opened 2014-03-03 15:49:38-0600 [default] DEBUG: Redirecting (301) to <GET https://www. yahoo.com/> from <GET http://www.yahoo.com> Traceback (most recent call last): File "c:Python27librunpy.py",line 162,in _run_module_as_main "__main__",fname,loader,pkg_name) File "c:Python27librunpy.py",line 72,in _run_code exec code in run_globals File "c:Python27libsite-packagesscrapycmdline.py",line 168,in <module> execute() File "c:Python27libsite-packagesscrapycmdline.py",line 143,in execute _run_print_help(parser,_run_command,cmd,args,opts) File "c:Python27libsite-packagesscrapycmdline.py",line 89,in _run_print _help func(*a,**kw) File "c:Python27libsite-packagesscrapycmdline.py",line 150,in _run_comm and cmd.run(args,opts) File "c:Python27libsite-packagesscrapycommandsshell.py",line 50,in run shell.start(url=url,spider=spider) File "c:Python27libsite-packagesscrapyshell.py",line 45,in start self.fetch(url,spider) File "c:Python27libsite-packagesscrapyshell.py",line 90,in fetch reactor,self._schedule,request,spider) File "c:Python27libsite-packagestwistedinternetthreads.py",line 122,in blockingCallFromThread result.raiseException() File "<string>",line 2,in raiseException OverflowError: integer 2147486719 does not fit '32-bit int' 解决方法
看起来您正在运行32位版本的Windows,而Scrapy需要64位操作系统.
(编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |