python – 当使用futures.ProcessPoolExecutor而不是future.Thre
发布时间:2020-12-20 13:50:21 所属栏目:Python 来源:网络整理
导读:我正在使用此代码来刮取API: submissions = get_submissions(1)with futures.ProcessPoolExecutor(max_workers=4) as executor:#or using this: with futures.ThreadPoolExecutor(max_workers=4) as executor: for s in executor.map(map_func,submissions)
我正在使用此代码来刮取API:
submissions = get_submissions(1) with futures.ProcessPoolExecutor(max_workers=4) as executor: #or using this: with futures.ThreadPoolExecutor(max_workers=4) as executor: for s in executor.map(map_func,submissions): collection_front.update({"time_recorded":time_recorded},{'$push':{"thread_list":s}},upsert=True) 它可以很好地/快速地使用线程,但是当我尝试使用进程时,我得到一个完整的队列并且出现此错误: File "/usr/local/lib/python3.4/dist-packages/praw/objects.py",line 82,in __getattr__ if not self.has_fetched: RuntimeError: maximum recursion depth exceeded Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python3.4/threading.py",line 920,in _bootstrap_inner self.run() File "/usr/lib/python3.4/threading.py",line 868,in run self._target(*self._args,**self._kwargs) File "/usr/lib/python3.4/concurrent/futures/process.py",line 251,in _queue_management_worker shutdown_worker() File "/usr/lib/python3.4/concurrent/futures/process.py",line 209,in shutdown_worker call_queue.put_nowait(None) File "/usr/lib/python3.4/multiprocessing/queues.py",line 131,in put_nowait return self.put(obj,False) File "/usr/lib/python3.4/multiprocessing/queues.py",in put raise Full queue.Full Traceback (most recent call last): File "reddit_proceses.py",line 64,in <module> for s in executor.map(map_func,submissions): File "/usr/lib/python3.4/concurrent/futures/_base.py",line 549,in result_iterator yield future.result() File "/usr/lib/python3.4/concurrent/futures/_base.py",line 402,in result return self.__get_result() File "/usr/lib/python3.4/concurrent/futures/_base.py",line 354,in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 请注意,最初这些过程对于小数据检索来说效果很好而且非常快,但现在它们根本不起作用.这是一个错误还是正在发生的事情,PRAW对象会导致进程的递归错误而不是Threads? 解决方法
我有一个类似的问题从线程转移到进程只有我使用executor.submit.我认为这可能与您遇到的问题相同,但我无法确定,因为我不知道您的代码在什么上下文中运行.
在我的情况下发生的事情是:我将我的代码作为脚本运行,并且我没有使用始终建议的if __name__ ==“__ main__”:看起来当使用执行程序运行新进程时,python会加载.py文件并运行submit中指定的函数.因为它加载文件,所以主文件上存在的代码(不在函数内部或上面的句子中)运行,因此每个进程将再次运行一个具有无限递归的新进程. 看起来线程不会发生这种情况. (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |