加入收藏 | 设为首页 | 会员中心 | 我要投稿 李大同 (https://www.lidatong.com.cn/)- 科技、建站、经验、云计算、5G、大数据,站长网!
当前位置: 首页 > 编程开发 > Python > 正文

python – 当使用futures.ProcessPoolExecutor而不是future.Thre

发布时间:2020-12-20 13:50:21 所属栏目:Python 来源:网络整理
导读:我正在使用此代码来刮取API: submissions = get_submissions(1)with futures.ProcessPoolExecutor(max_workers=4) as executor:#or using this: with futures.ThreadPoolExecutor(max_workers=4) as executor: for s in executor.map(map_func,submissions)
我正在使用此代码来刮取API:

submissions = get_submissions(1)
with futures.ProcessPoolExecutor(max_workers=4) as executor:
#or using this: with futures.ThreadPoolExecutor(max_workers=4) as executor:
    for s in executor.map(map_func,submissions):
        collection_front.update({"time_recorded":time_recorded},{'$push':{"thread_list":s}},upsert=True)

它可以很好地/快速地使用线程,但是当我尝试使用进程时,我得到一个完整的队列并且出现此错误:

File "/usr/local/lib/python3.4/dist-packages/praw/objects.py",line 82,in __getattr__
    if not self.has_fetched:
RuntimeError: maximum recursion depth exceeded
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py",line 920,in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py",line 868,in run
    self._target(*self._args,**self._kwargs)
  File "/usr/lib/python3.4/concurrent/futures/process.py",line 251,in _queue_management_worker
    shutdown_worker()
  File "/usr/lib/python3.4/concurrent/futures/process.py",line 209,in shutdown_worker
    call_queue.put_nowait(None)
  File "/usr/lib/python3.4/multiprocessing/queues.py",line 131,in put_nowait
    return self.put(obj,False)
  File "/usr/lib/python3.4/multiprocessing/queues.py",in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "reddit_proceses.py",line 64,in <module>
    for s in executor.map(map_func,submissions):
  File "/usr/lib/python3.4/concurrent/futures/_base.py",line 549,in result_iterator
    yield future.result()
  File "/usr/lib/python3.4/concurrent/futures/_base.py",line 402,in result
    return self.__get_result()
  File "/usr/lib/python3.4/concurrent/futures/_base.py",line 354,in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

请注意,最初这些过程对于小数据检索来说效果很好而且非常快,但现在它们根本不起作用.这是一个错误还是正在发生的事情,PRAW对象会导致进程的递归错误而不是Threads?

解决方法

我有一个类似的问题从线程转移到进程只有我使用executor.submit.我认为这可能与您遇到的问题相同,但我无法确定,因为我不知道您的代码在什么上下文中运行.

在我的情况下发生的事情是:我将我的代码作为脚本运行,并且我没有使用始终建议的if __name__ ==“__ main__”:看起来当使用执行程序运行新进程时,python会加载.py文件并运行submit中指定的函数.因为它加载文件,所以主文件上存在的代码(不在函数内部或上面的句子中)运行,因此每个进程将再次运行一个具有无限递归的新进程.

看起来线程不会发生这种情况.

(编辑:李大同)

【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容!

    推荐文章
      热点阅读