multiprocessing.Pool在Linux / Python2.7上的terminate()之后产
发布时间：2020-12-13 23:56:13 所属栏目：Linux 来源：网络整理
导读：我有一个可执行文件,我需要经常运行,具有不同的参数.为此,我使用多处理模块编写了一个小的 Python(2.7)包装器,遵循给定的模式 here. 我的代码看起来像这样： try: logging.info("starting pool runs") pool.map(run_nlin,params) pool.close() except Keyboa
|
我有一个可执行文件,我需要经常运行,具有不同的参数.为此,我使用多处理模块编写了一个小的
Python(2.7)包装器,遵循给定的模式
here.
我的代码看起来像这样： try:
logging.info("starting pool runs")
pool.map(run_nlin,params)
pool.close()
except KeyboardInterrupt:
logging.info("^C pressed")
pool.terminate()
except Exception,e:
logging.info("exception caught: ",e)
pool.terminate()
finally:
time.sleep(5)
pool.join()
logging.info("done")
我的工作人员在这里： class KeyboardInterruptError(Exception): pass
def run_nlin((path_config,path_log,path_nlin,update_method)):
try:
with open(path_log,"w") as log_:
cmdline = [path_nlin,path_config]
if update_method:
cmdline += [update_method,]
sp.call(cmdline,stdout=log_,stderr=log_)
except KeyboardInterrupt:
time.sleep(5)
raise KeyboardInterruptError()
except:
raise
path_config是二进制程序的配置文件的路径;在那里,例如运行程序的日期. 当我启动包装器时,一切看起来都很好.但是,当我按下^ C时,包装器脚本似乎在终止之前从池中启动了一个额外的numproc进程.例如,当我在1-10天启动脚本时,我可以在ps aux输出中看到二进制程序的两个实例正在运行(通常是第1天和第3天).现在,当我按^ C时,包装器脚本退出,第1天和第3天的二进制程序消失了,但是有新的二进制程序在第5天和第7天运行. 所以对我而言,似乎Pool在最终死亡之前启动了另一个numproc进程. 任何想法在这里发生了什么,我能做些什么呢？ 解决方法
在
this page,多处理模块的作者Jesse Noller表明,处理KeyboardInterrupt的正确方法是让子进程返回 – 而不是重新加载异常.这允许主进程终止池.
但是,如下面的代码所示,直到运行了pool.map生成的所有任务之后,主进程才会到达除KeyboardInterrupt块之外的其他内容.这就是为什么(我相信)你在按下Ctrl-C后看到对你的工作函数run_nlin的额外调用. 一种可能的解决方法是,如果已设置multiprocessing.Event,则测试所有工作器函数.如果事件已经设定,那么让工人提早纾困,否则,继续进行长时间的计算. import logging
import multiprocessing as mp
import time
logger = mp.log_to_stderr(logging.WARNING)
def worker(x):
try:
if not terminating.is_set():
logger.warn("Running worker({x!r})".format(x = x))
time.sleep(3)
else:
logger.warn("got the message... we're terminating!")
except KeyboardInterrupt:
logger.warn("terminating is set")
terminating.set()
return x
def initializer(terminating_):
# This places terminating in the global namespace of the worker subprocesses.
# This allows the worker function to access `terminating` even though it is
# not passed as an argument to the function.
global terminating
terminating = terminating_
def main():
terminating = mp.Event()
result = []
pool = mp.Pool(initializer=initializer,initargs=(terminating,))
params = range(12)
try:
logger.warn("starting pool runs")
result = pool.map(worker,params)
pool.close()
except KeyboardInterrupt:
logger.warn("^C pressed")
pool.terminate()
finally:
pool.join()
logger.warn('done: {r}'.format(r = result))
if __name__ == '__main__':
main()
运行脚本会产生： % test.py [WARNING/MainProcess] starting pool runs [WARNING/PoolWorker-1] Running worker(0) [WARNING/PoolWorker-2] Running worker(1) [WARNING/PoolWorker-3] Running worker(2) [WARNING/PoolWorker-4] Running worker(3) 按下Ctrl-C;每个工人都设置终止事件.我们真的只需要一个设置它,但这可以工作尽管效率低下. C-c C-c[WARNING/PoolWorker-4] terminating is set [WARNING/PoolWorker-2] terminating is set [WARNING/PoolWorker-3] terminating is set [WARNING/PoolWorker-1] terminating is set 现在运行pool.map排队的所有其他任务： [WARNING/PoolWorker-4] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-1] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-4] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-1] got the message... we're terminating! [WARNING/PoolWorker-3] got the message... we're terminating! 最后,主进程到达除KeyboardInterrupt块之外. [WARNING/MainProcess] ^C pressed [WARNING/MainProcess] done: [] （编辑：李大同） 【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容! |