Python子进程并行
发布时间:2020-12-20 10:32:01 所属栏目:Python 来源:网络整理
导读:我希望能够并行运行多个进程,并且能够随时使用stdout.我该怎么办?我是否需要为每个subprocess.Popen()调用运行线程,什么? 解决方法 你可以在一个线程中完成. 假设您有一个随机打印行的脚本: #!/usr/bin/env python#file: child.pyimport osimport randomi
我希望能够并行运行多个进程,并且能够随时使用stdout.我该怎么办?我是否需要为每个subprocess.Popen()调用运行线程,什么?
解决方法
你可以在一个线程中完成.
假设您有一个随机打印行的脚本: #!/usr/bin/env python #file: child.py import os import random import sys import time for i in range(10): print("%2d %s %s" % (int(sys.argv[1]),os.getpid(),i)) sys.stdout.flush() time.sleep(random.random()) 并且您希望在输出可用后立即收集输出,您可以在POSIX系统上使用 #!/usr/bin/env python from __future__ import print_function from select import select from subprocess import Popen,PIPE # start several subprocesses processes = [Popen(['./child.py',str(i)],stdout=PIPE,bufsize=1,close_fds=True,universal_newlines=True) for i in range(5)] # read output timeout = 0.1 # seconds while processes: # remove finished processes from the list (O(N**2)) for p in processes[:]: if p.poll() is not None: # process ended print(p.stdout.read(),end='') # read the rest p.stdout.close() processes.remove(p) # wait until there is something to read rlist = select([p.stdout for p in processes],[],timeout)[0] # read a line from each process that has output ready for f in rlist: print(f.readline(),end='') #NOTE: it can block 更便携的解决方案(适用于Windows,Linux,OSX)可以为每个进程使用读取器线程,请参阅Non-blocking read on a subprocess.PIPE in python. 这是基于 #!/usr/bin/env python from __future__ import print_function import io import os import sys from subprocess import Popen ON_POSIX = 'posix' in sys.builtin_module_names # create a pipe to get data input_fd,output_fd = os.pipe() # start several subprocesses processes = [Popen([sys.executable,'child.py',stdout=output_fd,close_fds=ON_POSIX) # close input_fd in children for i in range(5)] os.close(output_fd) # close unused end of the pipe # read output line by line as soon as it is available with io.open(input_fd,'r',buffering=1) as file: for line in file: print(line,end='') # for p in processes: p.wait() (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |