python – Threads中的subprocess.Popen
发布时间:2020-12-20 11:26:15 所属栏目:Python 来源:网络整理
导读:我有很多文件(超过4000个),我想同时加载到PostgreSQL中.我已将它们分成4个不同的文件列表,我想要一个线程迭代加载数据的每个列表. 我遇到的问题是我使用os.system来调用加载程序但这会阻止其他线程同时运行.如果我使用subprocess.Popen然后它们同时运行但是
我有很多文件(超过4000个),我想同时加载到PostgreSQL中.我已将它们分成4个不同的文件列表,我想要一个线程迭代加载数据的每个列表.
我遇到的问题是我使用os.system来调用加载程序但这会阻止其他线程同时运行.如果我使用subprocess.Popen然后它们同时运行但是线程认为它们已经完成了execeuting,所以移动到我的脚本的下一部分. 我这样做是对的吗?或者是否有更好的方法从线程内调用子进程. def thread1Load(self,thread1fileList): connectionstring = settings.connectionstring postgreshost = settings.postgreshost postgresdatabase = settings.postgresdatabase postgresport = settings.postgresport postgresusername = settings.postgresusername postgrespassword = settings.postgrespassword tablename = None encoding = None connection = psycopg2.connect(connectionstring) for filename in thread1fileList: load_cmd = #load command run = subprocess.Popen(load_cmd,shell=True) print "finished loading thread 1" def thread2Load(self,thread2fileList): connectionstring = settings.connectionstring postgreshost = settings.postgreshost postgresdatabase = settings.postgresdatabase postgresport = settings.postgresport postgresusername = settings.postgresusername postgrespassword = settings.postgrespassword tablename = None connection = psycopg2.connect(connectionstring) for filename in thread2fileList: load_cmd = #load command run = subprocess.Popen(load_cmd,shell=True) print "finished loading thread 2" def thread3Load(self,thread3fileList): connectionstring = settings.connectionstring postgreshost = settings.postgreshost postgresdatabase = settings.postgresdatabase postgresport = settings.postgresport postgresusername = settings.postgresusername postgrespassword = settings.postgrespassword tablename = None connection = psycopg2.connect(connectionstring) for shapefilename in thread3fileList: load_cmd = #load command run = subprocess.Popen(load_cmd,shell=True) print "finished loading thread 3" def thread4Load(self,thread4fileList): connectionstring = settings.connectionstring postgreshost = settings.postgreshost postgresdatabase = settings.postgresdatabase postgresport = settings.postgresport postgresusername = settings.postgresusername postgrespassword = settings.postgrespassword tablename = None connection = psycopg2.connect(connectionstring) for filename in thread4fileList: load_cmd = #load command run = subprocess.Popen(load_cmd,shell=True) print "finished loading thread 4" def finishUp(self): print 'finishing up' def main(): load = Loader() thread1 = threading.Thread(target=(load.thread1Load),args=(thread1fileList,)) thread2 = threading.Thread(target=(load.thread2Load),args=(thread2fileList,)) thread3 = threading.Thread(target=(load.thread3Load),args=(thread3fileList,)) thread4 = threading.Thread(target=(load.thread4Load),args=(thread4fileList,)) threads = [thread1,thread2,thread3,thread4] for thread in threads: thread.start() thread.join() load.finishUp(connectionstring) if __name__ == '__main__': main() 解决方法
>
Don’t repeat yourself.一个threadLoad方法就足够了.这样,如果您需要修改方法中的某些内容,则无需在4个不同的位置进行相同的修改.
>使用run.communicate()阻止,直到子进程完成. >这将启动一个线程,然后阻塞直到该线程完成,然后 启动另一个线程等: for thread in threads: thread.start() thread.join() 相反,首先启动所有线程,然后加入所有线程: for thread in threads: thread.start() for thread in threads: thread.join() import subprocess import threading class Loader(object): def threadLoad(self,threadfileList): connectionstring = settings.connectionstring ... connection = psycopg2.connect(connectionstring) for filename in threadfileList: load_cmd = # load command run = subprocess.Popen(load_cmd,shell=True) # block until subprocess is done run.communicate() name = threading.current_thread().name print "finished loading {n}".format(n=name) def finishUp(self): print 'finishing up' def main(): load = Loader() threads = [threading.Thread(target=load.threadLoad,args=(fileList,)) for fileList in (thread1fileList,thread2fileList,thread3fileList,thread4fileList)] for thread in threads: thread.start() for thread in threads: thread.join() load.finishUp(connectionstring) if __name__ == '__main__': main() (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |