批量下载文件

以下代码由PHP站长网 52php.cn收集自互联网

现在PHP站长网小编把它分享给大家，仅供参考

#!/usr/bin/env python
# -*- coding:utf-8 -*-

from gevent import monkey
monkey.patch_all()
from gevent.pool import Pool
import requests
import sys
import os


def download(url):
    chrome = 'Mozilla/5.0 (X11; Linux i86_64) AppleWebKit/537.36 ' + 
        '(KHTML,like Gecko) Chrome/41.0.2272.101 Safari/537.36'
    headers = {'User-Agent': chrome}
    filename = url.split('/')[-1].strip()
    r = requests.get(url.strip(),headers=headers,stream=True)
    with open(filename,'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
                f.write(chunk)
                f.flush()
    print filename,"is ok"


def removeLine(key,filename):
    os.system('sed -i /%s/d %s' % (key,filename))


if __name__ == "__main__":
    if len(sys.argv) == 2:
        filename = sys.argv[1]
        f = open(filename,"r")
        p = Pool(4)
        for line in f.readlines():
            if line:
                p.spawn(download,line.strip())
                key = line.split('/')[-1].strip()
                removeLine(key,filename)
        f.close()
        p.join()
    else:
        print 'Usage: python %s urls.txt' % sys.argv[0]

以上内容由PHP站长网【52php.cn】收集整理供大家参考研究

如果以上内容对您有帮助，欢迎收藏、点赞、推荐、分享。

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!