python – Pandas：使用read_csv时如何获取读取行的状态？

发布时间：2020-12-16 22:38:45 所属栏目：Python 来源：网络整理

导读：我正在使用pandas和read_csv方法加载一个非常庞大的csv文件,如1000万条记录,我想知道是否有办法显示该加载的进度,如： 100,000 lines read150,000 lines read 谢谢. 最佳答案要显示这样的进度： Completed 1 %Completed 2 % ... Completed 99 % Completed 1

我正在使用pandas和read_csv方法加载一个非常庞大的csv文件,如1000万条记录,我想知道是否有办法显示该加载的进度,如：

100,000 lines read
150,000 lines read

谢谢.

最佳答案

要显示这样的进度：

Completed 1 %
Completed 2 % 
... 
Completed 99 % 
Completed 100 %

你可以试试这个：

import os,pandas
filename = "VeryLong.csv"
lines_number = sum(1 for line in open(filename))
lines_in_chunk = 500 # I don't know what size is better
counter = 0
completed = 0
reader = pandas.read_csv(filename,chunksize=lines_in_chunk)
for chunk in reader:
    # < ... reading the chunk somehow... >
    # showing progress:
    counter += lines_in_chunk
    new_completed = int(round(float(counter)/lines_number * 100))
    if (new_completed > completed): 
        completed = new_completed
        print "Completed",completed,"%"

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!