python – 从pandas数据帧中选择时的内存优化

出于这个问题的目的,我简化了数据框并使用了假数据.我的代码和内存占用量如下所示,

from memory_profiler import profile
import pandas as pnd
import random


@profile
def main():
    cols = [chr(i) for i in range(65,91)]
    d = {}
    n = 1000000
    for c in cols:
        d[c] = [random.randint(0,100) for i in range(n)]
    df = pnd.DataFrame(d)
    items = ['A','F','G']
    a = df[items]
    b = a.max(axis=0)


if __name__ == "__main__":
    main()


Line #    Mem usage    Increment   Line Contents
================================================
     6     42.3 MiB      0.0 MiB   @profile
     7                             def main():
     8     42.3 MiB      0.0 MiB       cols = [chr(i) for i in range(65,91)]
     9     42.3 MiB      0.0 MiB       d = {}
    10     42.3 MiB      0.0 MiB       n = 1000000
    11    240.6 MiB    198.3 MiB       for c in cols:
    12    240.6 MiB      0.0 MiB           d[c] = [random.randint(0,100) for i in range(n)]
    13    446.7 MiB    206.1 MiB       df = pnd.DataFrame(d)
    14    446.7 MiB      0.0 MiB       items = ['A','G']
    15    469.7 MiB     23.1 MiB       a = df[items]
    16    469.8 MiB      0.1 MiB       b = a.max(axis=0)

在上面的操作中,似乎df [items]占用了23MB的内存.我推测这是因为它正在复制df并将其放在’a’中.

有没有办法在选择列时摆脱这种内存开销？