短
我的python程序占用了比预期更多的内存或由内存分析工具返回 . 我需要一个策略来查找内存泄漏并修复它 .
详细
我在64位Linux机器上运行python3脚本 . 几乎所有代码都捆绑在一个对象中:
obj = MyObject(*myArguments)
result = obj.doSomething()
print(result)
在创建 obj
期间,程序将读取大小为ca.的文本文件 . 100MB . 由于我以多种方式保存信息,我希望整个对象占用几个hundret MB内存 .
实际上,从包pympler中使用 asizeof.asized(obj)
测量其大小会返回大约123MB . 然而, top
告诉我,我的程序占据了大约 1GB 内存 .
我知道方法中的局部变量会占用更多的RAM . 但是,查看我的代码,我发现这些局部变量都不会那么大 . 我再次使用 asizeof.asized
再次检查了这一点 .
我不担心脚本需要1GB的内存 . 但是,我并行执行了一些方法(在12个方面):
class MyObject()
def doSomething(arg):
# do something
def myParallelMethod(args)
with sharedmem.MapReduce() as pool:
result = pool.map(self.doSomething, args)
return result
这使得总内存使用量变为 8GB ,即使我将所有大对象放在共享内存中:
self.myLargeNumPyArray = sharedmem.copy(self.myLargeNumPyArray)
我向测试程序保证内存真的是共享的 .
检查 asizeof
,我在每个子进程中获得了
-
asizeof.asized(self)
是1MB(即比"original"对象小得多 - 可能是由于共享内存,不计算加倍) -
asizeof.asized(myOneAndOnlyBigLocalVariable)
是230MB .
总而言之,我的程序应该占用不超过123MB 12 * 230MB = 2.8GB << 8GB . 那么,为什么程序需要这么多内存呢?
一种解释可能是当程序并行运行时,我的对象中存在一些被隐藏的部分(垃圾?) .
Does anyone know a strategy to find out, where the memory leak is? How could I fix it?
我已经阅读了许多关于内存分析的线程,例如: Profiling memory in python 3,Is there any working memory profiler for Python3,Which Python memory profiler is recommended?或How do I profile memory usage in Python?,但所有推荐的工具都没有解释内存使用情况 .
更新
我被要求提供代码的最小示例 . 下面的代码显示了并行部分中的内存消耗与原始代码相同的问题 . 我已经找到了我的代码的非并行部分的问题,这是我有一个大的numpy数组,数据类型 object
作为对象变量 . 由于此数据类型,数组不能放入共享内存, asized
只返回浅的大小 . 感谢@ user2357112,帮助我解决这个问题!
因此,我想集中讨论并行部分中的问题:在方法 singleSourceShortestPaths
(下面标有注释)中将值插入 queue
会将内存消耗从大约1.5GB更改为 10GB . 对于如何解释这种行为有什么想法吗?
import numpy as np
from heapdict import heapdict
from pympler import asizeof
import sharedmem
class RoadNetwork():
strType = "|S10"
def __init__(self):
vertexNo = 1000000
self.edges = np.zeros(1500000, dtype = {"names":["ID", "from_to", "from_to_original", "cost", "inspection", "spot"],
'formats':[self.strType, '2int', '2'+self.strType, "double", "3bool", "2int", "2int"]})
self.edges["ID"] = np.arange(self.edges.size)
self.edges["from_to_original"][:vertexNo, 0] = np.arange(vertexNo)
self.edges["from_to_original"][vertexNo:, 0] = np.random.randint(0, vertexNo, self.edges.size-vertexNo)
self.edges["from_to_original"][:,1] = np.random.randint(0, vertexNo, self.edges.size)
vertexIDs = np.unique(self.edges["from_to_original"])
self.vertices = np.zeros(vertexIDs.size, {"names":["ID", "type", "lakeID"],
'formats':[self.strType, 'int', self.strType]})
def singleSourceShortestPaths(self, sourceIndex):
vertexData = np.zeros(self.vertices.size, dtype={"names":["predecessor", "edge", "cost"],
'formats':['int', "2int", "double"]})
queue = np.zeros((self.vertices.size, 2), dtype=np.double)
#Crucual line!! Commetning this decreases memory usage by 7GB in the parallel part
queue[:,0] = np.arange(self.vertices.size)
queue = heapdict(queue)
print("self in singleSourceShortestPaths", asizeof.asized(self))
print("queue in singleSourceShortestPaths", asizeof.asized(queue))
print("vertexData in singleSourceShortestPaths", asizeof.asized(vertexData))
# do stuff (in my real program Dijkstra's algorithm would follow)
# I inserted this lines as an ugly version for 'wait()' to
# give me enough time to measure the memory consumption in 'top'
for i in range(10000000000):
pass
return vertexData
def determineFlowInformation(self):
print("self in determineFlowInformation", asizeof.asized(self))
f = lambda i: self.singleSourceShortestPaths(i)
self.parmap(f, range(30))
def parmap(self, f, argList):
"""
Executes f(arg) for arg in argList in parallel
returns a list of the results in the same order as the
arguments, invalid results (None) are ignored
"""
self.__make_np_arrays_sharable()
with sharedmem.MapReduce() as pool:
results, to_do_list = zip(*pool.map(f, argList))
return results
def __make_np_arrays_sharable(self):
"""
Replaces all numpy array object variables,
which should have the same
behaviour / properties as the numpy array
"""
varDict = self.__dict__
for key, var in varDict.items():
if type(var) is np.ndarray:
varDict[key] = sharedmem.copy(var)
if __name__ == '__main__':
network = RoadNetwork()
print(asizeof.asized(network, detail=1))
for key, var in network.__dict__.items():
print(key, asizeof.asized(var))
network.determineFlowInformation()