拆分大量数据并为所有数据块循环相同的代码

2 years ago

首先，你可以将这些东西分成长度为 n 的组

dfgroups = [df[x:x+n] for x in range(0, len(df), n)]

将 20000 替换为 n ，你将得到每个最多20,000个的块 . 然后，您可以循环 dfgroups 中每个项目的代码 . 此外，您还希望 matches 成为您可以追加的自己的列表 . 最后，为了便于阅读，对于一个很长的行，你可能只想写一个 mapper 函数而不是使用一个庞大的lambda .

把这一切放在一起，你的代码就可以像这样重写 .

df = pd.read_csv('dec 10.csv')

# split df into groups of 20,000
dfgroups = [df[x:x+20000] for x in range(0, len(df), 20000)]
matches = [] # empty list to store matches

for dfgroup in dfgroups:

    # a function to replace that long line, more readable
    # this function will get redefined every loop, using the new `dfgroup` each iteration
    # this is optional, and you can instead keep that long line, replacing `df` with `dfgroup`
    def mapper(x):
        values = dfgroup['Customer Name'].values
        result = difflib.get_close_matches(x, values, n=2, cutoff=0.8))
        result = result.apply(pd.Series)
        result = result.dropna(axis=0)
        return result

    match = group['Customer Name'].map(mapper) # passing the function as an argument rather than using a lambda
    matches.append(match) # append it to the matches list

现在 matches 相当于 [match1, match2, match3, ...] ，可以像 matches[0] matches[1] 等一样使用

2 years ago

您可以循环遍历数据框列表，以便在每次迭代时只需引用 df 并避免重复代码：

match = pd.Dataframe()
for df in [df1,df2,df3]:
    match_ = df['Customer Name'].map(lambda x: difflib
                 .get_close_matches(x, df['Customer Name'].values, n=2, cutoff=0.8))
                 .apply(pd.Series).dropna(axis=0)
    match = match.append(match_, ignore_index =True)