使用Python多处理解决令人难以置信的并行问题-Java 学习之路

如何使用multiprocessing来解决embarrassingly parallel problems？

令人尴尬的并行问题通常包括三个基本部分：

Read 输入数据（来自文件，数据库，tcp连接等） .
Run 对输入数据的计算，其中每个计算独立于任何其他计算 .
Write 计算结果（对文件，数据库，tcp连接等） .

我们可以在两个方面并行化程序：

第2部分可以在多个核上运行，因为每个计算都是独立的;处理顺序无关紧要 .
每个部分都可以独立运行 . 第1部分可以将数据放在输入队列中，第2部分可以从输入队列中提取数据并将结果放到输出队列中，第3部分可以将结果从输出队列中拉出并写出来 .

这似乎是并发编程中最基本的模式，但我仍然在尝试解决它时迷失了，所以 let's write a canonical example to illustrate how this is done using multiprocessing .

下面是示例问题：给定CSV file，其中整数行作为输入，计算它们的总和 . 将问题分成三个部分，这些部分可以并行运行：

将输入文件处理为原始数据（整数列表/可迭代）
并行计算数据的总和
输出总和

下面是传统的单进程绑定Python程序，它解决了以下三个任务：

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# basicsums.py
"""A program that reads integer values from a CSV file and writes out their
sums to another CSV file.
"""

import csv
import optparse
import sys

def make_cli_parser():
    """Make the command line interface parser."""
    usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV",
            __doc__,
            """
ARGUMENTS:
    INPUT_CSV: an input CSV file with rows of numbers
    OUTPUT_CSV: an output file that will contain the sums\
"""])
    cli_parser = optparse.OptionParser(usage)
    return cli_parser


def parse_input_csv(csvfile):
    """Parses the input CSV and yields tuples with the index of the row
    as the first element, and the integers of the row as the second
    element.

    The index is zero-index based.

    :Parameters:
    - `csvfile`: a `csv.reader` instance

    """
    for i, row in enumerate(csvfile):
        row = [int(entry) for entry in row]
        yield i, row


def sum_rows(rows):
    """Yields a tuple with the index of each input list of integers
    as the first element, and the sum of the list of integers as the
    second element.

    The index is zero-index based.

    :Parameters:
    - `rows`: an iterable of tuples, with the index of the original row
      as the first element, and a list of integers as the second element

    """
    for i, row in rows:
        yield i, sum(row)


def write_results(csvfile, results):
    """Writes a series of results to an outfile, where the first column
    is the index of the original row of data, and the second column is
    the result of the calculation.

    The index is zero-index based.

    :Parameters:
    - `csvfile`: a `csv.writer` instance to which to write results
    - `results`: an iterable of tuples, with the index (zero-based) of
      the original row as the first element, and the calculated result
      from that row as the second element

    """
    for result_row in results:
        csvfile.writerow(result_row)


def main(argv):
    cli_parser = make_cli_parser()
    opts, args = cli_parser.parse_args(argv)
    if len(args) != 2:
        cli_parser.error("Please provide an input file and output file.")
    infile = open(args[0])
    in_csvfile = csv.reader(infile)
    outfile = open(args[1], 'w')
    out_csvfile = csv.writer(outfile)
    # gets an iterable of rows that's not yet evaluated
    input_rows = parse_input_csv(in_csvfile)
    # sends the rows iterable to sum_rows() for results iterable, but
    # still not evaluated
    result_rows = sum_rows(input_rows)
    # finally evaluation takes place as a chain in write_results()
    write_results(out_csvfile, result_rows)
    infile.close()
    outfile.close()


if __name__ == '__main__':
    main(sys.argv[1:])

让我们采用这个程序并重写它以使用多处理来并行化上面概述的三个部分 . 下面是这个新的并行化程序的框架，需要充实以解决注释中的部分：

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# multiproc_sums.py
"""A program that reads integer values from a CSV file and writes out their
sums to another CSV file, using multiple processes if desired.
"""

import csv
import multiprocessing
import optparse
import sys

NUM_PROCS = multiprocessing.cpu_count()

def make_cli_parser():
    """Make the command line interface parser."""
    usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV",
            __doc__,
            """
ARGUMENTS:
    INPUT_CSV: an input CSV file with rows of numbers
    OUTPUT_CSV: an output file that will contain the sums\
"""])
    cli_parser = optparse.OptionParser(usage)
    cli_parser.add_option('-n', '--numprocs', type='int',
            default=NUM_PROCS,
            help="Number of processes to launch [DEFAULT: %default]")
    return cli_parser


def main(argv):
    cli_parser = make_cli_parser()
    opts, args = cli_parser.parse_args(argv)
    if len(args) != 2:
        cli_parser.error("Please provide an input file and output file.")
    infile = open(args[0])
    in_csvfile = csv.reader(infile)
    outfile = open(args[1], 'w')
    out_csvfile = csv.writer(outfile)

    # Parse the input file and add the parsed data to a queue for
    # processing, possibly chunking to decrease communication between
    # processes.

    # Process the parsed data as soon as any (chunks) appear on the
    # queue, using as many processes as allotted by the user
    # (opts.numprocs); place results on a queue for output.
    #
    # Terminate processes when the parser stops putting data in the
    # input queue.

    # Write the results to disk as soon as they appear on the output
    # queue.

    # Ensure all child processes have terminated.

    # Clean up files.
    infile.close()
    outfile.close()


if __name__ == '__main__':
    main(sys.argv[1:])

这些代码段以及another piece of code that can generate example CSV files用于测试目的，可以是found on github .

关于你如何让并发专家来解决这个问题，我将不胜感激 .

Here are some questions I had when thinking about this problem. 寻址任何/所有的奖励积分：

我是否应该有子进程来读取数据并将其放入队列中，或者主进程是否可以不阻塞地执行此操作直到读取所有输入？
同样，我是否应该有一个子进程从处理过的队列中写出结果，或者主进程是否可以执行此操作而无需等待所有结果？
我应该使用processes pool进行总和操作吗？
如果是，我会在池上调用哪种方法让它开始处理进入输入队列的结果，而不会阻塞输入和输出进程？ apply_async()？ map_async()？ imap()？ imap_unordered()？
假设我们不需要在数据输入时虹吸输入和输出队列，但可以等到解析所有输入并计算所有结果（例如，因为我们知道所有输入和输出都适合系统内存） . 我们是否应该以任何方式更改算法（例如，不与I / O同时运行任何进程）？

5 回答

我的解决方案有一个额外的铃声和哨子，以确保输出的顺序与输入的顺序相同 . 我使用multiprocessing.queue来在进程之间发送数据，发送停止消息，以便每个进程知道退出检查队列 . 我认为来源中的评论应该清楚说明发生了什么，但如果没有让我知道 .

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# multiproc_sums.py
"""A program that reads integer values from a CSV file and writes out their
sums to another CSV file, using multiple processes if desired.
"""

import csv
import multiprocessing
import optparse
import sys

NUM_PROCS = multiprocessing.cpu_count()

def make_cli_parser():
    """Make the command line interface parser."""
    usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV",
            __doc__,
            """
ARGUMENTS:
    INPUT_CSV: an input CSV file with rows of numbers
    OUTPUT_CSV: an output file that will contain the sums\
"""])
    cli_parser = optparse.OptionParser(usage)
    cli_parser.add_option('-n', '--numprocs', type='int',
            default=NUM_PROCS,
            help="Number of processes to launch [DEFAULT: %default]")
    return cli_parser

class CSVWorker(object):
    def __init__(self, numprocs, infile, outfile):
        self.numprocs = numprocs
        self.infile = open(infile)
        self.outfile = outfile
        self.in_csvfile = csv.reader(self.infile)
        self.inq = multiprocessing.Queue()
        self.outq = multiprocessing.Queue()

        self.pin = multiprocessing.Process(target=self.parse_input_csv, args=())
        self.pout = multiprocessing.Process(target=self.write_output_csv, args=())
        self.ps = [ multiprocessing.Process(target=self.sum_row, args=())
                        for i in range(self.numprocs)]

        self.pin.start()
        self.pout.start()
        for p in self.ps:
            p.start()

        self.pin.join()
        i = 0
        for p in self.ps:
            p.join()
            print "Done", i
            i += 1

        self.pout.join()
        self.infile.close()

    def parse_input_csv(self):
            """Parses the input CSV and yields tuples with the index of the row
            as the first element, and the integers of the row as the second
            element.

            The index is zero-index based.

            The data is then sent over inqueue for the workers to do their
            thing.  At the end the input process sends a 'STOP' message for each
            worker.
            """
            for i, row in enumerate(self.in_csvfile):
                row = [ int(entry) for entry in row ]
                self.inq.put( (i, row) )

            for i in range(self.numprocs):
                self.inq.put("STOP")

    def sum_row(self):
        """
        Workers. Consume inq and produce answers on outq
        """
        tot = 0
        for i, row in iter(self.inq.get, "STOP"):
                self.outq.put( (i, sum(row)) )
        self.outq.put("STOP")

    def write_output_csv(self):
        """
        Open outgoing csv file then start reading outq for answers
        Since I chose to make sure output was synchronized to the input there
        is some extra goodies to do that.

        Obviously your input has the original row number so this is not
        required.
        """
        cur = 0
        stop = 0
        buffer = {}
        # For some reason csv.writer works badly across processes so open/close
        # and use it all in the same process or else you'll have the last
        # several rows missing
        outfile = open(self.outfile, "w")
        self.out_csvfile = csv.writer(outfile)

        #Keep running until we see numprocs STOP messages
        for works in range(self.numprocs):
            for i, val in iter(self.outq.get, "STOP"):
                # verify rows are in order, if not save in buffer
                if i != cur:
                    buffer[i] = val
                else:
                    #if yes are write it out and make sure no waiting rows exist
                    self.out_csvfile.writerow( [i, val] )
                    cur += 1
                    while cur in buffer:
                        self.out_csvfile.writerow([ cur, buffer[cur] ])
                        del buffer[cur]
                        cur += 1

        outfile.close()

def main(argv):
    cli_parser = make_cli_parser()
    opts, args = cli_parser.parse_args(argv)
    if len(args) != 2:
        cli_parser.error("Please provide an input file and output file.")

    c = CSVWorker(opts.numprocs, args[0], args[1])

if __name__ == '__main__':
    main(sys.argv[1:])

回复于 2024-05-04T20:17:25+08:00

65
我意识到我最近发现了GNU parallel，并希望展示用它完成这个典型任务是多么容易 .
```
cat input.csv | parallel ./sum.py --pipe > sums
```
像这样的东西可以用于 sum.py ：
```
#!/usr/bin/python

from sys import argv

if __name__ == '__main__':
    row = argv[-1]
    values = (int(value) for value in row.split(','))
    print row, ':', sum(values)
```
对于 input.csv 中的每一行，并行将运行 sum.py （当然并行），然后将结果输出到 sums . 显然比_1118146麻烦好
回复于 2024-05-04T20:17:25+08:00
5

来晚了...

joblib在多处理之上有一个层，以帮助进行并行循环 . 除了非常简单的语法之外，它还为您提供了诸如延迟调度作业以及更好的错误报告等功能 .

作为免责声明，我是joblib的原作者 .

回复于 2024-05-04T20:17:25+08:00
0
老套 .

p1.py
```
import csv
import pickle
import sys

with open( "someFile", "rb" ) as source:
    rdr = csv.reader( source )
    for line in eumerate( rdr ):
        pickle.dump( line, sys.stdout )
```
p2.py
```
import pickle
import sys

while True:
    try:
        i, row = pickle.load( sys.stdin )
    except EOFError:
        break
    pickle.dump( i, sum(row) )
```
p3.py
```
import pickle
import sys
while True:
    try:
        i, row = pickle.load( sys.stdin )
    except EOFError:
        break
    print i, row
```
这是多处理最终结构 .
```
python p1.py | python p2.py | python p3.py
```
是的，shell在操作系统级别将这些编织在一起 . 这对我来说似乎更简单，而且效果非常好 .

是的，使用pickle（或cPickle）的开销略高一些 . 然而，简化似乎值得付出努力 .

如果您希望文件名成为 p1.py 的参数，那么这是一个简单的更改 .

更重要的是，像下面这样的功能非常方便 .
```
def get_stdin():
    while True:
        try:
            yield pickle.load( sys.stdin )
        except EOFError:
            return
```
这允许你这样做：
```
for item in get_stdin():
     process item
```
这很简单，但它不容易让你运行多个P2.py副本 .

你有两个问题：扇出和扇入 . P1.py必须以某种方式扇出多个P2.py . 并且P2.py必须以某种方式将他们的结果合并为单个P3.py.

旧式的扇出方法是一种“推”式架构，非常有效 .

从理论上讲，从公共队列中拉出多个P2.py是资源的最佳分配 . 这通常是理想的，但它也是相当多的编程 . 编程真的有必要吗？或者循环处理是否足够好？

实际上，你'll find that making P1.py do a simple 1118155 dealing among multiple P2.py'可能相当不错 . 您已将P1.py配置为通过命名管道处理n个P2.py副本 . P2.py将从适当的管道中读取 .

如果一个P2.py得到所有“最坏情况”的数据并且落后了怎么办？是的，循环赛并不完美 . 但它只比一个P2.py好，你可以通过简单的随机化解决这个偏见 .

从多个P2.py到一个P3.py的扇入仍然有点复杂 . 在这一点上，老派的做法不再有利 . P3.py需要使用 select 库读取多个命名管道以交错读取 .
回复于 2024-05-04T20:17:25+08:00
5

也许有可能在第1部分中引入一些并行性 . 对于像CSV这样简单的格式可能不是问题，但如果输入数据的处理明显慢于读取数据，则可以读取更大的块，然后继续读取，直到找到“行分隔符”（ CSV情况下的换行符，但同样取决于读取的格式;如果格式足够复杂则不起作用） .

这些块，每个可能包含多个条目，然后可以养殖到一群并行进程从队列中读取作业，在那里它们被解析和拆分，然后放置在第2阶段的队列中 .

回复于 2024-05-04T20:17:25+08:00

使用Python多处理解决令人难以置信的并行问题

5 回答

相关问题