使用Python 2.7 / Windows运行多处理作业


Run a multiprocessing job with Python 2.7 / Windows

基于这个答案，我想用python 2.7/windows运行这个multiprocessing作业：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def main():
    import itertools as it
    from multiprocessing import Pool

    def dothejob(i, j, k):
        print i, j, k

    the_args = it.product(range(100), range(100), range(100))
    pool = Pool(4)

    def jobWrapper(args):
        return dothejob(*args)

    res = pool.map(jobWrapper, the_args)

if __name__ == '__main__':
    main()

main()和最后两行是必需的，因为没有它们，就有众所周知的错误：

This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:

1
2
if __name__ == '__main__':
    ....

但即使这样，我也会得到这个错误：

File"C:\Users\User\Desktop\test.py", line 14, in main
res = pool.map(jobWrapper, the_args)
File"C:\Python27\lib\multiprocessing\pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File"C:\Python27\lib\multiprocessing\pool.py", line 558, in get
raise self._value
cPickle.PicklingError: Can't pickle : attribute lookup >builtin.function failed

这里涉及到的cPickle在哪里？为什么会出现这种错误/如何解决它？

相关讨论

型

所有定义必须在模块范围内：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import itertools as it
from multiprocessing import Pool, freeze_support

def dothejob(i, j, k):
    print i, j, k

def jobWrapper(args):
    return dothejob(*args)

def main():
    the_args = it.product(range(100), range(100), range(100))
    pool = Pool(4)
    res = pool.map(jobWrapper, the_args)

if __name__ == '__main__':
    freeze_support() #you need this in windows
    main()

您还需要在Windows中调用freeze_support。

相关讨论