Home/ Questions/Q 373
Next
Answered
Kenil Vasani
Kenil Vasani

Kenil Vasani

View Profile
  • 5
Asked: 2020-12-10T22:51:11+00:00 In:

Python Multiprocessing Pool Map: AttributeError: Can’t pickle local object

  • 5

I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores.

I wrote the following code, which works if I use normal map, but with pool.map returns an error.

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      other = OtherClass()

      def single(params):
          sentences, graph = params
          return [other.run(sentence, graph) for sentence in sentences]

      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

Error:

AttributeError: Can’t pickle local object ‘SomeClass.some_method..single’

Why can’t it pickle single? I even tried to movesingle to the global module scope (not inside the class – makes it independent of the context):

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

and I get the following error:

AttributeError: Can’t get attribute ‘single’ on module ‘main‘ from ‘…/test.py’

multiprocessingpythonpython-multiprocessing
Share

    1 Answer

    1. Kenil Vasani

      Kenil Vasani

      View Profile
      Best Answer

      Error 1:

      AttributeError: Can’t pickle local object
      ‘SomeClass.some_method..single’

      You solved this error yourself by moving the nested target-function single() out to the top-level.

      Background:

      Pool needs to pickle (serialize) everything it sends to its worker-processes (IPC). Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. For that to work, the function needs to be defined at the top-level, nested functions won’t be importable by the child and already trying to pickle them raises an exception (more).

      Error 2:

      AttributeError: Can’t get attribute ‘single’ on module ‘main’ from
      ‘…/test.py’

      You are starting the pool before you define your function and classes, that way the child processes cannot inherit any code. Move your pool start up to the bottom and protect (why?) it with if __name__ == '__main__':

      import multiprocessing

class OtherClass:
  def run(self, sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
   def __init__(self):
       self.sentences = [["Some string"]]
       self.graphs = ["string"]

   def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))

if __name__ == '__main__':  # <- prevent RuntimeError for 'spawn'
    # and 'forkserver' start_methods
    with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
        print(SomeClass().some_method())

      Appendix

      …I would like to spread the work over all of my cores.

      Potentially helpful background on how multiprocessing.Pool is chunking work:

      Python multiprocessing: understanding logic behind chunksize

      • 5

    You must login to add an answer.