Here is example of using multiprocessing and queue in Python.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | from multiprocessing import Process, Queue from multiprocessing.process import current_process def arbitary_function(number): print "Executing number %s from %s" % (number, current_process().name) return True def worker(worker_queue, result_queue): try: for number in iter(worker_queue.get, None): arbitary_function(number) result_queue.put("%s success with: %s" % (number, current_process().name)) except Exception, e: result_queue.put("%s failed with: %s" % (current_process().name, e.message)) if __name__ == "__main__": numbers = [x for x in range(1000)] worker_queue = Queue() result_queue = Queue() for n in numbers: worker_queue.put(n) core_worker = 2 workers = [Process(target=worker, args=(worker_queue, result_queue)) for i in range(core_worker)] for each in workers: each.start() for r in iter(results.get, None): print r |
You can get the results from “result_queue.get()” and keep in mind that queue is generator.
It will give results:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | Executing number 0 from Process-1 Executing number 1 from Process-1 Executing number 2 from Process-1 Executing number 4 from Process-2 Executing number 3 from Process-1 Executing number 5 from Process-1 Executing number 6 from Process-1 Executing number 7 from Process-1 Executing number 8 from Process-1 Executing number 9 from Process-2 Executing number 10 from Process-1 Executing number 11 from Process-2 Executing number 12 from Process-1 Executing number 13 from Process-2 Executing number 14 from Process-2 Executing number 15 from Process-1 Executing number 16 from Process-2 Executing number 17 from Process-1 Executing number 18 from Process-2 Executing number 19 from Process-1 Executing number 20 from Process-2 |
Fyi, If we want to run several multipleprocess in asynchronous way, better don’t use join(). Because it will block main process until all process finished.