MultiProcessing in Python

 

Learning Sections          show

MultiProcessing in Python

Multiprocessing in Python involves using the multiprocessing module to run multiple processes concurrently, taking advantage of multiple CPU cores. This module provides a higher level of concurrency than threading and is especially useful for CPU-bound tasks.

---

Creating Processes

You can create and start a new process by using the multiprocessing module:

import multiprocessing

def print_numbers():
    for i in range(10):
        print(i)

p1 = multiprocessing.Process(target=print_numbers)
p1.start()
p1.join()  # Wait for the process to complete

---

Using Process Pools

The multiprocessing module provides a Pool class, which allows you to manage a pool of worker processes:

from multiprocessing import Pool

def square(n):
    return n * n

with Pool(4) as pool:
    result = pool.map(square, range(10))
    print(result)

---

Inter-Process Communication

Processes can communicate using pipes or queues. The multiprocessing.Queue class provides a FIFO mechanism for this purpose:

import multiprocessing

def producer(q):
    for i in range(5):
        q.put(i)
        print("Produced", i)

def consumer(q):
    while not q.empty():
        item = q.get()
        print("Consumed", item)

q = multiprocessing.Queue()
p1 = multiprocessing.Process(target=producer, args=(q,))
p2 = multiprocessing.Process(target=consumer, args=(q,))

p1.start()
p2.start()
p1.join()
p2.join()

---

Shared Memory

The multiprocessing module provides shared memory capabilities using Value or Array:

import multiprocessing

def increment(shared_value):
    with shared_value.get_lock():  # Synchronize access
        shared_value.value += 1

shared_value = multiprocessing.Value('i', 0)
processes = [multiprocessing.Process(target=increment, args=(shared_value,)) for _ in range(10)]

for p in processes:
    p.start()

for p in processes:
    p.join()

print(shared_value.value)  # Should print 10

---

Daemon Processes

Like threads, processes can also be run as daemons. Daemon processes are terminated when the main program exits:

import multiprocessing
import time

def background_task():
    while True:
        print("Running in the background")
        time.sleep(2)

p = multiprocessing.Process(target=background_task)
p.daemon = True
p.start()

print("Main process exiting")