The concurrent.futures
module can be used to run tasks in parallel in Python. Here is an example:
import concurrent.futures from concurrent.futures import ProcessPoolExecutor with ProcessPoolExecutor(max_workers = 10) as executor: futures = [executor.submit(perform_task, task) for task in tasks] results = [future.result() for future in futures]
The ProcessPoolExecutor
class uses a pool of processes to execute tasks asynchronously. The submit
function immediately returns a Future
object, and you can call future.result()
, which will block until the task has completed.
Note that there is also a ThreadPoolExecutor
class which uses a pool of threads; however, due to the Global Interpreter Lock, only one thread can execute python bytecode at any one time, which means that you will not achieve any parallelisation in most cases.