openclean.engine.parallel module
Collection of helper functions for parallel processing.
- openclean.engine.parallel.process_list(func: Callable, values: Iterable, processes: int) List
Process a given list of values in parallel. Applies the given function to each value in the list and returnes the processed result.
The current implementation uses a multiprocess pool to process the list with the default map function. In the future we may want to modify this behavior, e.g., use imap to account for large lists.
- Parameters
func (callable) – Function that is applied to list values.
values (iterable) – Iterable of values that are processed by the given function.
processes (int) – Number of parallel proceses to use.
- Return type
list