hispecdrp.utils.parallelization_utils

Shared helpers for parallel CPU work: shared-memory views, Numba thread limits, and Numba decorators for the pipeline.

The HISPEC DRP stack depends on Numba at install time; this module imports it eagerly. numba_thread_scope() and resolve_numba_threads() share logic that probes Numba for the maximum allowed thread count (cached). On some CI runners Numba caps parallel threads at 2 even when os.cpu_count() is higher; clamping avoids ValueError from set_num_threads.

Functions

_arraytonumpy(shared_array[, shape, dtype])

Covert a shared array to a numpy array

_effective_numba_threads(requested)

Map requested parallelism to a safe count (uses cached Numba max-thread probe).

_numba_max_parallel_threads()

Probe and cache the maximum thread count Numba allows for prange / set_num_threads.

numba_thread_scope([num_threads])

Temporarily set Numba's thread pool, then restore the previous setting.

resolve_numba_threads(max_cores)

Map a requested parallelism level to a thread count safe for this host and Numba build.

_arraytonumpy(shared_array, shape=None, dtype=None)[source]

Covert a shared array to a numpy array

Originally from pyklip Wang, J. J., Ruffio, J.-B., De Rosa, R. J., et al. 2015, ASCL, ascl:1506.001 https://bitbucket.org/pyKLIP/pyklip/src/main/pyklip/parallelized.py

Parameters:
  • shared_array – a multiprocessing.Array array

  • shape – a shape for the numpy array. otherwise, will assume a 1d array

  • dtype – data type of the arrays. Should be either ctypes.c_float(default) or ctypes.c_double

Returns:

numpy_array – numpy array for vectorized operation. still points to the same memory! returns None is shared_array is None

numba_thread_scope(num_threads=None)[source]

Temporarily set Numba’s thread pool, then restore the previous setting.

On entry, this uses the same logic as resolve_numba_threads(): it determines the maximum number of threads Numba allows on this build (probed once and cached; often matches CPU count but may be as low as 2 on CI), then sets the active count to num_threads clamped to that maximum and to os.cpu_count(). This addresses environments where set_num_threads(6) raises because Numba enforces Nthreads <= 2.

Parameters:

num_threads (int or None, optional) – Explicit thread count, or None / 0 for all logical CPUs (capped). Default is None.

Examples

>>> with numba_thread_scope(4):  # noqa: SIM117
...     ...  # Numba parallel regions here
resolve_numba_threads(max_cores)[source]

Map a requested parallelism level to a thread count safe for this host and Numba build.

Uses the same maximum-thread probe and clamping as numba_thread_scope().

  • None or 0: use all logical CPUs, capped to Numba’s maximum.

  • Positive max_cores: use at most that many threads, capped the same way.

This prevents ValueError from set_num_threads on runners where Numba allows fewer threads than os.cpu_count() (e.g. GitHub Actions).