hispecdrp.utils.parallelization_utils
Shared helpers for parallel CPU work: shared-memory views, Numba thread limits, and Numba decorators for the pipeline.
The HISPEC DRP stack depends on Numba at install time; this module imports it
eagerly. numba_thread_scope() and resolve_numba_threads() share logic
that probes Numba for the maximum allowed thread count (cached). On some CI runners
Numba caps parallel threads at 2 even when os.cpu_count() is higher; clamping
avoids ValueError from set_num_threads.
Functions
|
Covert a shared array to a numpy array |
|
Map requested parallelism to a safe count (uses cached Numba max-thread probe). |
|
Probe and cache the maximum thread count Numba allows for |
|
Temporarily set Numba's thread pool, then restore the previous setting. |
|
Map a requested parallelism level to a thread count safe for this host and Numba build. |
- _arraytonumpy(shared_array, shape=None, dtype=None)[source]
Covert a shared array to a numpy array
Originally from pyklip Wang, J. J., Ruffio, J.-B., De Rosa, R. J., et al. 2015, ASCL, ascl:1506.001 https://bitbucket.org/pyKLIP/pyklip/src/main/pyklip/parallelized.py
- Parameters:
shared_array – a multiprocessing.Array array
shape – a shape for the numpy array. otherwise, will assume a 1d array
dtype – data type of the arrays. Should be either ctypes.c_float(default) or ctypes.c_double
- Returns:
numpy_array – numpy array for vectorized operation. still points to the same memory! returns None is shared_array is None
- numba_thread_scope(num_threads=None)[source]
Temporarily set Numba’s thread pool, then restore the previous setting.
On entry, this uses the same logic as
resolve_numba_threads(): it determines the maximum number of threads Numba allows on this build (probed once and cached; often matches CPU count but may be as low as 2 on CI), then sets the active count tonum_threadsclamped to that maximum and toos.cpu_count(). This addresses environments whereset_num_threads(6)raises because Numba enforcesNthreads <= 2.- Parameters:
num_threads (int or None, optional) – Explicit thread count, or
None/0for all logical CPUs (capped). Default isNone.
Examples
>>> with numba_thread_scope(4): # noqa: SIM117 ... ... # Numba parallel regions here
- resolve_numba_threads(max_cores)[source]
Map a requested parallelism level to a thread count safe for this host and Numba build.
Uses the same maximum-thread probe and clamping as
numba_thread_scope().Noneor0: use all logical CPUs, capped to Numba’s maximum.Positive
max_cores: use at most that many threads, capped the same way.
This prevents
ValueErrorfromset_num_threadson runners where Numba allows fewer threads thanos.cpu_count()(e.g. GitHub Actions).