hispecdrp.pipelines.base_pipeline
Classes
|
Base pipeline class for managing a sequence of primitives. |
- class BasePipeline(dataset=None, config=None, context=None, log_config=None, primitives=None, **kwargs)[source]
Bases:
ProcessingComponentBase pipeline class for managing a sequence of primitives.
- Parameters:
None) (suffix (str |)
None)
- apply(dataset=None, primitives=None, **kwargs)[source]
The main method a user calls to run a pipeline.
- Parameters:
dataset – The dataset to process and is up to the subclass to decide what this represents, but is typically a datamodel, or simple containers of datamodels (or filepaths).
primitives (dict | None) – A dictionary of primitive aliases and user-specified parameters to override the default primitive parameters for this pipeline.
**kwargs – Additional keyword arguments to set or override for this pipeline. These will override any arguments set during initialization.
- Returns:
dict – The main output of the pipeline (self._output[‘output’]), which is typically a datamodel or a container of datamodels. This is typically the last primitive’s output, but is up to the pipeline implementation.
- classmethod call(dataset, config=None, context=None, primitives=None, return_instance=False, **kwargs)[source]
Top level method to instantiate a Step or Pipeline and run it in one call.
- Parameters:
dataset (object) – The dataset to process.
context (object, optional) – The context in which the step is being called.
config (object, optional) – The configuration to use for the step. Can be a dict, a config file path, or a ConfigFile object.
parent (LigerIRISStep, optional) – The parent step or pipeline of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
return_instance (bool, optional) – If True, return a tuple of (result, step_instance). If False, return only the result.
kwargs (dict) – Override step parameters.
- Returns:
results (Any) – The result of processing the input with the step.
- classmethod from_config_file(config, dataset=None, log_config=None, **kwargs)[source]
Initialize a pipeline from a configuration file.
- Parameters:
dataset (Any, optional) – The dataset to process.
config (str, dict, or ConfigFile, optional.) –
- The configuration for the pipeline. Can be one of:
str: A full path to a configuration file or a filename in the ConfigStore.
dict: A dictionary of configuration parameters.
ConfigFile: A ConfigFile instance.
If not provided, the pipeline will attempt to auto-configure based on the input dataset and context. If the dataset is required for auto-configuration, it must be provided.
log_config (dict, str, or None, optional) – The logging configuration to use for the pipeline.
kwargs (dict) – Additional keyword arguments to pass to the pipeline constructor.
- Returns:
BasePipeline – An instance of the pipeline with the configuration loaded.
- get_effective_config()[source]
- init_config(dataset=None, config=None, context=None)[source]
- init_parameters(**kwargs)[source]
Initialize parameter values as top-level attributes.
- Parameters:
**kwargs – Any additional parameter values to override.
- init_primitives(context=None, primitives=None)[source]
Initialize all primitives in the pipeline.
- class ConfigFile(name, class_name, description=None, filepath=None, context=None, parameters=None, primitives=None, logging=None)[source]
Bases:
objectA class representing a configuration file.
- _DEFAULT_CONTEXT = {'instrument_era': ['0.0.1', None], 'instrument_mode': None, 'processing_mode': None, 'quicklook': None}
- classmethod from_dict(config)[source]
- classmethod from_file(filepath, **kwargs)[source]
- save(output_path, output_dir=None)[source]
- to_dict()[source]
- class ConfigSelector(class_name, context=None, skip_filters=None)[source]
Bases:
CalibrationSelector- _filter_class(candidates)[source]
Filter config files by the primitive or pipeline class they are meant to configure.
- Parameters:
candidates (list[ConfigFile]) – The list of candidate config files to filter.
- Returns:
filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the class.
- _filter_ifs_mode(meta, candidates)[source]
Filter candidates by IFS mode.
- Parameters:
meta (dict) – Datamodel metadata
candidates (list[ConfigFile]) – List of candidate config files to filter.
- Returns:
filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the IFS mode.
- _filter_instrument_era(meta, candidates)[source]
Filter candidates by the valid instrument_era range.
- Parameters:
meta (dict) – Datamodel metadata
candidates (list[ConfigFile]) – List of candidate config files to filter.
- Returns:
filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the instrument era.
- _filter_instrument_mode(meta, candidates)[source]
Filter candidates by instrument mode.
- Parameters:
meta (dict) – Datamodel metadata
candidates (list[ConfigFile]) – List of candidate config files to filter.
- Returns:
filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the instrument mode.
- _filter_processing_mode(candidates)[source]
Filter candidates by the processing mode (
self.context.processing_mode).Unlike
quicklook,processing_modehas no natural default value, so an unspecified input (None) leaves the candidates unconstrained. When a value is specified, a config file that explicitly matches it always wins; a config file that omitsprocessing_mode(None, a wildcard) is only used as a fallback when nothing explicitly matches.- Parameters:
candidates (list[ConfigFile]) – List of candidate config files to filter.
- Returns:
filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the processing mode criteria.
- _filter_quicklook(candidates)[source]
Filter candidates by the quicklook flag (
self.quicklook).A config file that omits
quicklook(context valueNone) is a wildcard: it matches any requested value. It’s only used as a fallback when no candidate explicitly matches the requested value, so an explicitTrue/Falseconfig always takes precedence over a wildcard one for the same class.Cases
Input context = None ConfigFile contexts = (None, …) -> All ConfigFile contexts = (True, False) -> False
- Input context = False
ConfigFile contexts = (True, False) -> False ConfigFile contexts = (None, …) -> All
- Input context = True
ConfigFile contexts = (True, None) -> True
- param candidates:
List of candidate config files to filter.
- type candidates:
list[ConfigFile]
- Returns:
filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the quicklook criteria.
- _select(meta, candidates)[source]
Internal method to perform the core config file selection logic.
This method first retrieves a set of candidate config files using
get_candidatesand then selects the best one (or ones) from these candidates usingselect_best. Subclasses can override this method to implement custom selection workflows, but it’s often more appropriate to overrideget_candidatesorselect_best.- Parameters:
meta (dict) – The metadata for which a config file is to be selected.
candidates (list[ConfigFile]) – The list of candidate config files to consider.
- Returns:
result (ConfigFile | None) – The selected config file if one is found, otherwise None.
- get_candidates(meta, candidates)[source]
Select the most appropriate config file for the given metadata.
- Parameters:
meta (dict) – Data metadata to match against the
metablocks of the candidate config files.candidates (list[ConfigFile]) – List of candidate config files to consider.
- Returns:
config_file (ConfigFile)
- classmethod register_filter(key)[source]
Register custom match logic for a selection key with non-equality semantics.
- select(meta, candidates)[source]
Select the most appropriate config file for the given metadata.
- Parameters:
meta (dict) – Data model metadata.
candidates (list[ConfigFile]) – List of candidate config files to consider.
- Returns:
result (ConfigFile | None) – The selected config file if one is found, otherwise None.
- select_best(meta, candidates)[source]
- class ConfigStore(keep_in_memory=True, config_dir=None)[source]
Bases:
objectA store for managing configuration files.
Examples
>>> with ConfigStore() as store: config_file : ConfigFile = store.select_and_get_config( dataset, selector ) filepath = config_file.filepath- classmethod clear_cache()[source]
- populate()[source]
- select_and_get_config(dataset, selector)[source]
Given an input dataset and a primitive class to run, select a config file.
- Parameters:
dataset (Any) – The dataset for which to select a config.
selector (ConfigSelector) – The config selector to use for selecting the config.
- Returns:
config_file (ConfigFile) – The selected config file object.
- class ProcessingComponent(action, context)[source]
Bases:
BasePrimitive- _create_preview()[source]
Create a preview image for the output data model. This method can be overridden by subclasses to implement specific preview generation logic. By default, this method does nothing.
- _get_first_model(dataset, open=True, return_type=False, **kwargs)[source]
Get the first input model and its container type from the dataset.
- Parameters:
dataset (Any) – The dataset to extract the first model from.
open (bool, optional) – If True, open the model if it’s a file path. Default is True.
return_type (bool, optional) – If True, also return the container type and length. Default is False.
**kwargs – Additional keyword arguments to pass to the datamodels.open() function.
- Returns:
DataModel | tuple[DataModel, type | None, int] – If return_type is False, returns the first DataModel. If return_type is True, returns a tuple of (first DataModel, container type, length). Supported container types are Sequence and dict. For a single model, the type is None.
- _init_log_config(log_config=None)[source]
Resolve and store
self.log_configby merging (lowest → highest priority):Package defaults from
hispecdrp/configs/logging.yamlloggingattribute embedded inself.config.log_configkwarg passed at construction time (dict or file path/name)
- _post_condition()[source]
- _pre_condition()[source]
- apply(dataset=None, **kwargs)[source]
One of the two primary methods a user calls to run a primitive. The other is
Primitive.call(). Users should not need to override this method, and care should be taken if doing so.Both BasePrimitive and BasePipeline implement apply().
- generate_output_path(model, output_dir=None)[source]
Generate an output filepath for a model.
Parameter
- modelstr | DataModel
The model to generate the output path for.
- output_dirstr | None, optional
The output directory to use. If None, the output directory is determined by
self.get_output_dir(model).
- classmethod get_config(dataset, context=None)[source]
Retrieve the configuration for this primitive based on the dataset and context.
- Parameters:
dataset (Any) – The dataset to use for selecting the configuration. Cannot be None.
context (dict | None) – Optional context to use for selecting the configuration.
- Returns:
config_file (ConfigFile) – The configuration file for this primitive.
- static get_do_traces(input_model, do_fibers=None, do_orders=None)[source]
Determine which fibers and orders to process based on the input model and optional user-specified lists.
- Parameters:
- Returns:
do_fibers (list[str | int]) – The list of fiber numbers to process.
do_orders (list[int]) – The list of order numbers to process.
- get_input_dir(dataset=None)[source]
Get the input directory for a given model.
- Parameters:
dataset (Any, optional) – The input model to determine the input directory from. If None, uses
self.dataset.
- get_output_dir(model)[source]
Get the output directory for saving any results in the context of a model. The priority for determining the output directory is as follows:
self.output_dirif explicitly set (input model not used).model._filepathdirectory if available If an input model is provided, the fallback is the directory of self.dataset.Current working directory as a final fallback.
- get_pars()[source]
Get the current parameter values.
- Returns:
dict – A dictionary of parameter names and their current values. Primitive parameters include both parameters and calibrations.
- property pars_str
Pretty print the current parameters and calibrations for this primitive or pipeline.
- Returns:
str – A formatted string showing all current parameter values.
- save_model(model, output_path=None)[source]
Save a DataModel to a file in the context of this processing component. The file suffix will be added to the existing filename (
model.meta.filename) if present and filepath is not set.
- save_output()[source]
Saves all datamodels stored at self._output[‘output’].
- set_pars(**kwargs)[source]
Set parameter values including calibrations as top-level attributes.
- Parameters:
**kwargs – Parameters to set as attributes. For primitives, one can also override calibration values.
Notes
Selector logic is not run here.
- get_config_filepath(config_file)[source]
Resolve the path to a configuration file.
- Parameters:
config_file (str) – The base filename of the configuration file or a valid file path.
- Returns:
config_filepath (str) – The resolved path to the configuration file.
- logging_context(config_dict, queue=None)[source]
The single entry point for all logging in hispecdrp.
Outermost call owns the queue and listener for its full lifetime. Nested calls (e.g. a pipeline invoking a primitive that also calls logging_context) are no-ops — they yield immediately without touching the queue, so the outer context stays alive.