hispecdrp.pipelines.base_pipeline

Classes

BasePipeline([dataset, config, context, ...])

Base pipeline class for managing a sequence of primitives.

class BasePipeline(dataset=None, config=None, context=None, log_config=None, primitives=None, **kwargs)[source]

Bases: ProcessingComponent

Base pipeline class for managing a sequence of primitives.

Parameters:
  • None) (suffix (str |)

  • None)

apply(dataset=None, primitives=None, **kwargs)[source]

The main method a user calls to run a pipeline.

Parameters:
  • dataset – The dataset to process and is up to the subclass to decide what this represents, but is typically a datamodel, or simple containers of datamodels (or filepaths).

  • primitives (dict | None) – A dictionary of primitive aliases and user-specified parameters to override the default primitive parameters for this pipeline.

  • **kwargs – Additional keyword arguments to set or override for this pipeline. These will override any arguments set during initialization.

Returns:

dict – The main output of the pipeline (self._output[‘output’]), which is typically a datamodel or a container of datamodels. This is typically the last primitive’s output, but is up to the pipeline implementation.

classmethod call(dataset, config=None, context=None, primitives=None, return_instance=False, **kwargs)[source]

Top level method to instantiate a Step or Pipeline and run it in one call.

Parameters:
  • dataset (object) – The dataset to process.

  • context (object, optional) – The context in which the step is being called.

  • config (object, optional) – The configuration to use for the step. Can be a dict, a config file path, or a ConfigFile object.

  • parent (LigerIRISStep, optional) – The parent step or pipeline of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.

  • return_instance (bool, optional) – If True, return a tuple of (result, step_instance). If False, return only the result.

  • kwargs (dict) – Override step parameters.

Returns:

results (Any) – The result of processing the input with the step.

classmethod from_config_file(config, dataset=None, log_config=None, **kwargs)[source]

Initialize a pipeline from a configuration file.

Parameters:
  • dataset (Any, optional) – The dataset to process.

  • config (str, dict, or ConfigFile, optional.) –

    The configuration for the pipeline. Can be one of:
    • str: A full path to a configuration file or a filename in the ConfigStore.

    • dict: A dictionary of configuration parameters.

    • ConfigFile: A ConfigFile instance.

    If not provided, the pipeline will attempt to auto-configure based on the input dataset and context. If the dataset is required for auto-configuration, it must be provided.

  • log_config (dict, str, or None, optional) – The logging configuration to use for the pipeline.

  • kwargs (dict) – Additional keyword arguments to pass to the pipeline constructor.

Returns:

BasePipeline – An instance of the pipeline with the configuration loaded.

get_effective_config()[source]
init_config(dataset=None, config=None, context=None)[source]
init_parameters(**kwargs)[source]

Initialize parameter values as top-level attributes.

Parameters:

**kwargs – Any additional parameter values to override.

init_primitives(context=None, primitives=None)[source]

Initialize all primitives in the pipeline.

Parameters:
  • context (dict, optional) – A dictionary of context information to use for selecting the configuration file for each primitive.

  • primitives (dict, optional) – A dictionary of primitive aliases and user-specified parameters.

class ConfigFile(name, class_name, description=None, filepath=None, context=None, parameters=None, primitives=None, logging=None)[source]

Bases: object

A class representing a configuration file.

class_name

The name of the primitive or pipeline class (with or without the module path).

Type:

str

filepath

Full filepath to the config file.

Type:

str

context

A dictionary of metadata for this config file.

Type:

dict

_DEFAULT_CONTEXT = {'instrument_era': ['0.0.1', None], 'instrument_mode': None, 'processing_mode': None, 'quicklook': None}
property class_
property filename
classmethod from_dict(config)[source]
classmethod from_file(filepath, **kwargs)[source]
property is_pipeline
property name

Unique name for a config file.

save(output_path, output_dir=None)[source]
to_dict()[source]
class ConfigSelector(class_name, context=None, skip_filters=None)[source]

Bases: CalibrationSelector

_CUSTOM_FILTERS = {}
_filter_class(candidates)[source]

Filter config files by the primitive or pipeline class they are meant to configure.

Parameters:

candidates (list[ConfigFile]) – The list of candidate config files to filter.

Returns:

filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the class.

_filter_ifs_mode(meta, candidates)[source]

Filter candidates by IFS mode.

Parameters:
  • meta (dict) – Datamodel metadata

  • candidates (list[ConfigFile]) – List of candidate config files to filter.

Returns:

filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the IFS mode.

_filter_instrument_era(meta, candidates)[source]

Filter candidates by the valid instrument_era range.

Parameters:
  • meta (dict) – Datamodel metadata

  • candidates (list[ConfigFile]) – List of candidate config files to filter.

Returns:

filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the instrument era.

_filter_instrument_mode(meta, candidates)[source]

Filter candidates by instrument mode.

Parameters:
  • meta (dict) – Datamodel metadata

  • candidates (list[ConfigFile]) – List of candidate config files to filter.

Returns:

filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the instrument mode.

_filter_processing_mode(candidates)[source]

Filter candidates by the processing mode (self.context.processing_mode).

Unlike quicklook, processing_mode has no natural default value, so an unspecified input (None) leaves the candidates unconstrained. When a value is specified, a config file that explicitly matches it always wins; a config file that omits processing_mode (None, a wildcard) is only used as a fallback when nothing explicitly matches.

Parameters:

candidates (list[ConfigFile]) – List of candidate config files to filter.

Returns:

filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the processing mode criteria.

_filter_quicklook(candidates)[source]

Filter candidates by the quicklook flag (self.quicklook).

A config file that omits quicklook (context value None) is a wildcard: it matches any requested value. It’s only used as a fallback when no candidate explicitly matches the requested value, so an explicit True/False config always takes precedence over a wildcard one for the same class.

Cases

  1. Input context = None ConfigFile contexts = (None, …) -> All ConfigFile contexts = (True, False) -> False

  2. Input context = False

    ConfigFile contexts = (True, False) -> False ConfigFile contexts = (None, …) -> All

  3. Input context = True

    ConfigFile contexts = (True, None) -> True

param candidates:

List of candidate config files to filter.

type candidates:

list[ConfigFile]

Returns:

filtered_candidates (list[ConfigFile]) – The filtered list of config files that match the quicklook criteria.

_select(meta, candidates)[source]

Internal method to perform the core config file selection logic.

This method first retrieves a set of candidate config files using get_candidates and then selects the best one (or ones) from these candidates using select_best. Subclasses can override this method to implement custom selection workflows, but it’s often more appropriate to override get_candidates or select_best.

Parameters:
  • meta (dict) – The metadata for which a config file is to be selected.

  • candidates (list[ConfigFile]) – The list of candidate config files to consider.

Returns:

result (ConfigFile | None) – The selected config file if one is found, otherwise None.

get_candidates(meta, candidates)[source]

Select the most appropriate config file for the given metadata.

Parameters:
  • meta (dict) – Data metadata to match against the meta blocks of the candidate config files.

  • candidates (list[ConfigFile]) – List of candidate config files to consider.

Returns:

config_file (ConfigFile)

classmethod register_filter(key)[source]

Register custom match logic for a selection key with non-equality semantics.

select(meta, candidates)[source]

Select the most appropriate config file for the given metadata.

Parameters:
  • meta (dict) – Data model metadata.

  • candidates (list[ConfigFile]) – List of candidate config files to consider.

Returns:

result (ConfigFile | None) – The selected config file if one is found, otherwise None.

select_best(meta, candidates)[source]
class ConfigStore(keep_in_memory=True, config_dir=None)[source]

Bases: object

A store for managing configuration files.

Examples

>>> with ConfigStore() as store:
        config_file : ConfigFile = store.select_and_get_config(
            dataset,
            selector
        )
        filepath = config_file.filepath
_cache = None
classmethod clear_cache()[source]
populate()[source]
select_and_get_config(dataset, selector)[source]

Given an input dataset and a primitive class to run, select a config file.

Parameters:
  • dataset (Any) – The dataset for which to select a config.

  • selector (ConfigSelector) – The config selector to use for selecting the config.

Returns:

config_file (ConfigFile) – The selected config file object.

class ProcessingComponent(action, context)[source]

Bases: BasePrimitive

LOG_CONFIG_FILE = 'logging.yaml'
_create_preview()[source]

Create a preview image for the output data model. This method can be overridden by subclasses to implement specific preview generation logic. By default, this method does nothing.

_get_first_model(dataset, open=True, return_type=False, **kwargs)[source]

Get the first input model and its container type from the dataset.

Parameters:
  • dataset (Any) – The dataset to extract the first model from.

  • open (bool, optional) – If True, open the model if it’s a file path. Default is True.

  • return_type (bool, optional) – If True, also return the container type and length. Default is False.

  • **kwargs – Additional keyword arguments to pass to the datamodels.open() function.

Returns:

DataModel | tuple[DataModel, type | None, int] – If return_type is False, returns the first DataModel. If return_type is True, returns a tuple of (first DataModel, container type, length). Supported container types are Sequence and dict. For a single model, the type is None.

_init_log_config(log_config=None)[source]

Resolve and store self.log_config by merging (lowest → highest priority):

  1. Package defaults from hispecdrp/configs/logging.yaml

  2. logging attribute embedded in self.config.

  3. log_config kwarg passed at construction time (dict or file path/name)

_post_condition()[source]
_pre_condition()[source]
property allowed_parameters
apply(dataset=None, **kwargs)[source]

One of the two primary methods a user calls to run a primitive. The other is Primitive.call(). Users should not need to override this method, and care should be taken if doing so.

Both BasePrimitive and BasePipeline implement apply().

generate_output_path(model, output_dir=None)[source]

Generate an output filepath for a model.

Parameter

modelstr | DataModel

The model to generate the output path for.

output_dirstr | None, optional

The output directory to use. If None, the output directory is determined by self.get_output_dir(model).

classmethod get_config(dataset, context=None)[source]

Retrieve the configuration for this primitive based on the dataset and context.

Parameters:
  • dataset (Any) – The dataset to use for selecting the configuration. Cannot be None.

  • context (dict | None) – Optional context to use for selecting the configuration.

Returns:

config_file (ConfigFile) – The configuration file for this primitive.

static get_do_traces(input_model, do_fibers=None, do_orders=None)[source]

Determine which fibers and orders to process based on the input model and optional user-specified lists.

Parameters:
  • input_model (datamodels.DataModel) – The input data model.

  • do_fibers (list[str] | None, optional) – A list of fiber numbers to process. If None, all fibers are processed.

  • do_orders (list[int] | None, optional) – A list of order numbers to process. If None, all orders are processed.

Returns:
  • do_fibers (list[str | int]) – The list of fiber numbers to process.

  • do_orders (list[int]) – The list of order numbers to process.

get_input_dir(dataset=None)[source]

Get the input directory for a given model.

Parameters:

dataset (Any, optional) – The input model to determine the input directory from. If None, uses self.dataset.

get_output_dir(model)[source]

Get the output directory for saving any results in the context of a model. The priority for determining the output directory is as follows:

  1. self.output_dir if explicitly set (input model not used).

  2. model._filepath directory if available If an input model is provided, the fallback is the directory of self.dataset.

  3. Current working directory as a final fallback.

Parameters:

model (str | DataModel) – The input model to determine the output directory from if self.output_dir is not set.

Returns:

str – The output directory for saving results.

get_pars()[source]

Get the current parameter values.

Returns:

dict – A dictionary of parameter names and their current values. Primitive parameters include both parameters and calibrations.

property pars_str

Pretty print the current parameters and calibrations for this primitive or pipeline.

Returns:

str – A formatted string showing all current parameter values.

save_model(model, output_path=None)[source]

Save a DataModel to a file in the context of this processing component. The file suffix will be added to the existing filename (model.meta.filename) if present and filepath is not set.

Parameters:
  • model (DataModel) – The model to save.

  • output_path (str | None, optional) – The full filepath where to save the model. Supersedes self.output_dir and self.suffix.

Returns:

str – The file path where the model was saved.

save_output()[source]

Saves all datamodels stored at self._output[‘output’].

set_pars(**kwargs)[source]

Set parameter values including calibrations as top-level attributes.

Parameters:

**kwargs – Parameters to set as attributes. For primitives, one can also override calibration values.

Notes

Selector logic is not run here.

get_config_filepath(config_file)[source]

Resolve the path to a configuration file.

Parameters:

config_file (str) – The base filename of the configuration file or a valid file path.

Returns:

config_filepath (str) – The resolved path to the configuration file.

logging_context(config_dict, queue=None)[source]

The single entry point for all logging in hispecdrp.

Outermost call owns the queue and listener for its full lifetime. Nested calls (e.g. a pipeline invoking a primitive that also calls logging_context) are no-ops — they yield immediately without touching the queue, so the outer context stays alive.