Run Pipelines and Primitives
Examples require access to the calibration database hosted at Keck Observatory. See installation to configure access.
Run a Pipeline
Pipelines represent a sequence of primitives (individual processing steps) and are the typical objects users will interact with to process their data.
It is recommended to use the call method which configures and runs a pipeline in one function call.
from hispecdrp.pipelines import Stage0Pipeline
from hispecdrp.utils import download_gdrive_file
# Load credentials for calibration DB at Keck
from dotenv import load_dotenv
load_dotenv()
# Download simulated ramp from Google drive
sci_path_raw = download_gdrive_file('HB.20240924.00000.00_raw.fits')
# Setup and call the pipeline in one go
# Default output_dir is current working directory
model_L0 = Stage0Pipeline.call(
dataset=sci_path_raw,
output_dir='/path/to/output/dir/'
)
The pipeline input dataset can be a single file path or datamodel, or a list of multiple files/models processed in sequence.
The pipeline output depends on the input type and pipeline class. Science pipelines (Stages 0-2) return a data model or list of models corresponding to their stage name For example, the Stage0Pipeline returns an L0Model.
Configuration
Parameters for pipelines and primitives are specified in each the class’s PARAMETERS dictionary (class attribute).
Common Primitive and Pipeline Parameters:
dataset: The primary input, which can be anything a primitive/pipeline expects. This is typically a filepath, a datamodel, or a list of either. Default is
None.save_result: Whether or not to save the results. Default is
Falsefor primitives, andTruefor pipelines.output_dir: The directory where output files will be saved. If
None, the output_dir is set to the input directory. If the input directory is alsoNone, the output_dir is set to the current working directory. Default isNone. If a primitive does not specify output_dir, it will adopt the value from the pipeline.output_file: The base filename where output will be saved. Default is
None.max_cores: The maximum number of CPU cores to use for parallel processing. The default value is 1. Primitives will inherit this value from the pipeline if not specified.
Primitives only
run: Whether or not to run the primitive. Default is
True.suffix: A suffix to append to output filenames. Each primitive specifies a unique default suffix.
output_model: The output data model to use. See
_get_output_model()for more information. The default for many primitives is'input', which means the output will be the same as the input.save_preview: Whether or not to save the preview of the results. Default is
False. If either save_preview or show_preview isTrue, the primitive will generate a preview summary of the results, (typically a figure). The exact behavior depends on the primitive.show_preview: Whether or not to display the preview of the results. Default is
False.
The appropriate configuration files for a pipeline and all of its primitives are automatically selected from hispecdrp/configs/. See DRP Pipeline and Primitive Configuration for details on how configuration files are selected.
Users can override the defaults with their own config file. In this example, primitive config files are still automatically selected, but any parameters specified in the user config file will override those defaults.
User Config File
class: Stage0Pipeline
parameters:
output_dir: /path/to/output_dir/
max_cores: 4
primitives:
dq_init:
suffix: proc-dq_init
ramp_fit:
method: ols
jwst_jump_detection: False
Using the config file:
model_L0 = Stage0Pipeline.call(sci_path_raw, config='stage0_pipeline_config.yaml')
Python Interface
Pipelines can also be configured within Python using the primitives argument:
primitives = {
# Keys = primitive alias defined in the pipeline class.
'nonlin_corr': {
'run': False, # Skip nonlinearity correction
},
'ramp_fit': {
'method': 'ols',
},
}
model_L0 = Stage0Pipeline.call(
sci_path_raw,
primitives=primitives,
output_dir='/path/to/output/dir/'
)
Users can also interact with the pipeline object directly:
pipe = Stage0Pipeline(dataset=sci_path_raw)
# Change a parameter for a primitive
pipe.ramp_fit.method = 'ols'
model_L0 = pipe.apply(output_dir='/path/to/output/dir/')
Running Primitives Directly
Primitives can also be run on their own, outside of a pipeline, using the same call interface.
from hispecdrp.primitives import FitRampPrimitive
model_L0 = FitRampPrimitive.call(dataset=ramp_model, method='ols')
The result model_L0 returned from FitRampPrimitive is not-yet corrected for dark current, gain, or flat-fielding which are included in the Stage 0 pipeline after ramp fitting.