hispecdrp.datamodels.base_model

Functions

_build_empty_table_with_cols(extdef)

Build a 0-row astropy.table.Table from an extension's 'columns' spec.

_canonical_dtype(dtype_spec)

The dtype to construct/cast to: the first entry if dtype_spec is a tuple (see extensions.IMAGE_HDU()), else dtype_spec itself.

_correct_image_dtype(data, dtype_spec[, extname])

Cast an image extension's array to its declared dtype, if needed.

_doc_table_render_axis(axis[, detector_size])

_doc_table_render_dtype(dtype)

_doc_table_render_dtype_name(dtype)

_doc_table_render_shape(shape[, detector_size])

_dtype_family(dtype)

Broad numeric family of a dtype, for kind-aware auto-casting.

Classes

CalibrationModel([filepath, meta, ...])

DataModel([filepath, meta, meta_only, ...])

Base class for data models.

_classproperty(fget)

A read-only property accessible on the class itself, not just instances.

class CalibrationModel(filepath=None, meta=None, meta_only=False, extensions=None, correct_dtypes=False, config=None, validate_meta=True, instrument_name=None, shape=None, **kwargs)[source]

Bases: DataModel

CAL_TYPE = None
EXTENSIONS = {'CALIBRATIONS': {'columns': {'CAL_ROLE': <class 'str'>, 'CAL_TYPE': <class 'str'>, 'CLASS': <class 'str'>, 'DATETIME_OBS': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'ID': <class 'str'>, 'LAST_PROCESSED': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'DATA_ANCESTORS': {'columns': {'CLASS': <class 'str'>, 'FILENAME': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'PRIMARY': {'dtype': None, 'hdu_type': <class 'astropy.io.fits.hdu.image.PrimaryHDU'>}, 'RECEIPT': {'columns': {'CLASS': <class 'str'>, 'CONFIG_FILE': <class 'str'>, 'DRP_GIT_COMMIT_ID': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'PARAMETERS': <class 'str'>, 'PROC_END_TIME': <class 'str'>, 'PROC_START_TIME': <class 'str'>, 'STATUS': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}}
static _generate_filename(instrument_alias, datetime_obs, cal_type, master_cal=False, cal_version=None, origin=None, suffix=None)[source]

Create a filename for a calibration data product.

Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}.[master-]{cal_type}.{cal_version}[_{suffix}].fits

datetime_obs : YYYYMMDD.SSSSSS.SS. Where SSSSSS.SS is the number of seconds since midnight. cal_type : The type of calibration data. master_cal : Insert ‘master-’ before the cal_type if master_cal=True. cal_version : The version of the calibration file. If None, queries the local cache for the latest version if auto_version=True. suffix : optional suffix

Examples

‘HB.20240924.00000.00.dark.001.fits’ ‘HB.20240924.00000.00.dark.001-local.fits’ ‘HB.20240924.00000.00.master-dark.001.fits’ ‘HB.20240924.00000.00.master-dark.001_custom.fits’ ‘HB.20240924.00000.00.master-dark.001-local_custom.fits’ ‘HB.20240924.00000.00.dark.001_custom.fits’

Parameters:
  • datetime_obs (str) – The date and time of the observation.

  • instrument_alias (str) – The two letter instrument alias.

  • cal_type (str) – The type of calibration data.

  • master_cal (bool, optional) – If True, insert ‘master-’ before the cal_type. Defaults to False.

  • cal_version (str, optional) – The version of the calibration file. If None, queries the local cache.

  • origin (str, optional) – The origin of the calibration file, either ‘LOCAL’ or ‘KECK’. If None,

  • suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.

Returns:

filename (str) – The generated filename.

generate_calibration_id()[source]

Generate the calibration ID which is reproducible UUID 7.

Returns:

cal_id (str) – The calibration ID

generate_calibration_version(origin=None)[source]
generate_filename(suffix=None, origin=None, auto_version=None)[source]
Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}.[master-]{cal_type}.{cal_version}[_{suffix}].fits

datetime_obs : YYYYMMDD.SSSSSS.SS. Where SSSSSS.SS is the number of seconds since midnight. cal_type : The type of calibration data. master_cal : Insert ‘master-’ before the cal_type if master_cal=True. cal_version : The version of the calibration file. If None, queries the local cache for the latest version if auto_version=True. suffix : optional suffix

Parameters:
  • suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.

  • origin (str, optional) – The origin of the calibration, either ‘LOCAL’ or ‘KECK’.

  • auto_version (bool, optional) – If True, generate a new calibration version.

Returns:

filename (str) – The generated filename.

on_init()[source]
order_index(order)[source]

Return the index of the given order in the order vector.

Parameters:

order (int) – The order number to find the index for.

Returns:

int – The index of the order in the order vector.

property orders

Return the order vector for the calibration model based on the order_start and order_end metadata.

to_record(note=None)[source]

Get the calibration record as a dictionary from the metadata.

Parameters:

note (str, optional) – An optional note to add to or update the record with.

Returns:

dict – The calibration record as a dictionary.

class DataModel(filepath=None, meta=None, meta_only=False, extensions=None, correct_dtypes=False, config=None, validate_meta=True, instrument_name=None, shape=None, **kwargs)[source]

Bases: object

Base class for data models.

Parameters:
  • EXTENSIONS (dict) – Extensions and their properties.

  • _INHERIT_EXTENSIONS (bool)

  • True. (Whether to inherit extensions from parent class. Defaults to) – Whether to inherit extensions from parent class. Defaults to True.

  • _ALLOWED_ATTRIBUTES (list[str]) – List of attributes that can be set directly on the DataModel.

  • _DEFAULT_INSTRUMENT (str) – Default instrument name for the DataModel.

EXTENSIONS = {'CALIBRATIONS': {'columns': {'CAL_ROLE': <class 'str'>, 'CAL_TYPE': <class 'str'>, 'CLASS': <class 'str'>, 'DATETIME_OBS': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'ID': <class 'str'>, 'LAST_PROCESSED': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'DATA_ANCESTORS': {'columns': {'CLASS': <class 'str'>, 'FILENAME': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'PRIMARY': {'dtype': None, 'hdu_type': <class 'astropy.io.fits.hdu.image.PrimaryHDU'>}, 'RECEIPT': {'columns': {'CLASS': <class 'str'>, 'CONFIG_FILE': <class 'str'>, 'DRP_GIT_COMMIT_ID': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'PARAMETERS': <class 'str'>, 'PROC_END_TIME': <class 'str'>, 'PROC_START_TIME': <class 'str'>, 'STATUS': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}}
_ALLOWED_ATTRIBUTES = ['meta', 'config', 'EXTENSIONS']
_DEFAULT_INSTRUMENT = 'HISPEC'
_INHERIT_EXTENSIONS = True
_config = None
classmethod _doc_default_detector_size()[source]

Resolve (ny, nx) purely to render real numbers in a generated extensions table’s Dimensions column (see extensions_table()) – from cls._DEFAULT_INSTRUMENT’s config, since this runs from __init_subclass__() at class-definition time, before any instance (and so any live self.config/self.default_detector_size, which a multi-detector model could have overridden) exists.

Deliberately does not go through utils.get_instrument_config() (which reads INSTRUMENT_CONFIGS, a dict this very package’s __init__.py only builds after importing every model class – every __init_subclass__ call happens before it exists). Instead imports <instrument>_config directly by name, matching this package’s own hispec_config.py/parvi_config.py convention – that submodule only needs base_model/meta_data, already loaded by the time any model class is being defined, regardless of how far this package’s own __init__.py has otherwise gotten.

Never raises; None (axes stay symbolic, e.g. Ny/Nx) if the default instrument’s config can’t be resolved this way or has no DETECTOR_SIZE.

classmethod _expand_enumerated_extensions(extensions, num_fibers)[source]

Expand any extension marked enumerate=<tag> (see extensions.IMAGE_HDU() / extensions.TABLE_HDU()) into one <NAME>_<tag>{n} entry per fiber – the format is always prefix_{enumerate}# – for n in 1..num_fibers. Extensions without enumerate pass through unchanged.

_extend_data_ancestors_columns(new_columns)[source]
_extension_missing_or_empty(extname)[source]

True if extension data is absent or an empty BinTable.

Empty astropy.table.Table (0 rows) must be treated like missing when building FITS output with include_empty_extensions=False: Astropy cannot serialize 0-row tables that include object columns (e.g. variable-length CCF arrays) because no row exists to infer inner dtypes (astropy.io.fits.table_to_hdu() / ColDefs._init_from_array).

Reads the raw (un-auto-initialized) value: this is only called when include_empty_extensions=False, in which case an image extension that was never explicitly set should be treated as missing rather than lazily materialized into a default array just to decide whether to save it.

classmethod _extensions_rows()[source]

Shared data behind both extensions_table() (plain aligned text, for standalone/interactive use, e.g. help()) and the RST table spliced into the class docstring by __init_subclass__() – one source of truth read straight from cls.EXTENSIONS, so neither can drift from it the way a hand-written docstring table could. title=/units= (see extensions.IMAGE_HDU()/ extensions.TABLE_HDU()) fill the Description/Units columns; everything else (name, HDU type, dtype, shape) is read directly off the extension. DETECTOR_SIZE_Y/DETECTOR_SIZE_X render as real numbers (see _doc_default_detector_size()); other symbolic axes (N_ORDERS, etc.) stay symbolic since they have no single fixed doc-time answer.

Returns:
  • headers (list[str])

  • rows (list[list[str]])

  • tag_notes (list[str]) – One explanatory line per enumerate tag in use, e.g. “‘_FIB#’ in an HDU name denotes the fiber number.”

static _generate_filename(instrument_alias, datetime_obs, suffix)[source]

Generate a filename for a non-calibration datamodel. This method should not be called by the user. Instead, call self.generate_filename which will determine the instrument alias and suffix automatically based on the datamodel’s metadata and config. Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}_{suffix}.fits

Examples

  • HISPEC (BSPEC):

    ‘HB.20240924.00000.00_rate-darksub.fits’

  • HISPEC (RSPEC):

    ‘HR.20240924.00000.00_rate.fits’

  • PARVI:

    ‘PR.20240924.00000.00_rate.fits’

Parameters:
  • instrument_alias (str) – The two letter instrument alias.

  • datetime_obs (str) – The ISO date and time of the observation.

  • suffix (str) – The suffix to append to the filename. Required.

Returns:

str – The filename

_generate_output_path(filename=None, output_dir=None, suffix=None)[source]
classmethod _get_extensions(num_fibers)[source]
classmethod _get_instrument_config(meta, filepath, instrument_name=None)[source]
_infer_shape_symbols()[source]

Scan every currently-set image extension and, for each, match its real array shape positionally against its own declared shape= spec (see extensions.IMAGE_HDU(); populated for every extension, even one that omitted it itself, by _populate_shape_info() at construction time), recording {symbol: size} for every non-literal, non-ANY symbol position.

This is what lets e.g. an explicitly-assigned model.bias = np.ones((10, 10)) inform the shape used to auto-init model.ktc, model.dq, etc. on that same model – nothing is hardcoded about “bias” being special; any extension whose declared shape shares a symbol with another can inform it, as long as it’s actually been set. Also the only source N_READS (no independent resolver – see _resolve_shape_axis()) ever has.

_make_default_array(extname, info)[source]

Lazily build the default array for an image extension, if eligible.

If the extension declares 'default' (see extensions.IMAGE_HDU()) and its 'shape' spec (populated for every extension by _populate_shape_info() at construction time) fully resolves (see _resolve_shape()), the array is built directly as np.full(resolved_shape, default, dtype=...). Otherwise, building it is delegated to make_default_array() – tried unconditionally, whether or not default was given – which subclasses can override for extensions whose shape can’t be expressed with the available symbols (e.g. polynomial coefficients, a per-instance fit result).

static _make_order_vec(order_start, order_end)[source]

Create an order vector based on the order start and end metadata. If order_end < order_start, the vector will be in descending order.

Parameters:
  • order_start (int) – The starting order number.

  • order_end (int) – The ending order number.

Returns:

np.ndarray – An array of integers from order_start to order_end.

classmethod _populate_shape_info(extensions)[source]

Fill in 'shape' for any image extension that omitted it (see extensions.IMAGE_HDU()’s shape=None), copying the primary extension’s declared shape – the first key not in the reserved special names (PRIMARY, RECEIPT, DATA_ANCESTORS, CALIBRATIONS), matching the base primary_extension_name default. A copy of the info dict is only made where a fill-in is actually needed, so shared info dicts (e.g. DQ_L0, reused as-is across many models) are never mutated in place.

A model with a dynamic primary_extension_name override (returning None until some fiber’s data is actually set) can’t rely on this – it always declares every extension’s own shape= explicitly, so there’s nothing left here to fill in for it.

_raw_extension_value(extname)[source]

Read an extension’s current value without triggering lazy auto-initialization.

_resolve_extensions(extensions=None)[source]
_resolve_shape(shape_spec)[source]

Resolve a full shape= spec (see extensions.IMAGE_HDU()) to a concrete shape tuple, or None if any axis can’t be resolved.

_resolve_shape_axis(axis, inferred)[source]

Resolve one axis of a shape= spec (see extensions.IMAGE_HDU()) to a concrete size, or None if it can’t be resolved on this model right now.

A string axis is matched case-insensitively against the recognized symbolic names (DETECTOR_SIZE_Y, DETECTOR_SIZE_X, N_ORDERS, N_FIBERS, N_READS, ANY). Checked in order: (1) inferred (see _infer_shape_symbols()); (2) the symbol’s own independent source, if it has one – DETECTOR_SIZE_Y/ DETECTOR_SIZE_X fall back to the constructor’s shape= override, then default_detector_size; N_ORDERS to meta.order_start/order_end; N_FIBERS to config.num_fibers. N_READS and any other symbol with no independent source are only ever resolvable via inferred. ANY is never resolvable at all, by design.

_save_fits(hdulist, **kwargs)[source]
config = None
copy()[source]

Create a deep copy of the DataModel object.

property data_dict
property data_level_suffix
property default_detector_size

This model’s (ny, nx) detector plane size – the last-resort source for the DETECTOR_SIZE_Y/DETECTOR_SIZE_X shape-symbol resolvers (see _resolve_shape_axis()) when no extension’s already-set real data and no explicit constructor shape= override provide it instead.

Defaults to config.DETECTOR_SIZE, which is all a single-detector instrument ever needs. An instrument with multiple, differently sized detectors has no single automatic answer – override this property (in a subclass, or an intermediate base shared by that detector’s models) to compute the right one from whatever instance state actually distinguishes them, e.g.:

@property
def default_detector_size(self):
    return self.config.DETECTOR_SIZES.get(self.meta.instrument_mode)
classmethod extensions_table()[source]

Render the “HDU Name / HDU Type / Data Type / Dimensions / Units / Description” table from _extensions_rows() as plain, whitespace-aligned text – the layout help() and other interactive tools show. Spliced into the class docstring (as a proper RST table, not this plain-text form) by __init_subclass__(); call this directly only if you need the plain-text table standalone (e.g. to print it, or embed it elsewhere). Returns '' when there are no non-special extensions to show, e.g. for an abstract intermediate base class.

property extinfo
classmethod from_datamodels(dataset, append_receipts=True, ancestor_rule='these', append_calibrations=True, include_data=False, primitive=None, **kwargs)[source]

Create a new DataModel by merging metadata from multiple input models.

Parameters:
  • dataset (DataModel | str | list[DataModel | str] | dict[str, DataModel | str]) – Input DataModel(s) or file path(s) to merge.

  • append_receipts (bool, optional) – Whether to append receipts from input models. Defaults to True.

  • ancestor_rule (str, optional) –

    Rule for handling data ancestors. Defaults to these.

    • these: Add the input models themselves to the data ancestors of the output model, but do not copy their ancestors.

    • append: Copy the data ancestors from the input models to the output model, but do not include the input models themselves as ancestors.

    • none: Do not copy ancestors or add input models as ancestors.

  • append_calibrations (bool, optional) – Whether to append calibrations from input models. Defaults to True.

  • include_data (bool, optional) – Whether to include data from the input model (must only be one). Defaults to False.

Returns:

DataModel – A new DataModel with merged metadata and optional data.

generate_filename(suffix=None)[source]

Generate a filename for the datamodel. Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}_{suffix}.fits

  • Instrument aliases:

    • HISPEC (BSPEC): HB

    • HISPEC (RSPEC): HR

    • PARVI: PR

  • SSSSS.SS is the seconds of the day with two decimal places, calculated from the datetime_obs metadata attribute.

  • suffix is an optional string that can be appended to the end of the filename, separated by an underscore. This can be used to further specify any processing steps or other special information about the datamodel. The provided suffix suffix will automatically be appended to datamodel’s data_level_suffix attribute with a hyphen. Ex: suffix='darksub' would produce a suffix ‘rate-darksub’ for a level 0 model.

Stores the generated filename in self.meta.filename. Sets self._filepath to None since we do not know the

Parameters:

suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.

get_calibration_from_role(cal_role)[source]

Return the row from the DataModel’s calibrations table.

Parameters:

cal_role (str) – The role of the calibration to load.

Returns:

dict – The row from the DataModel’s calibrations table.

get_output_path(filename=None, output_dir=None, suffix=None)[source]

Return the FITS path that save() would write for output_dir / suffix.

classmethod has_extension(extname)[source]

Whether extname (case-insensitive) is a valid extension for this class.

Cheap: never rebuilds the enumerated-expanded EXTENSIONS dict (see _expand_enumerated_extensions()) – pattern-matches <BASE>_<tag>{n} (the format is always prefix_{enumerate}#) against cls.EXTENSIONS’s enumerate-marked base entries instead.

A _<tag>{n} suffix is bounds-checked against cls.config.num_fibers when cls.config is set (see the config classproperty and datamodels.utils.set_instrument()); otherwise any _<tag>{n} suffix on an enumerate-marked base is accepted without a bound.

has_extension_value(extname)[source]

Whether extname (case-insensitive) is both a valid extension for this model and currently holds a value.

Reads the raw (un-auto-initialized) value. This must not trigger lazy auto-initialization: several primary_extension_name overrides call this to find which fiber has actually been set, and lazy auto-initialization itself depends on primary_extension_name (via model.shape), which would recurse infinitely.

init_data(correct_dtypes=False, **kwargs)[source]

Initialize the data.

Parameters:
  • correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays upon initializing them. Defaults to False.

  • kwargs – Any valid data attributes to set.

init_meta(meta=None, validate=True)[source]

Initialize the metadata and optionally merge in provided metadata.

Parameters:
  • meta (dict or MetaModel, optional) – Existing metadata to store or merge.

  • validate (bool, optional) – Whether the resulting MetaModel validates values against the schema on every assignment. Only applies when a new MetaModel is built here (from a dict or None) – an already-built MetaModel passed in via meta keeps its own validate setting as-is.

property instrument_name
load_calibration_from_role(cal_role)[source]

Load a calibration from the DataModel’s calibrations table.

Parameters:

cal_role (str) – The role of the calibration to load.

Returns:

dict – The calibration record.

make_default_array(extname, info)[source]

Build a default array for an image extension whose shape can’t be resolved from its declared shape= spec alone.

Called by _make_default_array() whenever the automatic np.full(resolved_shape, default, dtype=...) path doesn’t apply – e.g. an extension containing ANY in its shape= (like polynomial coefficients, whose axis sizes are a per-instance fit result with no symbol at all), or one with no default given.

Override in a subclass to handle specific extension names, falling back to super().make_default_array(extname, info). The base implementation returns None (no custom default), so models that don’t override this see no behavior change.

Parameters:
make_hdu(extname, include_missing_metadata=False, correct_dtypes=False)[source]

Create an HDU for an extension name.

Parameters:
  • extname (str) – The extension name.

  • include_missing_metadata (bool, optional) – Whether to add missing metadata to the header. Defaults to False.

  • correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays. Defaults to False.

make_order_vec()[source]
migrate()[source]

Apply schema migrations to this model’s current data, in place.

Useful for upgrading an already-constructed model – e.g. one built from kwargs or merged via from_datamodels() – not just data freshly read from a FITS file.

Returns:

Self – The updated model.

property n_orders

Return the number of orders in the model based on the order_start and order_end metadata.

property num_fibers
on_init()[source]

Hook called after the DataModel is initialized.

on_save()[source]
Hook called before a datamodel is saved. The default implementation:
  1. Updates self.meta.filename with base filename.

order_index(order)[source]

Return the row index of order in make_order_vec().

property orders

Echelle order vector from meta.order_start / order_end.

property primary_extension_name

‘PRIMARY’, ‘RECEIPT’, ‘DATA_ANCESTORS’, or ‘CALIBRATIONS’.

Type:

Return the name of the primary extension for this datamodel. By default, this is the first extension declared within the EXTENSIONS dictionary that is not one of the special extensions

read_data(extensions=None, do_not_scale_image_data=None, correct_dtypes=False)[source]

Read the data from the FITS file.

Parameters:
  • extensions (str or list of str, optional) – List of extensions to read. Defaults to None.

  • do_not_scale_image_data (bool or None, optional) – Whether to not scale data according to BZERO and BSCALE. See astropy behavior.

  • correct_dtypes (bool, optional) – Whether to correct the data types of each data format to the specified data format.

read_extension_data(hdu, correct_dtype=False)[source]

Read data from a specific HDU.

Parameters:
  • hdu (_VALID_HDU) – The HDU to read data from.

  • correct_dtype (bool, optional) – Whether to correct the data type of the data. Defaults to False.

Returns:

data (np.ndarray or Table) – If an ImageHDU, returns a numpy ndarray. If a BinTableHDU, returns an astropy Table.

read_meta_from_fits_headers(schema)[source]

Read metadata from the FITS headers.

Returns:

meta_dict (dict) – Dictionary of metadata read from the FITS headers. Keys are property names, values are the corresponding header values.

save(filepath=None, filename=None, output_dir=None, suffix=None, include_missing_metadata=False, include_empty_extensions=False, correct_dtypes=False, **kwargs)[source]

Save the DataModel to a FITS file.

Parameters:
  • filepath (str, optional) – The filepath to save to. Defaults to self._filepath.

  • filename (str, optional) – The filename to save to. Defaults to None.

  • output_dir (str, optional) – The directory to save to. Defaults to None.

  • suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.

  • include_missing_metadata (bool, optional) – Whether to add missing metadata to the header. Defaults to False.

  • include_empty_extensions (bool, optional) – Whether to include empty extensions in the output. Defaults to False.

  • correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays. Defaults to False.

  • **kwargs – Additional keyword arguments to pass to the fits.HDUList.writeto method.

property shape

The shape of the model’s primary extension.

Reads the primary extension’s raw value (never triggers lazy auto-initialization). If the primary extension has no real array yet, falls back to the shape= override passed to the constructor, else None.

Note this is unrelated to auto-init’s own shape resolution (see _resolve_shape_axis()), which no longer depends on this property or primary_extension_name at all – it resolves each symbol (DETECTOR_SIZE_Y, N_ORDERS, etc.) independently, so this property stays a simple, general-purpose reflection of “what’s actually been assigned to the primary extension” for any other code that reads model.shape directly.

to_hdulist(include_empty_extensions=False, include_missing_metadata=False, correct_dtypes=False)[source]

Write the DataModel to a FITS file.

Parameters:
  • include_empty_extensions (bool, optional) – Whether to include empty extensions in the output. Defaults to False.

  • include_missing_metadata (bool, optional) – Whether to add missing metadata to the header. Defaults to False.

  • correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays. Defaults to False.

Returns:

str – The filepath the DataModel was saved to.

update_calibration_table(calibration, note=None)[source]
class HISPECCalibrationStore(*args, instrument_name=None, connect_remote=None, **kwargs)[source]

Bases: CalibrationStore

_convert_origin(cal, origin)[source]

Convert an existing calibration file’s origin (record and file) to a new origin. This creates a new file and record. This does not modify the input file or record.

TODO: Need to generate new claibration ID.

Parameters:
  • cal (str | CalibrationModel) – The calibration file or model to copy the origin from.

  • origin (str) – The new origin string to set.

Returns:

str – The standardized origin string.

_existing_calibration_path_and_record(existing)[source]

Return (path, record) for a pre-existing cache row, or (None, None).

Upstream calibration_record_in_cache(..., mode="version-family") returns a list[dict] when multiple version-family matches exist (default include_version=False). Normalize to a single record before path lookup.

_finalize_cal_record(cal, cal_record=None, local_filepath=None)[source]

Finalize a calibration record after the file has been saved to disk.

Adds filename/MD5 via the base store, then fills KOA identifiers that are set on the saved file during generate_filename() but are not yet present on the pre-save metadata record.

Compatible with both koa_middleware call signatures:

  • _finalize_cal_record(cal, cal_record, local_filepath) (current git)

  • _finalize_cal_record(cal_record, local_filepath) (older releases)

calibration_record_in_cache(cal, mode='id')[source]

Checks if a calibration is already present in the local cache.

Parameters:
  • calibration (dict | str | DataModel) –

    Can be one of:
    • str : A calibration ID string or filepath.

    • dict : A calibration metadata dict.

    • DataModel : A calibration data model instance.

  • mode (str) –

    The mode to check the cache. Can be one of:
    • ’id’ : Check by calibration ID (cal_id), the primary key in the database.

    • ’version-family’ : Check by the version family (cal_type, datetime_obs, master_cal, spectrograph) + version (cal_version).

    • ’md5’ : Check by the MD5 checksum of the calibration file (Not Implemented yet!).

Returns:

dict | None – The calibration metadata record if found, otherwise None.

download_calibration_file(calibration)[source]

Download a remote calibration into data_dir under the DB filename.

Wraps koa_middleware so a zip that uses a different member name is moved to the expected cache path, then verifies the file exists and is non-trivial before returning.

generate_calibration_version(calibration, origin=None)[source]

Generate the next version string; accept a model or a metadata dict.

Upstream koa_middleware assigns calibration['origin'] and therefore only works with dicts. Datamodel callers (and generate_calibration_version()) pass the model itself, so convert to a record first and write the version back onto the model when applicable.

get_calibration(cal)[source]

Retrieve a calibration file and verify it is present on disk.

get_version_family_column_names(cal_type)[source]

Retrieves the column names for the version family attributes. This takes in cal_type in case we want to have different version family fields for different calibration types in the future.

Parameters:

cal_type (str) – The type of calibration (e.g. ‘dark’, ‘flat’, etc.) to get the version family fields for.

register_calibration(cal, origin=None, new_version=False)[source]

Registers a calibration to the local cache and metadata database.

Parameters:
  • cal (CalibrationModel | str) – The filepath or Calibration Datamodel object to register.

  • origin (str, optional) – The origin to register the calibration under.

  • new_version (bool, optional) – Whether to generate a new version for this calibration. If False, the method will check if a calibration with the same version family already exists in the cache and skip registration if so. Defaults to False.

Returns:

tuple[str, dict] –

A tuple containing:
  • str: The local file path where the calibration was saved.

  • dict: The calibration metadata dictionary as added to the database.

Notes

Upstream koa_middleware.CalibrationStore.register_calibration snapshots cal.to_record() before cal.save() and discards the return value of generate_calibration_version, so filename / koa_id / cal_version can be stored as None even though the FITS file is written. This override assigns those fields on the model and record before insert, and returns any existing version-family match instead of (None, None).

save_calibration_file(cal, cal_record=None)[source]

Saves a calibration file to the local cache directory.

Parameters:
  • cal (CalibrationModel) – The calibration data model instance to save.

  • cal_record (dict | None) – The corresponding record.

Returns:

str – The absolute local file path where the calibration file was saved.

select_and_get_calibration(input, selector)[source]

Selects the best calibration based on input data and a selection rule, then retrieves it.

This method uses a CalibrationSelector to identify the most appropriate calibration for the given input data. Once selected, it retrieves the calibration file, downloading it if it’s not already cached locally.

Parameters:
  • input – The input data product for which a calibration is needed.

  • selector (CalibrationSelector) – An instance of a CalibrationSelector class.

Returns:
  • tuple[str, dict] –

    • str: The local file path of the retrieved calibration file.

    • dict: The record of the selected calibration from the local database.

  • Example – >>> # Assuming my_input_data and my_selector are defined >>> local_filepath, calibration_record = store.select_and_get_calibration(my_input_data, my_selector) >>> print(f”Calibration file: {local_filepath}”) >>> print(f”Calibration ID: {calibration_record[‘id’]}”)

sync_records_from_cached_files(cals=None)[source]

Populates the local database from existing cached calibration files.

Parameters:

cals (str | CalibrationModel | Sequence[str | CalibrationModel]) – A single calibration metadata dictionary or a data model instance, or a list of these.

Notes

This method may be removed in the future if not found useful.

sync_records_from_remote(cals=None, mode='id')[source]

Synchronize the local DB with the remote KOA catalog.

cals is accepted for compatibility with callers that pass an explicit remote query result; when omitted, records are discovered via get_missing_records (same as koa_middleware).

Remote metadata failures (e.g. KOA API 404) are logged and treated as a no-op so pipelines can continue with the local caldb/cache. Callers that need a hard failure should inspect logs or sync explicitly with their own error handling.

class MetaModel(schema, meta=None, validate=True)[source]

Bases: MutableMapping

Small helper class for handling metadata based on JSON schema.

Parameters:
  • schema (dict) – The metadata schema.

  • meta (dict | None, optional) – Initial metadata values. Defaults to None.

  • validate (bool, optional) – Whether to validate metadata values against the schema. Defaults to True.

boolean_columns()[source]

Return a set of columns that are logically booleans (for PostgreSQL)

property fields

Return a view of metadata fields names, including those with None values.

group_meta_by_section(flat=False)[source]

Group metadata properties by their FITS section.

Parameters:

flat (bool, optional) – Whether to return a flat dictionary instead of grouped by section. Defaults to False.

Returns:

dict – Dictionary of metadata grouped by FITS section. Keys are section names and values are dictionaries of metadata properties. Sections are ordered as they first appear in the metadata schema YAML file (self._schema['properties'] preserves that order).

pprint(show_none=False, header_style=True)[source]

Pretty-print the metadata, grouped by FITS section.

Parameters:

show_none (bool, optional) – Whether to show metadata fields with None values. Defaults to False.

set_default(prop_name, value)[source]

Set a default value for a specific metadata field.

Parameters:
  • prop_name (str) – The property name for which to set the default value.

  • value (Any) – The default value to set if the property is not already set.

set_default_values()[source]

Set default values for metadata fields based on the schema.

to_fits_header(extname='PRIMARY', include_missing_metadata=False)[source]

Convert the metadata to a FITS header.

Parameters:
  • extname (str, optional) – The extension name for which to create the header. Defaults to ‘PRIMARY’.

  • include_missing_metadata (bool, optional) – Whether to include None (unset) metadata values in the header. Defaults to False.

Returns:

header (fits.Header) – The FITS header containing the metadata.

to_record()[source]

Convert the metadata to a dictionary record.

Returns:

record (dict) – The metadata as a dictionary.

validate(prop_name, value)[source]

Validate a metadata field against the schema.

Parameters:
  • prop_name (str) – The property name to validate.

  • value (Any) – The value to validate.

Raises:

ValueError – If the value does not conform to the schema.

_append_ancestors(target, other)
_append_calibrations(target, other)
_append_receipts(target, other)
_build_empty_table_with_cols(extdef)[source]

Build a 0-row astropy.table.Table from an extension’s 'columns' spec.

_canonical_dtype(dtype_spec)[source]

The dtype to construct/cast to: the first entry if dtype_spec is a tuple (see extensions.IMAGE_HDU()), else dtype_spec itself.

class _classproperty(fget)[source]

Bases: object

A read-only property accessible on the class itself, not just instances.

Deliberately a non-data descriptor (no __set__): an instance’s own __dict__ entry (e.g. self.config set in __init__) always takes priority over this on that instance. This is only consulted for class-level access (SomeModel.config) or from within a classmethod (cls.config), where there is no instance to hold a per-instance value.

_correct_image_dtype(data, dtype_spec, extname='')[source]

Cast an image extension’s array to its declared dtype, if needed.

dtype_spec is an extension’s declared 'dtype' entry (see extensions.IMAGE_HDU()): a single numpy scalar type, or a precedence-ordered tuple of them for extensions with more than one legitimate dtype (e.g. RampModel.DATA: uint16 for real detector data, float32 for simulations run with convert_to_uint16=False).

If data’s dtype already matches one of the declared choices (subtype-wise), it’s returned unchanged. Otherwise the declared choice sharing data’s broad numeric family (see _dtype_family() – e.g. any integer array matches a declared integer dtype, any float array matches a declared float dtype) is used as the cast target, falling back to the first (canonical) declared dtype if none share that family.

_doc_table_render_axis(axis, detector_size=None)[source]
_doc_table_render_dtype(dtype)[source]
_doc_table_render_dtype_name(dtype)[source]
_doc_table_render_shape(shape, detector_size=None)[source]
_dtype_family(dtype)[source]

Broad numeric family of a dtype, for kind-aware auto-casting.

Numpy distinguishes signed (‘i’) and unsigned (‘u’) integers, but for auto-casting purposes any integer array should match a declared integer dtype, and any float array should match a declared float dtype.

_set_datetime_meta(model)[source]
_set_koa_meta(model)[source]
_set_target_meta(model, gaia_id=None)[source]
convert_endianness(arr, output_endianness='little')[source]
datamodel_open(init, copy=False, config=None, **kwargs)

Creates a DataModel from a FITS file. Automatically select the right DataModel class based on the ‘DATAMODL’ header keyword.

Parameters:
  • init – filepath (str): Full path to FITS file. DataModel: Return the given DataModel instance or make a copy if copy=True.

  • copy (bool) – If True, return a copy of the DataModel instance. If init is a filepath, this is ignored.

  • config (type, optional) – The instrument configuration class to use for the DataModel.

  • kwargs – Additional keyword arguments for the DataModel constructor.

Returns:

DataModel – The specific DataModel instance.

get_instrument_config(input=None, instrument_name=None)[source]

Get the instrument configuration class from either a FITS filename or an HDUList.

Parameters:
  • input (str or fits.HDUList, optional) – Either a FITS filename path or an already-opened HDUList object. The INSTRUME keyword will be read from the primary header.

  • instrument_name (str, optional) – The instrument name. If provided, this will be used instead of reading the instrument name from the FITS header.

Returns:

config_class (type) – The instrument config class if found.

Raises:
  • TypeError – If input is neither a string nor an HDUList.

  • ValueError – If no INSTRUME keyword is found in the header, or if the instrument config is not found in INSTRUMENT_CONFIGS.

get_meta_schema(instrument_name=None, config=None)[source]

Get the metadata schema for a given instrument.

Parameters:
  • instrument_name (str | None) – Name of the instrument. Optional.

  • config (type | None) – Instrument configuration class. Optional.

Returns:

meta (dict) – The metadata schema with defaults applied.

merge_meta(models)[source]

Merge the meta information from multiple DataModel instances.

Parameters:

models (list[DataModel]) – List of DataModel instances to merge meta information from.

Returns:

meta_merged (MetaModel) – Merged metadata.

migrate_receipt_table(table)[source]

Upgrade a RECEIPT table to the current schema by adding any missing columns.