hispecdrp.datamodels.base_model
Functions
|
Build a 0-row |
|
The dtype to construct/cast to: the first entry if dtype_spec is a tuple (see |
|
Cast an image extension's array to its declared dtype, if needed. |
|
|
|
|
|
|
|
|
|
Broad numeric family of a dtype, for kind-aware auto-casting. |
Classes
|
|
|
Base class for data models. |
|
A read-only property accessible on the class itself, not just instances. |
- class CalibrationModel(filepath=None, meta=None, meta_only=False, extensions=None, correct_dtypes=False, config=None, validate_meta=True, instrument_name=None, shape=None, **kwargs)[source]
Bases:
DataModel- EXTENSIONS = {'CALIBRATIONS': {'columns': {'CAL_ROLE': <class 'str'>, 'CAL_TYPE': <class 'str'>, 'CLASS': <class 'str'>, 'DATETIME_OBS': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'ID': <class 'str'>, 'LAST_PROCESSED': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'DATA_ANCESTORS': {'columns': {'CLASS': <class 'str'>, 'FILENAME': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'PRIMARY': {'dtype': None, 'hdu_type': <class 'astropy.io.fits.hdu.image.PrimaryHDU'>}, 'RECEIPT': {'columns': {'CLASS': <class 'str'>, 'CONFIG_FILE': <class 'str'>, 'DRP_GIT_COMMIT_ID': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'PARAMETERS': <class 'str'>, 'PROC_END_TIME': <class 'str'>, 'PROC_START_TIME': <class 'str'>, 'STATUS': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}}
- static _generate_filename(instrument_alias, datetime_obs, cal_type, master_cal=False, cal_version=None, origin=None, suffix=None)[source]
Create a filename for a calibration data product.
- Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}.[master-]{cal_type}.{cal_version}[_{suffix}].fits
datetime_obs : YYYYMMDD.SSSSSS.SS. Where SSSSSS.SS is the number of seconds since midnight. cal_type : The type of calibration data. master_cal : Insert ‘master-’ before the cal_type if master_cal=True. cal_version : The version of the calibration file. If None, queries the local cache for the latest version if
auto_version=True. suffix : optional suffix
Examples
‘HB.20240924.00000.00.dark.001.fits’ ‘HB.20240924.00000.00.dark.001-local.fits’ ‘HB.20240924.00000.00.master-dark.001.fits’ ‘HB.20240924.00000.00.master-dark.001_custom.fits’ ‘HB.20240924.00000.00.master-dark.001-local_custom.fits’ ‘HB.20240924.00000.00.dark.001_custom.fits’
- Parameters:
datetime_obs (str) – The date and time of the observation.
instrument_alias (str) – The two letter instrument alias.
cal_type (str) – The type of calibration data.
master_cal (bool, optional) – If True, insert ‘master-’ before the cal_type. Defaults to False.
cal_version (str, optional) – The version of the calibration file. If None, queries the local cache.
origin (str, optional) – The origin of the calibration file, either ‘LOCAL’ or ‘KECK’. If None,
suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.
- Returns:
filename (str) – The generated filename.
- generate_calibration_id()[source]
Generate the calibration ID which is reproducible UUID 7.
- Returns:
cal_id (str) – The calibration ID
- generate_calibration_version(origin=None)[source]
- generate_filename(suffix=None, origin=None, auto_version=None)[source]
- Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}.[master-]{cal_type}.{cal_version}[_{suffix}].fits
datetime_obs : YYYYMMDD.SSSSSS.SS. Where SSSSSS.SS is the number of seconds since midnight. cal_type : The type of calibration data. master_cal : Insert ‘master-’ before the cal_type if master_cal=True. cal_version : The version of the calibration file. If None, queries the local cache for the latest version if
auto_version=True. suffix : optional suffix
- Parameters:
- Returns:
filename (str) – The generated filename.
- on_init()[source]
- order_index(order)[source]
Return the index of the given order in the order vector.
- Parameters:
order (int) – The order number to find the index for.
- Returns:
int – The index of the order in the order vector.
- class DataModel(filepath=None, meta=None, meta_only=False, extensions=None, correct_dtypes=False, config=None, validate_meta=True, instrument_name=None, shape=None, **kwargs)[source]
Bases:
objectBase class for data models.
- Parameters:
EXTENSIONS (dict) – Extensions and their properties.
_INHERIT_EXTENSIONS (bool)
True. (Whether to inherit extensions from parent class. Defaults to) – Whether to inherit extensions from parent class. Defaults to True.
_ALLOWED_ATTRIBUTES (list[str]) – List of attributes that can be set directly on the DataModel.
_DEFAULT_INSTRUMENT (str) – Default instrument name for the DataModel.
- EXTENSIONS = {'CALIBRATIONS': {'columns': {'CAL_ROLE': <class 'str'>, 'CAL_TYPE': <class 'str'>, 'CLASS': <class 'str'>, 'DATETIME_OBS': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'ID': <class 'str'>, 'LAST_PROCESSED': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'DATA_ANCESTORS': {'columns': {'CLASS': <class 'str'>, 'FILENAME': <class 'str'>, 'TIME_APPLIED': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}, 'PRIMARY': {'dtype': None, 'hdu_type': <class 'astropy.io.fits.hdu.image.PrimaryHDU'>}, 'RECEIPT': {'columns': {'CLASS': <class 'str'>, 'CONFIG_FILE': <class 'str'>, 'DRP_GIT_COMMIT_ID': <class 'str'>, 'DRP_VERSION': <class 'str'>, 'FILENAME': <class 'str'>, 'PARAMETERS': <class 'str'>, 'PROC_END_TIME': <class 'str'>, 'PROC_START_TIME': <class 'str'>, 'STATUS': <class 'str'>}, 'hdu_type': <class 'astropy.io.fits.hdu.table.BinTableHDU'>}}
- classmethod _doc_default_detector_size()[source]
Resolve
(ny, nx)purely to render real numbers in a generated extensions table’s Dimensions column (seeextensions_table()) – fromcls._DEFAULT_INSTRUMENT’s config, since this runs from__init_subclass__()at class-definition time, before any instance (and so any liveself.config/self.default_detector_size, which a multi-detector model could have overridden) exists.Deliberately does not go through
utils.get_instrument_config()(which readsINSTRUMENT_CONFIGS, a dict this very package’s__init__.pyonly builds after importing every model class – every__init_subclass__call happens before it exists). Instead imports<instrument>_configdirectly by name, matching this package’s ownhispec_config.py/parvi_config.pyconvention – that submodule only needsbase_model/meta_data, already loaded by the time any model class is being defined, regardless of how far this package’s own__init__.pyhas otherwise gotten.Never raises;
None(axes stay symbolic, e.g.Ny/Nx) if the default instrument’s config can’t be resolved this way or has noDETECTOR_SIZE.
- classmethod _expand_enumerated_extensions(extensions, num_fibers)[source]
Expand any extension marked
enumerate=<tag>(seeextensions.IMAGE_HDU()/extensions.TABLE_HDU()) into one<NAME>_<tag>{n}entry per fiber – the format is alwaysprefix_{enumerate}#– forn in 1..num_fibers. Extensions withoutenumeratepass through unchanged.
- _extend_data_ancestors_columns(new_columns)[source]
- _extension_missing_or_empty(extname)[source]
True if extension data is absent or an empty BinTable.
Empty
astropy.table.Table(0 rows) must be treated like missing when building FITS output withinclude_empty_extensions=False: Astropy cannot serialize 0-row tables that include object columns (e.g. variable-lengthCCFarrays) because no row exists to infer inner dtypes (astropy.io.fits.table_to_hdu()/ColDefs._init_from_array).Reads the raw (un-auto-initialized) value: this is only called when
include_empty_extensions=False, in which case an image extension that was never explicitly set should be treated as missing rather than lazily materialized into a default array just to decide whether to save it.
- classmethod _extensions_rows()[source]
Shared data behind both
extensions_table()(plain aligned text, for standalone/interactive use, e.g.help()) and the RST table spliced into the class docstring by__init_subclass__()– one source of truth read straight fromcls.EXTENSIONS, so neither can drift from it the way a hand-written docstring table could.title=/units=(seeextensions.IMAGE_HDU()/extensions.TABLE_HDU()) fill the Description/Units columns; everything else (name, HDU type, dtype, shape) is read directly off the extension.DETECTOR_SIZE_Y/DETECTOR_SIZE_Xrender as real numbers (see_doc_default_detector_size()); other symbolic axes (N_ORDERS, etc.) stay symbolic since they have no single fixed doc-time answer.- Returns:
headers (list[str])
rows (list[list[str]])
tag_notes (list[str]) – One explanatory line per enumerate tag in use, e.g. “‘_FIB#’ in an HDU name denotes the fiber number.”
- static _generate_filename(instrument_alias, datetime_obs, suffix)[source]
Generate a filename for a non-calibration datamodel. This method should not be called by the user. Instead, call self.generate_filename which will determine the instrument alias and suffix automatically based on the datamodel’s metadata and config. Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}_{suffix}.fits
Examples
- HISPEC (BSPEC):
‘HB.20240924.00000.00_rate-darksub.fits’
- HISPEC (RSPEC):
‘HR.20240924.00000.00_rate.fits’
- PARVI:
‘PR.20240924.00000.00_rate.fits’
- _generate_output_path(filename=None, output_dir=None, suffix=None)[source]
- classmethod _get_extensions(num_fibers)[source]
- classmethod _get_instrument_config(meta, filepath, instrument_name=None)[source]
- _infer_shape_symbols()[source]
Scan every currently-set image extension and, for each, match its real array shape positionally against its own declared
shape=spec (seeextensions.IMAGE_HDU(); populated for every extension, even one that omitted it itself, by_populate_shape_info()at construction time), recording{symbol: size}for every non-literal, non-ANYsymbol position.This is what lets e.g. an explicitly-assigned
model.bias = np.ones((10, 10))inform the shape used to auto-initmodel.ktc,model.dq, etc. on that same model – nothing is hardcoded about “bias” being special; any extension whose declared shape shares a symbol with another can inform it, as long as it’s actually been set. Also the only sourceN_READS(no independent resolver – see_resolve_shape_axis()) ever has.
- _make_default_array(extname, info)[source]
Lazily build the default array for an image extension, if eligible.
If the extension declares
'default'(seeextensions.IMAGE_HDU()) and its'shape'spec (populated for every extension by_populate_shape_info()at construction time) fully resolves (see_resolve_shape()), the array is built directly asnp.full(resolved_shape, default, dtype=...). Otherwise, building it is delegated tomake_default_array()– tried unconditionally, whether or notdefaultwas given – which subclasses can override for extensions whose shape can’t be expressed with the available symbols (e.g. polynomial coefficients, a per-instance fit result).
- static _make_order_vec(order_start, order_end)[source]
Create an order vector based on the order start and end metadata. If order_end < order_start, the vector will be in descending order.
- classmethod _populate_shape_info(extensions)[source]
Fill in
'shape'for any image extension that omitted it (seeextensions.IMAGE_HDU()’sshape=None), copying the primary extension’s declared shape – the first key not in the reserved special names (PRIMARY,RECEIPT,DATA_ANCESTORS,CALIBRATIONS), matching the baseprimary_extension_namedefault. A copy of the info dict is only made where a fill-in is actually needed, so shared info dicts (e.g.DQ_L0, reused as-is across many models) are never mutated in place.A model with a dynamic
primary_extension_nameoverride (returningNoneuntil some fiber’s data is actually set) can’t rely on this – it always declares every extension’s ownshape=explicitly, so there’s nothing left here to fill in for it.
- _raw_extension_value(extname)[source]
Read an extension’s current value without triggering lazy auto-initialization.
- _resolve_extensions(extensions=None)[source]
- _resolve_shape(shape_spec)[source]
Resolve a full
shape=spec (seeextensions.IMAGE_HDU()) to a concrete shape tuple, orNoneif any axis can’t be resolved.
- _resolve_shape_axis(axis, inferred)[source]
Resolve one axis of a
shape=spec (seeextensions.IMAGE_HDU()) to a concrete size, orNoneif it can’t be resolved on this model right now.A string axis is matched case-insensitively against the recognized symbolic names (
DETECTOR_SIZE_Y,DETECTOR_SIZE_X,N_ORDERS,N_FIBERS,N_READS,ANY). Checked in order: (1)inferred(see_infer_shape_symbols()); (2) the symbol’s own independent source, if it has one –DETECTOR_SIZE_Y/DETECTOR_SIZE_Xfall back to the constructor’sshape=override, thendefault_detector_size;N_ORDERStometa.order_start/order_end;N_FIBERStoconfig.num_fibers.N_READSand any other symbol with no independent source are only ever resolvable viainferred.ANYis never resolvable at all, by design.
- _save_fits(hdulist, **kwargs)[source]
- copy()[source]
Create a deep copy of the DataModel object.
- property default_detector_size
This model’s
(ny, nx)detector plane size – the last-resort source for theDETECTOR_SIZE_Y/DETECTOR_SIZE_Xshape-symbol resolvers (see_resolve_shape_axis()) when no extension’s already-set real data and no explicit constructorshape=override provide it instead.Defaults to
config.DETECTOR_SIZE, which is all a single-detector instrument ever needs. An instrument with multiple, differently sized detectors has no single automatic answer – override this property (in a subclass, or an intermediate base shared by that detector’s models) to compute the right one from whatever instance state actually distinguishes them, e.g.:@property def default_detector_size(self): return self.config.DETECTOR_SIZES.get(self.meta.instrument_mode)
- classmethod extensions_table()[source]
Render the “HDU Name / HDU Type / Data Type / Dimensions / Units / Description” table from
_extensions_rows()as plain, whitespace-aligned text – the layouthelp()and other interactive tools show. Spliced into the class docstring (as a proper RST table, not this plain-text form) by__init_subclass__(); call this directly only if you need the plain-text table standalone (e.g. to print it, or embed it elsewhere). Returns''when there are no non-special extensions to show, e.g. for an abstract intermediate base class.
- classmethod from_datamodels(dataset, append_receipts=True, ancestor_rule='these', append_calibrations=True, include_data=False, primitive=None, **kwargs)[source]
Create a new DataModel by merging metadata from multiple input models.
- Parameters:
dataset (DataModel | str | list[DataModel | str] | dict[str, DataModel | str]) – Input DataModel(s) or file path(s) to merge.
append_receipts (bool, optional) – Whether to append receipts from input models. Defaults to True.
ancestor_rule (str, optional) –
Rule for handling data ancestors. Defaults to
these.these: Add the input models themselves to the data ancestors of the output model, but do not copy their ancestors.append: Copy the data ancestors from the input models to the output model, but do not include the input models themselves as ancestors.none: Do not copy ancestors or add input models as ancestors.
append_calibrations (bool, optional) – Whether to append calibrations from input models. Defaults to True.
include_data (bool, optional) – Whether to include data from the input model (must only be one). Defaults to False.
- Returns:
DataModel – A new DataModel with merged metadata and optional data.
- generate_filename(suffix=None)[source]
Generate a filename for the datamodel. Format: {instrument_alias}.{YYYYMMDD}.{SSSSS.SS}_{suffix}.fits
Instrument aliases:
HISPEC (BSPEC): HB
HISPEC (RSPEC): HR
PARVI: PR
SSSSS.SS is the seconds of the day with two decimal places, calculated from the datetime_obs metadata attribute.
suffixis an optional string that can be appended to the end of the filename, separated by an underscore. This can be used to further specify any processing steps or other special information about the datamodel. The provided suffix suffix will automatically be appended to datamodel’sdata_level_suffixattribute with a hyphen. Ex:suffix='darksub'would produce a suffix ‘rate-darksub’ for a level 0 model.
Stores the generated filename in self.meta.filename. Sets self._filepath to None since we do not know the
- Parameters:
suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.
- get_calibration_from_role(cal_role)[source]
Return the row from the DataModel’s calibrations table.
- Parameters:
cal_role (str) – The role of the calibration to load.
- Returns:
dict – The row from the DataModel’s calibrations table.
- get_output_path(filename=None, output_dir=None, suffix=None)[source]
Return the FITS path that
save()would write foroutput_dir/suffix.
- classmethod has_extension(extname)[source]
Whether
extname(case-insensitive) is a valid extension for this class.Cheap: never rebuilds the enumerated-expanded
EXTENSIONSdict (see_expand_enumerated_extensions()) – pattern-matches<BASE>_<tag>{n}(the format is alwaysprefix_{enumerate}#) againstcls.EXTENSIONS’senumerate-marked base entries instead.A
_<tag>{n}suffix is bounds-checked againstcls.config.num_fiberswhencls.configis set (see theconfigclassproperty anddatamodels.utils.set_instrument()); otherwise any_<tag>{n}suffix on anenumerate-marked base is accepted without a bound.
- has_extension_value(extname)[source]
Whether
extname(case-insensitive) is both a valid extension for this model and currently holds a value.Reads the raw (un-auto-initialized) value. This must not trigger lazy auto-initialization: several
primary_extension_nameoverrides call this to find which fiber has actually been set, and lazy auto-initialization itself depends onprimary_extension_name(viamodel.shape), which would recurse infinitely.
- init_data(correct_dtypes=False, **kwargs)[source]
Initialize the data.
- Parameters:
correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays upon initializing them. Defaults to False.
kwargs – Any valid data attributes to set.
- init_meta(meta=None, validate=True)[source]
Initialize the metadata and optionally merge in provided metadata.
- Parameters:
meta (dict or MetaModel, optional) – Existing metadata to store or merge.
validate (bool, optional) – Whether the resulting
MetaModelvalidates values against the schema on every assignment. Only applies when a newMetaModelis built here (from a dict orNone) – an already-builtMetaModelpassed in viametakeeps its ownvalidatesetting as-is.
- load_calibration_from_role(cal_role)[source]
Load a calibration from the DataModel’s calibrations table.
- Parameters:
cal_role (str) – The role of the calibration to load.
- Returns:
dict – The calibration record.
- make_default_array(extname, info)[source]
Build a default array for an image extension whose shape can’t be resolved from its declared
shape=spec alone.Called by
_make_default_array()whenever the automaticnp.full(resolved_shape, default, dtype=...)path doesn’t apply – e.g. an extension containingANYin itsshape=(like polynomial coefficients, whose axis sizes are a per-instance fit result with no symbol at all), or one with nodefaultgiven.Override in a subclass to handle specific extension names, falling back to
super().make_default_array(extname, info). The base implementation returnsNone(no custom default), so models that don’t override this see no behavior change.- Parameters:
extname (str) – The extension name (uppercase).
info (dict) – The extension’s declared descriptor (see
extensions.IMAGE_HDU()).
- make_hdu(extname, include_missing_metadata=False, correct_dtypes=False)[source]
Create an HDU for an extension name.
- make_order_vec()[source]
- migrate()[source]
Apply schema migrations to this model’s current data, in place.
Useful for upgrading an already-constructed model – e.g. one built from kwargs or merged via
from_datamodels()– not just data freshly read from a FITS file.- Returns:
Self – The updated model.
- property n_orders
Return the number of orders in the model based on the order_start and order_end metadata.
- on_init()[source]
Hook called after the DataModel is initialized.
- on_save()[source]
- Hook called before a datamodel is saved. The default implementation:
Updates self.meta.filename with base filename.
- order_index(order)[source]
Return the row index of
orderinmake_order_vec().
- property primary_extension_name
‘PRIMARY’, ‘RECEIPT’, ‘DATA_ANCESTORS’, or ‘CALIBRATIONS’.
- Type:
Return the name of the primary extension for this datamodel. By default, this is the first extension declared within the EXTENSIONS dictionary that is not one of the special extensions
- read_data(extensions=None, do_not_scale_image_data=None, correct_dtypes=False)[source]
Read the data from the FITS file.
- Parameters:
extensions (str or list of str, optional) – List of extensions to read. Defaults to None.
do_not_scale_image_data (bool or None, optional) – Whether to not scale data according to BZERO and BSCALE. See astropy behavior.
correct_dtypes (bool, optional) – Whether to correct the data types of each data format to the specified data format.
- read_extension_data(hdu, correct_dtype=False)[source]
Read data from a specific HDU.
- Parameters:
hdu (_VALID_HDU) – The HDU to read data from.
correct_dtype (bool, optional) – Whether to correct the data type of the data. Defaults to False.
- Returns:
data (np.ndarray or Table) – If an ImageHDU, returns a numpy ndarray. If a BinTableHDU, returns an astropy Table.
- read_meta_from_fits_headers(schema)[source]
Read metadata from the FITS headers.
- Returns:
meta_dict (dict) – Dictionary of metadata read from the FITS headers. Keys are property names, values are the corresponding header values.
- save(filepath=None, filename=None, output_dir=None, suffix=None, include_missing_metadata=False, include_empty_extensions=False, correct_dtypes=False, **kwargs)[source]
Save the DataModel to a FITS file.
- Parameters:
filepath (str, optional) – The filepath to save to. Defaults to self._filepath.
filename (str, optional) – The filename to save to. Defaults to None.
output_dir (str, optional) – The directory to save to. Defaults to None.
suffix (str, optional) – An optional suffix to append to the filename. Defaults to None.
include_missing_metadata (bool, optional) – Whether to add missing metadata to the header. Defaults to False.
include_empty_extensions (bool, optional) – Whether to include empty extensions in the output. Defaults to False.
correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays. Defaults to False.
**kwargs – Additional keyword arguments to pass to the fits.HDUList.writeto method.
- property shape
The shape of the model’s primary extension.
Reads the primary extension’s raw value (never triggers lazy auto-initialization). If the primary extension has no real array yet, falls back to the
shape=override passed to the constructor, elseNone.Note this is unrelated to auto-init’s own shape resolution (see
_resolve_shape_axis()), which no longer depends on this property orprimary_extension_nameat all – it resolves each symbol (DETECTOR_SIZE_Y,N_ORDERS, etc.) independently, so this property stays a simple, general-purpose reflection of “what’s actually been assigned to the primary extension” for any other code that readsmodel.shapedirectly.
- to_hdulist(include_empty_extensions=False, include_missing_metadata=False, correct_dtypes=False)[source]
Write the DataModel to a FITS file.
- Parameters:
include_empty_extensions (bool, optional) – Whether to include empty extensions in the output. Defaults to False.
include_missing_metadata (bool, optional) – Whether to add missing metadata to the header. Defaults to False.
correct_dtypes (bool, optional) – Whether to correct the data types of the data arrays. Defaults to False.
- Returns:
str – The filepath the DataModel was saved to.
- update_calibration_table(calibration, note=None)[source]
- class HISPECCalibrationStore(*args, instrument_name=None, connect_remote=None, **kwargs)[source]
Bases:
CalibrationStore- _convert_origin(cal, origin)[source]
Convert an existing calibration file’s origin (record and file) to a new origin. This creates a new file and record. This does not modify the input file or record.
TODO: Need to generate new claibration ID.
- Parameters:
cal (str | CalibrationModel) – The calibration file or model to copy the origin from.
origin (str) – The new origin string to set.
- Returns:
str – The standardized origin string.
- _existing_calibration_path_and_record(existing)[source]
Return
(path, record)for a pre-existing cache row, or(None, None).Upstream
calibration_record_in_cache(..., mode="version-family")returns alist[dict]when multiple version-family matches exist (defaultinclude_version=False). Normalize to a single record before path lookup.
- _finalize_cal_record(cal, cal_record=None, local_filepath=None)[source]
Finalize a calibration record after the file has been saved to disk.
Adds filename/MD5 via the base store, then fills KOA identifiers that are set on the saved file during
generate_filename()but are not yet present on the pre-save metadata record.Compatible with both koa_middleware call signatures:
_finalize_cal_record(cal, cal_record, local_filepath)(current git)_finalize_cal_record(cal_record, local_filepath)(older releases)
- calibration_record_in_cache(cal, mode='id')[source]
Checks if a calibration is already present in the local cache.
- Parameters:
calibration (dict | str | DataModel) –
- Can be one of:
str : A calibration ID string or filepath.
dict : A calibration metadata dict.
DataModel : A calibration data model instance.
mode (str) –
- The mode to check the cache. Can be one of:
’id’ : Check by calibration ID (cal_id), the primary key in the database.
’version-family’ : Check by the version family (cal_type, datetime_obs, master_cal, spectrograph) + version (cal_version).
’md5’ : Check by the MD5 checksum of the calibration file (Not Implemented yet!).
- Returns:
dict | None – The calibration metadata record if found, otherwise None.
- download_calibration_file(calibration)[source]
Download a remote calibration into
data_dirunder the DB filename.Wraps
koa_middlewareso a zip that uses a different member name is moved to the expected cache path, then verifies the file exists and is non-trivial before returning.
- generate_calibration_version(calibration, origin=None)[source]
Generate the next version string; accept a model or a metadata dict.
Upstream
koa_middlewareassignscalibration['origin']and therefore only works with dicts. Datamodel callers (andgenerate_calibration_version()) pass the model itself, so convert to a record first and write the version back onto the model when applicable.
- get_calibration(cal)[source]
Retrieve a calibration file and verify it is present on disk.
- get_version_family_column_names(cal_type)[source]
Retrieves the column names for the version family attributes. This takes in cal_type in case we want to have different version family fields for different calibration types in the future.
- Parameters:
cal_type (str) – The type of calibration (e.g. ‘dark’, ‘flat’, etc.) to get the version family fields for.
- register_calibration(cal, origin=None, new_version=False)[source]
Registers a calibration to the local cache and metadata database.
- Parameters:
cal (CalibrationModel | str) – The filepath or Calibration Datamodel object to register.
origin (str, optional) – The origin to register the calibration under.
new_version (bool, optional) – Whether to generate a new version for this calibration. If False, the method will check if a calibration with the same version family already exists in the cache and skip registration if so. Defaults to False.
- Returns:
tuple[str, dict] –
- A tuple containing:
str: The local file path where the calibration was saved.dict: The calibration metadata dictionary as added to the database.
Notes
Upstream
koa_middleware.CalibrationStore.register_calibrationsnapshotscal.to_record()beforecal.save()and discards the return value ofgenerate_calibration_version, sofilename/koa_id/cal_versioncan be stored asNoneeven though the FITS file is written. This override assigns those fields on the model and record before insert, and returns any existing version-family match instead of(None, None).
- save_calibration_file(cal, cal_record=None)[source]
Saves a calibration file to the local cache directory.
- Parameters:
cal (CalibrationModel) – The calibration data model instance to save.
cal_record (dict | None) – The corresponding record.
- Returns:
str – The absolute local file path where the calibration file was saved.
- select_and_get_calibration(input, selector)[source]
Selects the best calibration based on input data and a selection rule, then retrieves it.
This method uses a CalibrationSelector to identify the most appropriate calibration for the given input data. Once selected, it retrieves the calibration file, downloading it if it’s not already cached locally.
- Parameters:
input – The input data product for which a calibration is needed.
selector (CalibrationSelector) – An instance of a
CalibrationSelectorclass.
- Returns:
tuple[str, dict] –
str: The local file path of the retrieved calibration file.dict: The record of the selected calibration from the local database.
Example – >>> # Assuming
my_input_dataandmy_selectorare defined >>> local_filepath, calibration_record = store.select_and_get_calibration(my_input_data, my_selector) >>> print(f”Calibration file: {local_filepath}”) >>> print(f”Calibration ID: {calibration_record[‘id’]}”)
- sync_records_from_cached_files(cals=None)[source]
Populates the local database from existing cached calibration files.
- Parameters:
cals (str | CalibrationModel | Sequence[str | CalibrationModel]) – A single calibration metadata dictionary or a data model instance, or a list of these.
Notes
This method may be removed in the future if not found useful.
- sync_records_from_remote(cals=None, mode='id')[source]
Synchronize the local DB with the remote KOA catalog.
calsis accepted for compatibility with callers that pass an explicit remote query result; when omitted, records are discovered viaget_missing_records(same askoa_middleware).Remote metadata failures (e.g. KOA API 404) are logged and treated as a no-op so pipelines can continue with the local caldb/cache. Callers that need a hard failure should inspect logs or sync explicitly with their own error handling.
- class MetaModel(schema, meta=None, validate=True)[source]
Bases:
MutableMappingSmall helper class for handling metadata based on JSON schema.
- Parameters:
- boolean_columns()[source]
Return a set of columns that are logically booleans (for PostgreSQL)
- group_meta_by_section(flat=False)[source]
Group metadata properties by their FITS section.
- Parameters:
flat (bool, optional) – Whether to return a flat dictionary instead of grouped by section. Defaults to False.
- Returns:
dict – Dictionary of metadata grouped by FITS section. Keys are section names and values are dictionaries of metadata properties. Sections are ordered as they first appear in the metadata schema YAML file (
self._schema['properties']preserves that order).
- pprint(show_none=False, header_style=True)[source]
Pretty-print the metadata, grouped by FITS section.
- Parameters:
show_none (bool, optional) – Whether to show metadata fields with None values. Defaults to False.
- set_default(prop_name, value)[source]
Set a default value for a specific metadata field.
- Parameters:
prop_name (str) – The property name for which to set the default value.
value (Any) – The default value to set if the property is not already set.
- set_default_values()[source]
Set default values for metadata fields based on the schema.
- to_fits_header(extname='PRIMARY', include_missing_metadata=False)[source]
Convert the metadata to a FITS header.
- Parameters:
- Returns:
header (fits.Header) – The FITS header containing the metadata.
- to_record()[source]
Convert the metadata to a dictionary record.
- Returns:
record (dict) – The metadata as a dictionary.
- validate(prop_name, value)[source]
Validate a metadata field against the schema.
- Parameters:
prop_name (str) – The property name to validate.
value (Any) – The value to validate.
- Raises:
ValueError – If the value does not conform to the schema.
- _build_empty_table_with_cols(extdef)[source]
Build a 0-row
astropy.table.Tablefrom an extension’s'columns'spec.
- _canonical_dtype(dtype_spec)[source]
The dtype to construct/cast to: the first entry if dtype_spec is a tuple (see
extensions.IMAGE_HDU()), else dtype_spec itself.
- class _classproperty(fget)[source]
Bases:
objectA read-only property accessible on the class itself, not just instances.
Deliberately a non-data descriptor (no
__set__): an instance’s own__dict__entry (e.g.self.configset in__init__) always takes priority over this on that instance. This is only consulted for class-level access (SomeModel.config) or from within a classmethod (cls.config), where there is no instance to hold a per-instance value.
- _correct_image_dtype(data, dtype_spec, extname='')[source]
Cast an image extension’s array to its declared dtype, if needed.
dtype_specis an extension’s declared'dtype'entry (seeextensions.IMAGE_HDU()): a single numpy scalar type, or a precedence-ordered tuple of them for extensions with more than one legitimate dtype (e.g.RampModel.DATA: uint16 for real detector data, float32 for simulations run withconvert_to_uint16=False).If
data’s dtype already matches one of the declared choices (subtype-wise), it’s returned unchanged. Otherwise the declared choice sharingdata’s broad numeric family (see_dtype_family()– e.g. any integer array matches a declared integer dtype, any float array matches a declared float dtype) is used as the cast target, falling back to the first (canonical) declared dtype if none share that family.
- _doc_table_render_axis(axis, detector_size=None)[source]
- _doc_table_render_dtype(dtype)[source]
- _doc_table_render_dtype_name(dtype)[source]
- _doc_table_render_shape(shape, detector_size=None)[source]
- _dtype_family(dtype)[source]
Broad numeric family of a dtype, for kind-aware auto-casting.
Numpy distinguishes signed (‘i’) and unsigned (‘u’) integers, but for auto-casting purposes any integer array should match a declared integer dtype, and any float array should match a declared float dtype.
- _set_datetime_meta(model)[source]
- _set_koa_meta(model)[source]
- _set_target_meta(model, gaia_id=None)[source]
- convert_endianness(arr, output_endianness='little')[source]
- datamodel_open(init, copy=False, config=None, **kwargs)
Creates a DataModel from a FITS file. Automatically select the right DataModel class based on the ‘DATAMODL’ header keyword.
- Parameters:
init – filepath (str): Full path to FITS file. DataModel: Return the given DataModel instance or make a copy if
copy=True.copy (bool) – If True, return a copy of the DataModel instance. If
initis a filepath, this is ignored.config (type, optional) – The instrument configuration class to use for the DataModel.
kwargs – Additional keyword arguments for the DataModel constructor.
- Returns:
DataModel – The specific DataModel instance.
- get_instrument_config(input=None, instrument_name=None)[source]
Get the instrument configuration class from either a FITS filename or an HDUList.
- Parameters:
input (str or fits.HDUList, optional) – Either a FITS filename path or an already-opened HDUList object. The INSTRUME keyword will be read from the primary header.
instrument_name (str, optional) – The instrument name. If provided, this will be used instead of reading the instrument name from the FITS header.
- Returns:
config_class (type) – The instrument config class if found.
- Raises:
TypeError – If input is neither a string nor an HDUList.
ValueError – If no INSTRUME keyword is found in the header, or if the instrument config is not found in INSTRUMENT_CONFIGS.
- get_meta_schema(instrument_name=None, config=None)[source]
Get the metadata schema for a given instrument.
- merge_meta(models)[source]
Merge the meta information from multiple DataModel instances.
- migrate_receipt_table(table)[source]
Upgrade a RECEIPT table to the current schema by adding any missing columns.