Calibration Database
Overview
The HISPEC DRP uses a local calibration database to manage calibration files and metadata. This database is implemented as a local SQLite database that interacts with a cache of calibration files stored on disk.
HISPECCalibrationStore
The HISPECCalibrationStore class provides the main interface for working with the calibration database.
Calibration Versioning
Each calibration file is associated with a version, which is determined by the metadata of the calibration file. Version numbers start at “001” and increment for each “new version of that same calibration” up to “999”.
Version Family
From a versioning context, calibration files are determined to be otherwise “identical” if they have the same values in a subset of metadata called the “version family”.
The method get_version_family_values() returns a list of metadata keys that are used to determine the version family. By default, the version family includes the following metadata keys:
Date time of observation:
meta.datetime_obsSpectrograph:
meta.spectrographCalibration Type:
meta.cal_typeMaster Calibration:
meta.master_cal
If two calibration files have the same values for all of the version family metadata keys andhave the same origin (See below), then they are considered to be in the same version family and must have a unique version number.
Calibration Origin
!! NOTE !! The behavior of *origin* is still being finalized subject to change. When creating new calibrations, populating the local/remote database, explicitly set the origin as needed.
The origin defines the source of where the calibration came from, and provides an independent local-only namespace for assigning versions to calibration files.
KECK: Calibrations generated at Keck which go through KOA.
LOCAL: Calibrations that are generated outside of the main DRP processing at Keck and registered to the local cache.
It is stored in the FITS metadata and used during version generation and file registration. One can set this value globally with the KOA_CALBRATION_ORIGIN environment variable.
Important methods that interact with the origin include:
- class RegisterCalibrationPrimitive(dataset=None, config=None, context=None, log_config=None, parent=None, **kwargs)[source]
Bases:
BasePrimitiveRegisters a calibration model in the local calibration database.
- Parameters:
input (CalibrationModel) – The calibration model to be registered in the local database.
origin (str, optional) – The origin to register the calibration under. Default is ‘local’.
- Returns:
output (CalibrationModel) – The same calibration model that was registered.
- HISPECCalibrationStore.register_calibration(cal, origin=None, new_version=False)[source]
Registers a calibration to the local cache and metadata database.
- Parameters:
cal (CalibrationModel | str) – The filepath or Calibration Datamodel object to register.
origin (str, optional) – The origin to register the calibration under.
new_version (bool, optional) – Whether to generate a new version for this calibration. If False, the method will check if a calibration with the same version family already exists in the cache and skip registration if so. Defaults to False.
- Returns:
tuple[str, dict] –
- A tuple containing:
str: The local file path where the calibration was saved.dict: The calibration metadata dictionary as added to the database.
Notes
Upstream
koa_middleware.CalibrationStore.register_calibrationsnapshotscal.to_record()beforecal.save()and discards the return value ofgenerate_calibration_version, sofilename/koa_id/cal_versioncan be stored asNoneeven though the FITS file is written. This override assigns those fields on the model and record before insert, and returns any existing version-family match instead of(None, None).
- HISPECCalibrationStore.sync_records_from_cached_files(cals=None)[source]
Populates the local database from existing cached calibration files.
- Parameters:
cals (str | CalibrationModel | Sequence[str | CalibrationModel]) – A single calibration metadata dictionary or a data model instance, or a list of these.
Notes
This method may be removed in the future if not found useful.
- HISPECCalibrationStore.generate_calibration_version(calibration, origin=None)[source]
Generate the next version string; accept a model or a metadata dict.
Upstream
koa_middlewareassignscalibration['origin']and therefore only works with dicts. Datamodel callers (andgenerate_calibration_version()) pass the model itself, so convert to a record first and write the version back onto the model when applicable.