Data Handling
Open Model
The method datamodels.open reads in a file and constructs the appropriate data model class for that data product as specified by the header key DATAMODL.
Data arrays are accessed with model.<attribute> where the attribute name is the extension name (case insensitive).
Metadata is accessed with model.meta.<attribute> where the attribute name is the property name in the metadata schema.
from hispecdrp import datamodels
from hispecdrp.utils import download_gdrive_file
# Download and open raw science data
sci_path = download_gdrive_file('HB.20240924.00000.00_raw.fits')
model = datamodels.open(sci_path)
print(model.meta.datetime_obs) # -> '2024-09-24T00:00:00.000'
print(model.meta.model_type) # -> 'RampModel'
print(model.meta.exposure_type) # -> 'SPEC'
print(model.meta.spectrograph) # -> 'BSPEC'
print(model.meta.instrument_mode) # -> 'YJHK'
print(model.data.shape) # -> (10, 4096, 4160) = (n_reads, ny, nx)
A file can also be passed to the constructor for the appropriate datamodel class:
from hispecdrp import datamodels
model = datamodels.L0Model(sci_path)
Save Model
Saving a datamodel is done with DataModel.save:
from hispecdrp import datamodels
model = datamodels.open(sci_path)
model.save(sci_path.replace('.fits', '_copy.fits'))
Instantiate DataModel from Scratch
Instead of reading in a datamodel from a file, one can instantiate a datamodel from scratch.
from hispecdrp import datamodels
model = datamodels.L0Model(
rate=np.ones((4096, 4160), dtype=np.float32),
meta={
'datetime_obs': '2024-09-24T00:00:00.000',
'exposure_type': 'DARK'
}
)
NOTE: When instantiating new datamodels without a file, one should at minimum pass the datetime_obs keyword to the metadata, or it will adopt the current time as the observation time.
Auto Initialization
Datamodels containing array extensions can be auto initialized to a given shape with a default fill value. The array is auto initialized upon the first attribute access.
from hispecdrp import datamodels
model = datamodels.L0Model(
rate=np.ones((4096, 4160), dtype=np.float32),
)
print(model.has_extension('rate_err')) # -> True
print(model.has_extension_value('rate_err')) # -> False
print(model.rate_err) # -> array of shape (4096, 4160) filled with zeros
Fiber Views
Many models have extensions enumerated by fiber (e.g, model.spec_fib1, model.spec_fib2, model.spec_fib3).
Instead of accessing the fiber extensions directly, one can use
from hispecdrp import datamodels
model = datamodels.L2Model()
model[1].spec # => model.spec_fib1
model[2].spec # => model.spec_fib2
# Alternatively:
model.spec[1] # => model.spec_fib1
# Other exts:
model[1].wave # => model.wave_fib1
model[1].spec_err # => model.spec_err_fib1
Processing History
Each datamodel has a receipt attribute which is a Table that contains a history of all primitives that have been applied to the datamodel.
CLASS — Primitive class
PROC_START_TIME — Processing start time YYYY-MM-DDTHH:MM:SS.sss
PROC_END_TIME — Processing end time YYYY-MM-DDTHH:MM:SS.sss
FILENAME — Base filename
PARAMETERS — JSON representation of parameters used for processing.
DRP_VERSION — DRP semver
DRP_GIT_COMMIT_ID — Full SHA1 commit ID
STATUS — COMPLETED, FAILED, SKIPPED
CONFIG_FILE — Name of the configuration file used for processing.
Calibration History
Each datamodel has a calibrations attribute which is a Table that contains a history of all calibrations that have been used within the context of this datamodel. Typically this means the calibration was ‘applied’ to the datamodel in some way, but it can also be used to track calibrations that were used in the creation of this datamodel, for example using an existing calibration as an initial guess to create a new calibration.
CAL_ROLE — Role of calibration
CAL_TYPE — Type of calibration
FILENAME — Base filename of the calibration
ID — UUID
DATETIME_OBS — Observation date/time of the calibration.
LAST_PROCESSED — When calibration was last processed.
TIME_APPLIED — Time calibration was applied YYYY-MM-DDTHH:MM:SS.sss
CLASS — Primitive class that used the calibration.
DRP_VERSION — DRP semver calibration was processed with
Data Ancestors
Each datamodel has a data_ancestors attribute which is a Table that contains the datamodels which were used to create this datamodel in contexts such as coadding.
The default columns are:
FILENAME - Base filename of the ancestor datamodel
CLASS - The class of the ancestor
TIME_APPLIED - The time this ancestor was used.
… plus other metadata fields as configured by
include_ancestorsin the metadata schema.
Additional columns are controlled through the metadata schema attributes include_ancestors=true/false.
Switching between HISPEC & PARVI
The HISPEC DRP supports both HISPEC and PARVI data products. By default, the DRP assumes all data products are from HISPEC. Changing to PARVI data products can be done by setting the environment variable HISPECDRP_INSTRUMENT_NAME to PARVI.
export HISPECDRP_INSTRUMENT_NAME=PARVI
or in Python:
import os
os.environ['HISPECDRP_INSTRUMENT_NAME'] = 'PARVI'
Each datamodel instance has an attribute config which is a Python class (not instance) that contains any instrument specific configuration and additional defaults not specified in the metadata schemas.