From e2f330c3cbe95d0625966fffd7938276540721f7 Mon Sep 17 00:00:00 2001 From: rachel3834 Date: Wed, 30 Sep 2026 16:37:19 -0700 Subject: [PATCH] Updated docs with new ReducedDatum types --- docs/conf.py | 4 ++-- docs/managing_data/index.rst | 35 +++++++++++++++++++++++++++++++++-- 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index eb1de9674..62e07c822 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -8,11 +8,11 @@ import sys import django +sys.path.insert(0, os.path.abspath("..")) + os.environ['DJANGO_SETTINGS_MODULE'] = 'tom_base.settings' django.setup() -sys.path.insert(0, os.path.abspath("..")) - extensions = [ "sphinx.ext.autodoc", "sphinx.ext.viewcode", diff --git a/docs/managing_data/index.rst b/docs/managing_data/index.rst index b28cb5a98..45be6fa2f 100644 --- a/docs/managing_data/index.rst +++ b/docs/managing_data/index.rst @@ -17,17 +17,36 @@ Managing Data The TOM's Data Models --------------------- -The TOM Toolkit includes two distinct models for data in the ``tom_dataproducts`` module: +The TOM Toolkit's' ``tom_dataproducts`` module recognizes a distinction between a *data product* and a *datum*: * ``DataProduct``: Corresponds to any file containing data, from a FITS, to a PNG, to a CSV. It can optionally be associated with a specific observation, and is required to be associated with a target. A ``DataProduct`` can have a specified type which can be used to trigger post-save hooks to perform automated process upon ingest. -* ``ReducedDatum``: Refers to a single piece of data - e.g., a spectrum, a single measurement or a set of timeseries +* ``*ReducedDatum`` (multiple types): Refers to a single piece of data - e.g., a spectrum, a single measurement or a set of timeseries photometry measurements. It is associated with a target, and optionally with the data product it came from. The TOM also allows a ``DataProductGroup`` to be defined. This allows TOM administrators control over which user groups can access which data products. +There are a number models to describe data types common in astronomy. + +* ``PhotometryReducedDatum``: Designed for measurements of brightness, this model has attributes ``brightness, brightness_error, limit, unit, bandpass`` and ``exposure_time``. It is designed to support both measured values and brightness limits for cases where direct measurement is not possible. +* ``SpectroscopyReducedDatum``: Designed for data with an associated wavelength, this model records the instrument ``setup`` and ``exposure_time`` in addition to the ``wavelength, flux, error``. The parameters ``flux_unit, wavelength_unit`` allow different spectral units to be stored. +* ``AstrometryReducedDatum``: Designed for objects with measured movement, this model records attributes ``ra, dec, ra_error, dec_error, ra_error_units, dec_error_units``. +* ``ReducedDatum``: Designed to be a general-purpose model to store data not represented by the other models. It's attribute is ``data_type``. + +All of the models inherit from the base class ``ReducedDatumCommon``, which has attributes common to all data, +including ``timestamp``. Foreign keys associate each datum with ``Target`` and ``DataProduct`` model entries. The +``value`` attribute is a JSON field which can store any dictionary of data and is designed to provide a +flexible means to store any further information the user requires. The ``telescope, instrument, +source_name, source_location`` and ``reduction_version`` attribues are character fields where users can +record the origin of the data. + +**Older versions**: These datum types were introduced in +`TOM Toolkit v3.0.0 `_; +older TOMs supported just the generic ReducedDatum. If you are upgrading an older TOM, please see +:doc:`these instructions <../introduction/updating>`. + Ingesting data into the TOM --------------------------- Data products for a given target can be uploaded through the ``Manage Data`` tab on the target's detail page, or @@ -47,6 +66,18 @@ reads a photometry data file and ingests the timeseries measurements as ``Reduce It's also possible for users to add their own custom data formats and corresponding specialized processors - see :doc:`Adding Custom Data Processing ` for more details. +Data Validation +--------------- +Before any ``*ReducedDatum`` is stored in the TOM it is validated to avoid duplicating data entries. +A ``ValidationError`` will be raised if the new datum has the same ``data_type``, ``timestamp`` and ``value``, +as an existing datum and is associated with the same ``target``. + +When performing a ``bulk_create`` of multiple ``*ReducedDatums ``, a ``ValidationError`` can cause the +whole batch to be aborted. If you just wish to skip duplicate rows and ingest only new entries, you can +use ``ignore_conflicts`` with most database types: + +``PhotometryReducedDatum.objects.bulk_create(reduced_datums_list, ignore_conflicts=True)`` + Data Visualization ------------------