diff --git a/docs/data_services/alerce.rst b/docs/data_services/alerce.rst new file mode 100644 index 000000000..a81d3e6f2 --- /dev/null +++ b/docs/data_services/alerce.rst @@ -0,0 +1,83 @@ +ALeRCE: ZTF and LSST +-------------------- + +The ALeRCE data service queries the `ALeRCE `_ broker for objects from +both ZTF and the Rubin Observatory's LSST. Choose the survey with the **Survey** field on the +query form. + +How ALeRCE is queried +********************* + +* **ZTF** searches use the ALeRCE REST API through the ``alerce`` Python client, as before. +* **LSST** searches, and the classifier lists for both surveys, use ALeRCE's + `TAP service `_ through ``pyvo``. The REST API can list LSST + objects, but it cannot yet filter them by classifier or list the LSST classifiers, and asteroid + (ssObject) searches need the MPC orbits that are only available through TAP. +* **Light curves** for both surveys come from the REST API. + +If the TAP service is unavailable, the query form still loads without the classifier fields, and +LSST searches report an error instead of failing the page. + +Searching for LSST objects +************************** + +With **Survey** set to LSST, choose an **LSST Object Type**: + +``diaObject`` + A static-sky transient or variable, for example a supernova candidate. + +``ssObject`` + A known Solar System object, linked by the MPC to an orbit. + +The **Object ID** field takes a diaObjectId, an ssObjectId, or an asteroid designation such as +``2010 WX64``. A designation is looked up in ALeRCE's copy of the MPC orbits, which takes about +a second. Without an Object ID, the cone search, first and last detection MJD, and number of +detections fields filter the search, as they do for ZTF. There are also three LSST-specific +options: + +* **Classifiers** filter diaObjects by class and minimum probability, for example ``SN`` from + ``stamp_classifier_rubin_beta``. ssObjects are not offered here, because ALeRCE assigns every + known ssObject the asteroid class with probability 1 rather than classifying it. +* **Max. Detection Time Span (days)** limits the time between the first and last detection. + Together with the stamp classifier's asteroid class, a span of less than a day finds candidate + new moving objects that are not yet linked to an ssObject. +* **Max. Results**, **Sort By** and **Sort Order** control the size and order of the results. + When you filter by classifier, the limit applies to each classifier separately, and results + are sorted by probability by default. + +Each result links to the object's page on the ALeRCE Explorer. + +Creating targets +**************** + +* A **diaObject** becomes a sidereal target named by its diaObjectId, at ALeRCE's mean position. +* An **ssObject** becomes a non-sidereal target named by its ssObjectId, with orbital elements + taken from its MPC orbit. Bound orbits use the ``MPC_MINOR_PLANET`` scheme; unbound ones use + ``MPC_COMET``. The target's aliases include its provisional designation and, where the MPC has + assigned them, its permanent number, name and any secondary designations. That way you can find + it under whichever name you know. + +Photometry +********** + +LSST light curves report difference-image PSF fluxes in nJy. These are converted to AB +magnitudes, with a zero point of 31.4, and their times are converted from TAI to UTC. Detections +with zero or negative difference flux have no magnitude and are skipped. LSST photometry is +recorded with the telescope ``Rubin`` and the instrument ``LSSTCam``. + +Updating the data for a target again leaves the photometry already stored unchanged, even if ALeRCE +has since revised a measurement's uncertainty. + +Updating targets from other sources +*********************************** + +A target doesn't have to come from ALeRCE for ALeRCE to update its data. The data service finds +the ALeRCE object from the target's name and aliases, in this order: + +#. A name that is already a ZTF object ID or an LSST diaObjectId/ssObjectId. +#. A TNS name (e.g. ``SN 2025abc``), resolved to its ZTF or LSST internal name through TNS. + This needs the TNS data service to be configured; otherwise TNS names are skipped. +#. An MPC designation (e.g. ``2010 WX64``), resolved to its LSST ssObjectId. + +This means, for example, that an asteroid target created from the MPC can pick up its LSST +photometry. diff --git a/docs/data_services/index.rst b/docs/data_services/index.rst index 4296c63c0..1db3a8f86 100644 --- a/docs/data_services/index.rst +++ b/docs/data_services/index.rst @@ -7,6 +7,7 @@ Data Services :hidden: Add a data service + ALeRCE: ZTF and LSST Available data services <../api/tom_dataservices/data_services> Data service views <../api/tom_dataservices/views> @@ -20,4 +21,6 @@ A TOM Toolkit Data Service Module contains the logic for querying a remote broke :doc:`Data Service Modules <../api/tom_dataservices/data_services>` - Take a look at the supported Data Services. +:doc:`ALeRCE: ZTF and LSST ` - Search ALeRCE for ZTF and LSST objects, including LSST asteroids, and ingest their light curves. + :doc:`Data Service Views <../api/tom_dataservices/views>` - Familiarize yourself with the available Data Service Views. diff --git a/poetry.lock b/poetry.lock index 646944719..8b046950b 100644 --- a/poetry.lock +++ b/poetry.lock @@ -4777,4 +4777,4 @@ type = ["pytest-mypy (>=1.0.1) ; platform_python_implementation != \"PyPy\""] [metadata] lock-version = "2.1" python-versions = ">=3.10.0,<3.14" -content-hash = "44b4d11b950c4bf859646ba630cb36b6668f6b0cfecee27c0b57b129e068da67" +content-hash = "7f549c70cd377c9957ec234778c6fda1b53d1fd2a5612c15ed138598d50af196" diff --git a/pyproject.toml b/pyproject.toml index 0d2424107..679ce5d64 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -72,6 +72,7 @@ dependencies = [ "importlib-resources >=6.4.5,<6.5.0", "django-tasks (>=0.6.1,<0.7.0)", "alerce >=2.3.1,<3.0", + "pyvo >=1.5,<2", ] [tool.poetry] diff --git a/tom_dataservices/data_services/alerce.py b/tom_dataservices/data_services/alerce.py index 34b6f15d3..3f270d912 100644 --- a/tom_dataservices/data_services/alerce.py +++ b/tom_dataservices/data_services/alerce.py @@ -1,31 +1,426 @@ +import json import logging +import math +import re from alerce.core import Alerce from alerce.exceptions import ObjectNotFoundError, APIError +from astropy.constants import GM_sun, au from astropy.time import Time, TimezoneInfo from django import forms from django.core.cache import cache +from django.core.exceptions import ValidationError from django.db.utils import IntegrityError +import numpy as np +import pyvo +import requests + from tom_dataproducts.models import PhotometryReducedDatum -from tom_dataservices.dataservices import DataService, QueryServiceError +from tom_dataservices.data_services.tns import TNSDataService +from tom_dataservices.dataservices import DataService, NotConfiguredError, QueryServiceError from tom_dataservices.forms import BaseQueryForm -from tom_targets.models import Target, TargetExtra +from tom_targets.models import Target logger = logging.getLogger(__name__) alerce = Alerce() - +TAP_URL = 'https://tap.alerce.online/tap' +tap_service = pyvo.dal.TAPService(TAP_URL) ALERCE_FILTERS = {1: "g", 2: "r", 3: "i"} +def _to_native_types(record: dict) -> dict: + """ + TAP query results (astropy Table rows) come back with numpy scalar types + (np.float64, np.int64, etc.) which aren't accepted by Django model fields + or JSON-serializable. Convert them to their native Python equivalents. + """ + return {k: (v.item() if isinstance(v, np.generic) else v) for k, v in record.items()} + + +def _normalize_tap_record(record: dict) -> dict: + """ + Converts numpy scalars to native types and renames the TAP `object` table's + `n_det` column to `ndet`, matching the ALeRCE REST API naming used downstream + (results table, target extras). + """ + record = _to_native_types(record) + if "n_det" in record: + record["ndet"] = record.pop("n_det") + return record + + +def _normalize_ztf_record(record: dict) -> dict: + """ + Renames the ZTF REST client's `deltajd` field to `deltamjd`, matching the TAP + `object` table naming (and the `firstmjd`/`lastmjd` naming already used elsewhere) + so downstream code sees one consistent field name regardless of survey. + """ + if "deltajd" in record: + record["deltamjd"] = record.pop("deltajd") + return record + + +def _append_tap_filters(query: str, query_parameters: dict, column_prefix: str = "") -> str: + """ + Appends the cone-search / mjd-range / ndet-range / max-deltamjd WHERE clauses shared by all + `alerce_tap.object`-based ADQL queries. `column_prefix` (e.g. `"obj."`) is + needed when `alerce_tap.object` is joined/aliased, as in + `_build_tap_classifier_query`. Only numeric parameters (already cleaned by + the form) are interpolated, so there is no string-injection surface. + """ + p = column_prefix + if all(query_parameters.get(k) is not None for k in ("ra", "dec", "radius")): + ra = query_parameters["ra"] + dec = query_parameters["dec"] + radius_deg = query_parameters["radius"] / 3600.0 + query += ( + f" AND 1 = CONTAINS(POINT('ICRS', {p}meanra, {p}meandec), " + f"CIRCLE('ICRS', {ra}, {dec}, {radius_deg}))" + ) + + if firstmjd := query_parameters.get("firstmjd"): + query += f" AND {p}firstmjd >= {firstmjd[0]} AND {p}firstmjd <= {firstmjd[1]}" + + if lastmjd := query_parameters.get("lastmjd"): + query += f" AND {p}lastmjd >= {lastmjd[0]}" + if len(lastmjd) == 2: + query += f" AND {p}lastmjd <= {lastmjd[1]}" + + if ndet := query_parameters.get("ndet"): + query += f" AND {p}n_det >= {ndet[0]}" + if len(ndet) == 2: + query += f" AND {p}n_det <= {ndet[1]}" + + if (deltamjd_max := query_parameters.get("deltamjd_max")) is not None: + query += f" AND {p}deltamjd <= {float(deltamjd_max)}" + + return query + + +DEFAULT_PAGE_SIZE = 20 +MAX_PAGE_SIZE = 1000 +# Sort choices offered by AlerceForm, mapped to `alerce_tap.object` columns. The keys are +# also valid ZTF REST `order_by` values. +TAP_ORDER_COLUMNS = {"lastmjd": "lastmjd", "firstmjd": "firstmjd", "ndet": "n_det"} + + +def _tap_order_clause(query_parameters: dict, column_prefix: str = "") -> str: + """ + Returns an ` ORDER BY` clause for `query_parameters["order_by"]`/`["order_mode"]`, or + "" if no (known) order was requested. Only whitelisted columns and ASC/DESC are + interpolated, since query_parameters can come from saved queries and not only from + the validated form. + """ + column = TAP_ORDER_COLUMNS.get(query_parameters.get("order_by")) + if not column: + return "" + mode = "ASC" if query_parameters.get("order_mode") == "ASC" else "DESC" + return f" ORDER BY {column_prefix}{column} {mode}" + + +def _build_tap_object_query(query_parameters: dict, page_size: int | None = None) -> str: + """ + Builds an ADQL query against `alerce_tap.object` for LSST (sid != 0) general + (non-oid) queries, consuming the same query_parameters shape produced by + `AlerceDataService.build_query_parameters`. `oid` lookups are handled + separately and are not built here. The row limit comes from `page_size`, then + `query_parameters["page_size"]`, then `DEFAULT_PAGE_SIZE`. + """ + sid = query_parameters.get("sid", 0) + page_size = int(page_size or query_parameters.get("page_size") or DEFAULT_PAGE_SIZE) + query = f"SELECT TOP {page_size} * FROM alerce_tap.object WHERE sid = {sid}" + query = _append_tap_filters(query, query_parameters) + return query + _tap_order_clause(query_parameters) + + +def _build_tap_classifier_query( + query_parameters: dict, classifier_id: int, class_id: int, probability: float | None = None, + page_size: int | None = None, +) -> str: + """ + Builds an ADQL query joining `alerce_tap.object` to `alerce_tap.probability` + for LSST diaObject (sid=1) classifier queries -- the TAP equivalent of the + ZTF REST classifier loop in `query_service`. `ranking = 1` restricts results + to objects whose *top-ranked* classification is the requested class (matching + ALeRCE's own usage in notebooks/LSST/ALeRCE_LSST_SSO.ipynb); `probability`, + when given, is treated as a minimum threshold. Only sid=1 is meaningful here: + known ssObjects (sid=2) are pre-assigned probability 1 "asteroid" rather than + classified, so classifier queries are not offered for sid=2 (see + `AlerceDataService.query_service`). Results are ordered by descending + probability unless `query_parameters` asks for another order. + """ + sid = query_parameters.get("sid", 1) + page_size = int(page_size or query_parameters.get("page_size") or DEFAULT_PAGE_SIZE) + query = ( + f"SELECT TOP {page_size} obj.*, prob.probability, prob.ranking " + f"FROM alerce_tap.object AS obj " + f"JOIN alerce_tap.probability AS prob " + f"ON obj.oid = prob.oid AND obj.sid = prob.sid " + f"WHERE obj.sid = {sid} AND prob.classifier_id = {classifier_id} " + f"AND prob.class_id = {class_id} AND prob.ranking = 1" + ) + if probability is not None: + query += f" AND prob.probability >= {probability}" + query = _append_tap_filters(query, query_parameters, column_prefix="obj.") + query += _tap_order_clause(query_parameters, column_prefix="obj.") or " ORDER BY prob.probability DESC" + return query + + +SURVEY_TID = {"ZTF": 0, "LSST": 1} +ALERCE_EXPLORER_URLS = { + "ztf": "https://alerce.online/object/{oid}", + "lsst": "https://lsst.alerce.online/object/{oid}?survey=lsst", +} + + +def _group_tap_classifier_rows(rows) -> list[dict]: + """ + Groups `alerce_tap.classifier` JOIN `alerce_tap.taxonomy` rows into the shape + the (deprecated) REST `query_classifiers()` used to return, plus the numeric + `classifier_id`/`class_id` TAP `probability` queries need (absent from rows + that don't carry them, e.g. older fixtures): + [{classifier_name, classifier_version, classifier_id, classes: [...], class_ids: {name: id}}, ...] + """ + grouped = {} + for row in rows: + row = dict(row) + key = (row["classifier_name"], row["classifier_version"]) + grouped.setdefault( + key, + { + "classifier_id": row.get("classifier_id"), + "classifier_name": row["classifier_name"], + "classifier_version": row["classifier_version"], + "classes": [], + "class_ids": {}, + }, + ) + grouped[key]["classes"].append(row["class_name"]) + if "class_id" in row: + grouped[key]["class_ids"][row["class_name"]] = row["class_id"] + return list(grouped.values()) + + +def _fetch_classifiers_for_tid(tid: int) -> list[dict]: + """ + Queries + caches (24h) `alerce_tap.classifier` JOIN `alerce_tap.taxonomy` for a + given `tid`, grouped via `_group_tap_classifier_rows`. Shared by + `AlerceForm.get_classifiers` (form field generation) and + `_resolve_classifier_ids` (TAP classifier-query construction) so both see the + same cached data under the same cache key. + """ + cache_key = f"ds_alerce_classifiers_{tid}" + classifiers = cache.get(cache_key) + if not classifiers: + query = ''' + SELECT c.classifier_id, c.classifier_name, c.classifier_version, + t.class_id, t.class_name + FROM alerce_tap.classifier c + JOIN alerce_tap.taxonomy t ON t.classifier_id = c.classifier_id + WHERE c.tid = %d ORDER BY c.classifier_name, t.taxonomy_order + ''' % tid + classifiers = _group_tap_classifier_rows(tap_service.search(query)) + cache.set(cache_key, classifiers, 3600 * 24) # One day + return classifiers + + +def _resolve_classifier_ids(tid: int, classifier_name: str, class_name: str) -> tuple[int, int] | None: + """ + Looks up the numeric (classifier_id, class_id) pair for a classifier/class + name pair, as needed to query `alerce_tap.probability`. Returns None if no + match is found (e.g. a stale classifier name from a different survey/tid). + """ + for classifier in _fetch_classifiers_for_tid(tid): + if classifier["classifier_name"] == classifier_name and class_name in classifier["class_ids"]: + return classifier["classifier_id"], classifier["class_ids"][class_name] + return None + + +# From ALeRCE's `alerce_tap.band` lookup table for tid=1 (identical for sid 1 and 2). +LSST_BANDS = {1: "g", 2: "r", 3: "i", 4: "z", 5: "y", 6: "u"} +LSST_AB_ZEROPOINT_NJY = 31.4 + + +def _lsst_detection_photometry(detection: dict) -> dict | None: + """ + Converts an LSST detection's difference-image PSF flux (`psfFlux`/`psfFluxErr`, + in nJy) to PhotometryReducedDatum field values in AB magnitudes. Returns None + when the flux is non-positive (source fainter than the template), which has no + magnitude. + """ + flux = detection.get("psfFlux") + if flux is None or flux <= 0: + return None + flux_err = detection.get("psfFluxErr") + return { + "brightness": LSST_AB_ZEROPOINT_NJY - 2.5 * math.log10(flux), + "brightness_error": 2.5 / math.log(10) * flux_err / flux if flux_err is not None else None, + "bandpass": detection.get("band_name") or LSST_BANDS[detection["band"]], + "telescope": "Rubin", + "instrument": "LSSTCam", + } + + +GAUSSIAN_K_DEG_PER_DAY = math.degrees(math.sqrt(GM_sun.value) * au.value ** -1.5 * 86400.0) + + +def _fetch_lsst_mpc_orbit(ss_object_id) -> dict | None: + """ + Returns the latest `alerce_tap.lsst_mpc_orbits` record for an LSST ssObjectId, or None + if ALeRCE has none stored. + """ + query = f"SELECT * FROM alerce_tap.lsst_mpc_orbits WHERE ssObjectId = {int(ss_object_id)}" + rows = tap_service.search(query) + return _to_native_types(dict(rows[0])) if len(rows) else None + + +LSST_DESIGNATION_PATTERN = re.compile(r"^[A-Za-z0-9 /()\-.]+$") + + +def _resolve_lsst_designation(designation: str) -> int | None: + """ + Returns the LSST ssObjectId for an MPC designation as stored by ALeRCE (e.g. + "2010 WX64"), or None if there is none. `designation` is not indexed in + `alerce_tap.lsst_mpc_orbits`, so this is slower than an ID lookup. Only designation + characters are accepted, so the value can be quoted into ADQL safely. + """ + designation = " ".join(designation.split()) + if not LSST_DESIGNATION_PATTERN.match(designation): + raise QueryServiceError(f"{designation!r} is not an LSST object ID or asteroid designation") + rows = tap_service.search( + f"SELECT ssObjectId FROM alerce_tap.lsst_mpc_orbits WHERE designation = '{designation}'" + ) + return int(rows[0]["ssobjectid"]) if len(rows) else None + + +ZTF_OID_PATTERN = re.compile(r"^ZTF\d{2}[a-z]{7}$") +# LSST diaObjectIds/ssObjectIds are 17-18 digits; shorter numbers are e.g. numbered asteroids like "6478" +LSST_OID_PATTERN = re.compile(r"^\d{15,}$") +TNS_NAME_PATTERN = re.compile(r"^(?:AT|SN)\s?(\d{4}[a-z]{1,3})$") +# Provisional ("1988 JC1"), survey ("2040 P-L") and comet ("C/2025 A6") designations; ALeRCE +# knows asteroids only by these, not by number or name +MPC_DESIGNATION_PATTERN = re.compile(r"^(?:[CPDXAI]/)?\d{4} (?:[A-Z]{1,2}\d*(?:-[A-Z])?|[PT]-[L123])$") + + +def _alerce_id_from_name(name: str) -> tuple[str, str] | None: + """Returns (survey, oid) if `name` is itself a ZTF or LSST ALeRCE object ID.""" + if ZTF_OID_PATTERN.match(name): + return "ZTF", name + if LSST_OID_PATTERN.match(name): + return "LSST", name + return None + + +def _alerce_id_from_remote_name(name: str) -> tuple[str, str] | None: + """ + Returns (survey, oid) for a TNS name (via the ZTF/LSST IDs in its TNS internal names) + or an MPC designation (via `alerce_tap.lsst_mpc_orbits`), or None. Lookup failures, + including TNS not being configured, are logged and give None, so a data update falls + back to other names rather than failing. + """ + if tns_match := TNS_NAME_PATTERN.match(name): + for internal_name in _tns_internal_names(tns_match.group(1)): + if found := _alerce_id_from_name(internal_name): + return found + return None + if MPC_DESIGNATION_PATTERN.match(name): + try: + ss_object_id = _resolve_lsst_designation(name) + except (pyvo.dal.DALAccessError, QueryServiceError): + logger.exception(f"Error resolving MPC designation {name} with ALeRCE") + return None + return ("LSST", str(ss_object_id)) if ss_object_id is not None else None + return None + + +def _tns_internal_names(objname: str) -> list[str]: + """ + Returns the internal (survey) names TNS lists for an object, given its name without + the AT/SN prefix, or [] if TNS isn't configured, has no such object, or fails. + """ + tns = TNSDataService() + try: + data = tns.query_service(tns.build_query_parameters({"objname": objname}), url=tns.get_urls("object_url")) + except NotConfiguredError: + logger.info(f"TNS is not configured; cannot resolve {objname} to ALeRCE IDs") + return [] + except (requests.RequestException, ValueError, KeyError): + logger.exception(f"Error querying TNS for {objname}") + return [] + if not isinstance(data, dict) or not data.get("objname"): + return [] + return [name.strip() for name in (data.get("internal_names") or "").split(",") if name.strip()] + + +def _mpc_designation_data(orbit: dict) -> dict: + """ + Returns the MPC `designation_data` (permid, name, secondary designations, ...) from an + `alerce_tap.lsst_mpc_orbits` record's `mpc_orb_jsonb`, which TAP may return as a JSON + string, or {} if it is missing or unparseable. + """ + mpc_orb = orbit.get("mpc_orb_jsonb") + if isinstance(mpc_orb, str): + try: + mpc_orb = json.loads(mpc_orb) + except ValueError: + return {} + if not isinstance(mpc_orb, dict): + return {} + return mpc_orb.get("designation_data") or {} + + +def _non_sidereal_target_from_mpc_orbit(name, orbit: dict) -> Target: + """ + Builds a NON_SIDEREAL Target from an `alerce_tap.lsst_mpc_orbits` record. The table's + `a`/`mean_anomaly`/`mean_motion` can be 0.0 placeholders rather than null, so, as in + `MPCExplorerDataService.create_target_from_query`, they are derived from the + always-present perihelion elements (`q`, `e`, `peri_time`). Bound orbits (e < 1) use + the MPC_MINOR_PLANET scheme; unbound ones use MPC_COMET. + """ + target = Target( + name=name, + type=Target.NON_SIDEREAL, + scheme="MPC_COMET", + epoch_of_elements=orbit["epoch_mjd"], + inclination=orbit["i"], + lng_asc_node=orbit["node"], + arg_of_perihelion=orbit["argperi"], + eccentricity=orbit["e"], + perihdist=orbit["q"], + epoch_of_perihelion=orbit["peri_time"], + abs_mag=orbit.get("h"), + slope=orbit.get("g"), + ) + if target.eccentricity < 1.0: + target.scheme = "MPC_MINOR_PLANET" + target.semimajor_axis = target.perihdist / (1.0 - target.eccentricity) + target.mean_daily_motion = GAUSSIAN_K_DEG_PER_DAY / target.semimajor_axis ** 1.5 + target.mean_anomaly = ( + (target.epoch_of_elements - target.epoch_of_perihelion) * target.mean_daily_motion + ) % 360.0 + return target + + class AlerceForm(BaseQueryForm): CLASSIFIER_FIELD_PREFIX = "cfield_" survey = forms.ChoiceField( - label="Survey", choices=[("ZTF", "ZTF"), ("LSST", "LSST")], initial="ZTF" + label="Survey", choices=[("ZTF", "ZTF"), ("LSST", "LSST")], initial="ZTF", + widget=forms.Select(attrs={"x-model": "$store.alerce.survey"}), + ) + lsst_object_type = forms.ChoiceField( + required=False, + label="LSST Object Type", + choices=[("diaObject", "diaObject"), ("ssObject", "ssObject")], + initial="diaObject", + help_text="Only used when Survey is LSST.", + ) + object_id = forms.CharField( + required=False, label="Object ID", + help_text="For LSST, a diaObjectId, an ssObjectId, or an asteroid designation such as 2010 WX64.", ) - object_id = forms.CharField(required=False, label="Object ID") ra = forms.FloatField(required=False, label="RA (deg)") dec = forms.FloatField(required=False, label="Dec (deg)") radius = forms.FloatField(required=False, label="Search Radius (arcsec)") @@ -35,70 +430,108 @@ class AlerceForm(BaseQueryForm): lastmjd_lt = forms.FloatField(required=False, label="Max MJD of last detection") ndet_min = forms.IntegerField(required=False, label="Min. Number of Detections") ndet_max = forms.IntegerField(required=False, label="Max Number of Detections") + deltamjd_max = forms.FloatField( + required=False, label="Max. Detection Time Span (days)", min_value=0, + help_text="LSST only. Time between first and last detection. Less than 1 day, combined with the stamp " + "classifier's asteroid class, finds candidate new moving objects.", + ) + max_results = forms.IntegerField( + required=False, label="Max. Results", initial=DEFAULT_PAGE_SIZE, min_value=1, max_value=MAX_PAGE_SIZE, + help_text="Per classifier, when filtering by classifier.", + ) + order_by = forms.ChoiceField( + required=False, label="Sort By", + choices=[("", "Default"), ("lastmjd", "Last detection"), ("firstmjd", "First detection"), + ("ndet", "Number of detections")], + help_text="Default is by probability for LSST classifier searches.", + ) + order_mode = forms.ChoiceField( + required=False, label="Sort Order", choices=[("DESC", "Descending"), ("ASC", "Ascending")], initial="DESC", + ) def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # Dynamically add the classifier fields to the form self.add_classifiers_fields() # Make the survey field hidden for now until the LSST API is more featured - self.fields['survey'].widget = forms.HiddenInput() - - def get_classifiers(self) -> list[dict]: - classifiers = cache.get("ds_alerce_classifiers") - if not classifiers: - classifiers = alerce.query_classifiers() - cache.set("ds_alerce_classifiers", classifiers, 3600 * 24) # One day + # self.fields['survey'].widget = forms.HiddenInput() - return classifiers + def get_classifiers(self, survey: str) -> list[dict]: + """ + Returns the survey's classifiers, or none if the TAP query fails, so the form still + renders (without classifier fields) during an ALeRCE TAP outage. The failure is not + cached, so the next form load retries. + """ + tid = SURVEY_TID.get(survey, SURVEY_TID["ZTF"]) + try: + return _fetch_classifiers_for_tid(tid) + except pyvo.dal.DALAccessError: + logger.exception(f"Error querying ALeRCE {survey} classifiers") + return [] def add_classifiers_fields(self) -> list[tuple[str, str]]: """ - Adds the fields dynamically to the form. + Adds classifier fields for *every* survey (not just whichever one happens to be + selected/initial), so the advanced form can show/hide the right classifier set + purely client-side (via Alpine, keyed off the `survey` field) when the user + switches surveys, without a full form re-render. Field names embed the owning + survey (`cfield_{survey}__{classifier_name}`) so `clean()` can tell which group a + submitted value belongs to and ignore stale values left in a hidden, + non-selected survey's fields. Returns a list of classifier, probability fields name pairs to be used by the crispy layout. """ - classifiers = self.get_classifiers() field_names = [] - for c in classifiers: - field_name = f"{self.CLASSIFIER_FIELD_PREFIX}{c['classifier_name']}" - # Add the field to the Django form - self.fields[field_name] = forms.ChoiceField( - label=f"{c['classifier_name']}", - choices=[(None, "")] + [(k, k) for k in c["classes"]], - required=False, - help_text=f'Classifier Version: {c["classifier_version"]}', - ) - prob_field_name = ( - f"prob_{self.CLASSIFIER_FIELD_PREFIX}{c['classifier_name']}" - ) - self.fields[prob_field_name] = forms.FloatField( - label=f"{c['classifier_name']} Probability", - required=False, - max_value=1, - min_value=0, - help_text="Value between 0 and 1" - ) - field_names.append((field_name, prob_field_name)) + for survey in SURVEY_TID: + for c in self.get_classifiers(survey): + field_name = f"{self.CLASSIFIER_FIELD_PREFIX}{survey}__{c['classifier_name']}" + # Add the field to the Django form + self.fields[field_name] = forms.ChoiceField( + label=f"{c['classifier_name']}", + choices=[(None, "")] + [(k, k) for k in c["classes"]], + required=False, + help_text=f'Classifier Version: {c["classifier_version"]}', + ) + prob_field_name = f"prob_{field_name}" + self.fields[prob_field_name] = forms.FloatField( + label=f"{c['classifier_name']} Probability", + required=False, + max_value=1, + min_value=0, + help_text="Value between 0 and 1" + ) + field_names.append((field_name, prob_field_name)) # Returns field names, not the actual field objects return field_names def clean(self): cleaned_data = super().clean() or {} + selected_survey = cleaned_data.get("survey") classifiers: list[dict] = [] - # Find the classifiers, if any + # Find the classifiers, if any, belonging to the currently selected survey. + # (Non-selected surveys' classifier fields are present in the form -- so they + # can be toggled client-side -- but merely hidden, not disabled, so a stale + # value left over from switching surveys must be ignored here.) for k, v in cleaned_data.items(): if k.startswith(self.CLASSIFIER_FIELD_PREFIX) and v: + survey, classifier_name = k[len(self.CLASSIFIER_FIELD_PREFIX):].split("__", 1) + if survey != selected_survey: + continue classifiers.append( { - "classifier": k.split(self.CLASSIFIER_FIELD_PREFIX)[1], + "classifier": classifier_name, "class": v, "probability": cleaned_data.get(f"prob_{k}", None), } ) cleaned_data["classifiers"] = classifiers + # The ZTF REST API has no deltajd filter + if cleaned_data.get("deltamjd_max") is not None and selected_survey != "LSST": + self.add_error("deltamjd_max", "Only supported for LSST searches.") + return cleaned_data def get_simple_form_partial(self): @@ -127,34 +560,84 @@ def query_service(self, query_parameters, **kwargs) -> list[dict]: if provided. """ results = [] + # Get the sid from the query parameters, defaulting to 0 (ZTF) if not provided + sid = query_parameters.get("sid", 0) try: if query_parameters.get("oid"): - # We might want to specify the survey based on the object id prefix - # once the LSST support in the alerce client is improved - object_result = alerce.query_object(**query_parameters) - if object_result: - results.append(object_result) - - return results - - classifier_params = query_parameters.pop("classifiers") - if len(classifier_params) == 0: - general_results = alerce.query_objects(**query_parameters).get("items", []) - results.extend(general_results) + if sid == 0: + query_parameters.pop("sid", None) + items = alerce.query_objects(**query_parameters).get("items", []) + results = [_normalize_ztf_record(item) for item in items] + else: + oid = str(query_parameters["oid"]).strip() + if not oid.isdigit(): + oid, sid = _resolve_lsst_designation(oid), 2 + if oid is not None: + query = ''' + SELECT * FROM alerce_tap.object + WHERE oid = %s AND sid = %d + ''' % (int(oid), sid) + results = [_normalize_tap_record(dict(row)) for row in tap_service.search(query)] + + elif sid != 0: + # LSST (diaObject/ssObject) general queries go through TAP. Classifier + # queries are only meaningful for diaObjects (sid=1): known ssObjects + # (sid=2) are pre-assigned probability 1 "asteroid" rather than + # classified (see alerce_tap.probability / ALeRCE's own LSST SSO + # notebook), so they're not offered here. + classifier_params = query_parameters.get("classifiers") + if classifier_params: + if sid != 1: + raise QueryServiceError("LSST classifier queries are only supported for diaObjects") + tid = SURVEY_TID["LSST"] + for classifier in classifier_params: + ids = _resolve_classifier_ids(tid, classifier["classifier"], classifier["class"]) + if ids is None: + raise QueryServiceError( + f"Unknown ALeRCE LSST classifier/class: " + f"{classifier['classifier']}/{classifier['class']}" + ) + classifier_id, class_id = ids + tap_query = _build_tap_classifier_query( + query_parameters, classifier_id, class_id, classifier.get("probability") + ) + # The query filters by class_id; add the names, as ZTF REST results carry them + results.extend( + {**_normalize_tap_record(dict(row)), + "class": classifier["class"], "classifier": classifier["classifier"]} + for row in tap_service.search(tap_query) + ) + else: + tap_query = _build_tap_object_query(query_parameters) + results = [_normalize_tap_record(dict(row)) for row in tap_service.search(tap_query)] + else: - for classifier in classifier_params: - classifier_results = alerce.query_objects( - classifier=classifier["classifier"], - class_name=classifier["class"], - probability=classifier["probability"], - **query_parameters, - ).get("items", []) - results.extend(classifier_results) + # "sid" is only used for the TAP-based object ID lookup above; the ALeRCE + # REST client used below doesn't accept it. + query_parameters.pop("sid", None) + classifier_params = query_parameters.pop("classifiers") + if len(classifier_params) == 0: + general_results = alerce.query_objects(**query_parameters).get("items", []) + results.extend(general_results) + else: + for classifier in classifier_params: + classifier_results = alerce.query_objects( + classifier=classifier["classifier"], + class_name=classifier["class"], + probability=classifier["probability"], + **query_parameters, + ).get("items", []) + results.extend(classifier_results) + results = [_normalize_ztf_record(result) for result in results] except (ObjectNotFoundError, ValueError, APIError) as e: raise QueryServiceError(str(e)) + except pyvo.dal.DALAccessError as e: + logger.exception("Error querying the ALeRCE TAP service") + raise QueryServiceError(f"ALeRCE TAP query failed: {e}") for result in results: result["survey"] = query_parameters["survey"] + result["alerce_url"] = ALERCE_EXPLORER_URLS[result["survey"]].format(oid=result["oid"]) return results @@ -165,18 +648,26 @@ def build_query_parameters(self, parameters: dict, **kwargs): See https://alerce.readthedocs.io/en/stable/ for details. """ form_parameters = parameters + survey = form_parameters.get("survey", "ZTF") query_params = { "format": "json", - "survey": form_parameters.get("survey", "").lower(), + "survey": survey.lower(), } + if survey == "LSST": + lsst_object_type = form_parameters.get("lsst_object_type") or "diaObject" + query_params["sid"] = 2 if lsst_object_type == "ssObject" else 1 + else: + query_params["sid"] = 0 if (firstmjd_gt := form_parameters.get("firstmjd_gt")) and ( firstmjd_lt := form_parameters.get("firstmjd_lt") ): query_params["firstmjd"] = [firstmjd_gt, firstmjd_lt] - if (lastmjd_gt := form_parameters.get("lastmjd_gt")) and ( - lastmjd_lt := form_parameters.get("lastmjd_lt") - ): - query_params["lastmjd"] = [lastmjd_gt, lastmjd_lt] + # lastmjd supports an open-ended lower bound (e.g. "last detected after X", + # a common recency filter -- see ALeRCE's own LSST TAP queries notebook), + # unlike firstmjd above which only makes sense as a closed range. + if lastmjd_gt := form_parameters.get("lastmjd_gt"): + lastmjd_lt = form_parameters.get("lastmjd_lt") + query_params["lastmjd"] = [lastmjd_gt, lastmjd_lt] if lastmjd_lt else [lastmjd_gt] # Build ndet list: # gives range of number of detections based on min/max set in form. @@ -199,28 +690,58 @@ def build_query_parameters(self, parameters: dict, **kwargs): query_params["dec"] = dec query_params["radius"] = radius + if survey == "LSST" and form_parameters.get("deltamjd_max") is not None: + query_params["deltamjd_max"] = form_parameters["deltamjd_max"] + + if max_results := form_parameters.get("max_results"): + query_params["page_size"] = max_results + if order_by := form_parameters.get("order_by"): + query_params["order_by"] = order_by + query_params["order_mode"] = form_parameters.get("order_mode") or "DESC" + if form_parameters.get("object_id"): query_params["oid"] = form_parameters.get("object_id") query_params["classifiers"] = form_parameters.get("classifiers", []) return query_params def build_query_parameters_from_target(self, target, **kwargs): - query_parameters = {"object_id": target.name} - try: - query_parameters["classifiers"] = [ - target.targetextra_set.get(key="classifier").value - ] - query_parameters["survey"] = target.targetextra_set.get(key="survey").value - except TargetExtra.DoesNotExist: - if target.name.startswith("ZTF"): - query_parameters["survey"] = "ZTF" - else: - query_parameters["survey"] = "LSST" - return query_parameters + """ + Finds the ALeRCE object for a target from its name or aliases, so data can be + updated for targets created elsewhere (e.g. named by TNS or the MPC) that carry a + ZTF or LSST ID, a TNS name or an MPC designation among their names. Names that are + already ALeRCE IDs are tried first, so no remote lookup happens when one exists; + then TNS names and MPC designations are resolved remotely, in the same name-then- + alias order. If nothing resolves, the target's name is used as before. + """ + names = [str(target.name)] + [str(alias) for alias in target.aliases.values_list("name", flat=True)] + for resolve in (_alerce_id_from_name, _alerce_id_from_remote_name): + for name in names: + if found := resolve(name): + survey, object_id = found + return {"object_id": object_id, "survey": survey} + return {"object_id": str(target.name), "survey": "ZTF" if str(target.name).startswith("ZTF") else "LSST"} def create_target_from_query(self, target_result: dict, **kwrags): + """ + LSST ssObjects (sid=2) become NON_SIDEREAL targets built from their + `alerce_tap.lsst_mpc_orbits` elements. If no orbit is stored, or the TAP query + fails, the target falls back to SIDEREAL at the object's mean position, since + that's better than failing target creation. Everything else is SIDEREAL. + """ + # LSST oids come back from TAP as integers; the Target must hold the string the DB will + # store, since code after to_target() uses the unsaved-then-saved instance directly. + name = str(target_result["oid"]) + if target_result.get("sid") == 2: + try: + orbit = _fetch_lsst_mpc_orbit(target_result["oid"]) + except pyvo.dal.DALAccessError: + logger.exception(f"Error querying ALeRCE MPC orbit for ssObject {name}") + orbit = None + if orbit: + return _non_sidereal_target_from_mpc_orbit(name, orbit) + logger.warning(f"No ALeRCE MPC orbit for ssObject {name}; creating a SIDEREAL target") target = Target( - name=target_result["oid"], + name=name, type="SIDEREAL", ra=target_result["meanra"], dec=target_result["meandec"], @@ -237,6 +758,37 @@ def create_target_extras_from_query(self, query_results, **kwrags): if k not in ["oid", "meanra", "meandec"] } + def query_aliases(self, query_parameters=None, target=None, **kwargs) -> list: + """ + Returns the MPC names of a non-sidereal target named by its LSST ssObjectId, so it + can be found by its familiar names: the provisional designation (e.g. "1988 JC1"), + and from the MPC orbit record the permanent number ("6478"), name ("Gault") and + secondary designations ("1995 KC1"). The target keeps the ssObjectId as its name. + Other targets have no ALeRCE aliases. A TAP failure is logged and yields no aliases + rather than failing target creation or data update. + """ + if target is None or target.type != Target.NON_SIDEREAL or not LSST_OID_PATTERN.match(str(target.name)): + return [] + try: + orbit = _fetch_lsst_mpc_orbit(target.name) + except pyvo.dal.DALAccessError: + logger.exception(f"Error querying ALeRCE MPC designation for {target.name}") + return [] + if not orbit: + return [] + designation_data = _mpc_designation_data(orbit) + names = [ + orbit.get("designation"), + designation_data.get("permid"), + designation_data.get("name"), + *(designation_data.get("unpacked_secondary_provisional_designations") or []), + ] + aliases = [] + for name in names: + if name and str(name) != str(target.name) and str(name) not in aliases: + aliases.append(str(name)) + return aliases + def query_photometry(self, query_parameters, **kwargs): try: return alerce.query_lightcurve( @@ -263,33 +815,55 @@ def query_forced_photometry(self, query_parameters, **kwargs): return [] def create_reduced_datums_from_query(self, target, data=None, data_type="photometry", **kwargs): + """ + Creates PhotometryReducedDatums from an ALeRCE light curve. Detections in the + LSST shape (`psfFlux` in nJy, integer `band`, TAI `mjd`) are converted to AB + magnitudes via `_lsst_detection_photometry`; ZTF detections (`magpsf`, `fid`) + and non-detections are stored as-is. LSST detections with non-positive + difference flux have no magnitude and are skipped. + + Existing datums are matched on the fields of the `PhotometryReducedDatum` + uniqueness constraints only, so re-querying a light curve whose uncertainties + have since been revised leaves the stored datums unchanged rather than failing. + """ reduced_datums = [] + + def get_or_create_datum(record, lookup, defaults): + try: + reduced_datum, __ = PhotometryReducedDatum.objects.get_or_create( + target=target, defaults={"unit": "mag", "source_name": self.name, **defaults}, **lookup + ) + except (IntegrityError, ValidationError) as e: + raise QueryServiceError(f"Error importing ReducedDatum (target:{target} data:{record}) -- {e}") + reduced_datums.append(reduced_datum) + if data: for detection in data.get("detections", []): - mjd = Time(detection["mjd"], format="mjd", scale="utc") - try: - reduced_datum, __ = PhotometryReducedDatum.objects.get_or_create( - timestamp=mjd.to_datetime(TimezoneInfo()), - target=target, - brightness=detection["magpsf"], - brightness_error=detection["sigmapsf"], - unit='mag', - bandpass=ALERCE_FILTERS[detection["fid"]], - defaults={'source_name': self.name} - ) - reduced_datums.append(reduced_datum) - except IntegrityError as e: - raise QueryServiceError(f"Error importing ReducedDatum (target:{target} data:{detection}) -- {e}") + if "psfFlux" in detection: + photometry = _lsst_detection_photometry(detection) + if photometry is None: + continue + mjd = Time(detection["mjd"], format="mjd", scale="tai").utc + else: + photometry = { + "brightness": detection["magpsf"], + "brightness_error": detection["sigmapsf"], + "bandpass": ALERCE_FILTERS[detection["fid"]], + } + mjd = Time(detection["mjd"], format="mjd", scale="utc") + defaults = { + field: photometry.pop(field) for field in ("brightness_error", "telescope") if field in photometry + } + get_or_create_datum( + detection, {"timestamp": mjd.to_datetime(TimezoneInfo()), **photometry}, defaults + ) for non_detection in data.get("non_detections", []): mjd = Time(non_detection["mjd"], format="mjd", scale="utc") - reduced_datum, __ = PhotometryReducedDatum.objects.get_or_create( - timestamp=mjd.to_datetime(TimezoneInfo()), - target=target, - limit=non_detection["diffmaglim"], - unit='mag', - bandpass=ALERCE_FILTERS[non_detection["fid"]], - defaults={'source_name': self.name} - ) - reduced_datums.append(reduced_datum) + lookup = { + "timestamp": mjd.to_datetime(TimezoneInfo()), + "limit": non_detection["diffmaglim"], + "bandpass": ALERCE_FILTERS[non_detection["fid"]], + } + get_or_create_datum(non_detection, lookup, {}) return reduced_datums diff --git a/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_advanced_form.html b/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_advanced_form.html index afbd30255..dce02055c 100644 --- a/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_advanced_form.html +++ b/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_advanced_form.html @@ -6,7 +6,13 @@

Classification Filters

{% for field in form %} {% if field|is_cfield %} -
{% bootstrap_field field %}
+ {# Every survey's classifier fields are present on the form (see #} + {# AlerceForm.add_classifiers_fields); only the ones for the currently selected #} + {# survey (tracked in the shared $store.alerce, set up in alerce_simple_form.html) #} + {# are shown, so switching the Survey dropdown updates this list with no page reload. #} +
+ {% bootstrap_field field %} +
{% endif %} {% endfor %}
@@ -33,3 +39,13 @@

Other Filters

{% bootstrap_field form.ndet_min %}
{% bootstrap_field form.ndet_max %}
+
+
{% bootstrap_field form.deltamjd_max %}
+
+

Results

+
+
+
{% bootstrap_field form.max_results %}
+
{% bootstrap_field form.order_by %}
+
{% bootstrap_field form.order_mode %}
+
diff --git a/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_results_table.html b/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_results_table.html index 0433c752a..d67a4932a 100644 --- a/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_results_table.html +++ b/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_results_table.html @@ -15,7 +15,13 @@ {% for result in results %} - {{ result.oid}} + + {% if result.alerce_url %} + {{ result.oid }} + {% else %} + {{ result.oid }} + {% endif %} + {{ result.ndet}} {{ result.meanra }} {{ result.meandec }} diff --git a/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_simple_form.html b/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_simple_form.html index e6cb3ed8f..77effef40 100644 --- a/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_simple_form.html +++ b/tom_dataservices/templates/tom_dataservices/alerce/partials/alerce_simple_form.html @@ -1,3 +1,17 @@ {% load django_bootstrap5 %} -{% bootstrap_field form.survey %} -{% bootstrap_field form.object_id %} + +
+ {% bootstrap_field form.survey %} +
+ {% bootstrap_field form.lsst_object_type %} +
+ {% bootstrap_field form.object_id %} +
diff --git a/tom_dataservices/templatetags/dataservices_extras.py b/tom_dataservices/templatetags/dataservices_extras.py index 49f2118e8..8c89caf66 100644 --- a/tom_dataservices/templatetags/dataservices_extras.py +++ b/tom_dataservices/templatetags/dataservices_extras.py @@ -28,6 +28,19 @@ def is_cfield(field): return "cfield" in field.name +@register.filter +def cfield_survey(field): + """ + For an AlerceForm classifier field (`cfield_{survey}__{classifier_name}` or + `prob_cfield_{survey}__{classifier_name}`), returns the survey it belongs to, e.g. + 'ZTF' or 'LSST'. Used to show only the currently selected survey's classifier + fields client-side. + """ + prefix = "cfield_" + start = field.name.index(prefix) + len(prefix) + return field.name[start:].split("__", 1)[0] + + @register.inclusion_tag('tom_dataservices/partials/update_data_from_dataservice.html', takes_context=True) def update_data_from_dataservice(context): initial = {'target': context['target']} diff --git a/tom_dataservices/tests/data_services/test_alerce.py b/tom_dataservices/tests/data_services/test_alerce.py new file mode 100644 index 000000000..e40923577 --- /dev/null +++ b/tom_dataservices/tests/data_services/test_alerce.py @@ -0,0 +1,1044 @@ +import json +import math +from unittest.mock import patch + +from alerce.exceptions import APIError, ObjectNotFoundError +from astropy.time import Time, TimezoneInfo +from django.core.cache import cache +from django.core.exceptions import ValidationError +from django.template.loader import render_to_string +from django.test import TestCase + +import numpy as np +import pyvo +import requests + +from tom_dataservices.data_services.alerce import ( + AlerceDataService, + AlerceForm, + _build_tap_classifier_query, + _build_tap_object_query, + _group_tap_classifier_rows, +) +from tom_dataproducts.models import PhotometryReducedDatum +from tom_dataservices.dataservices import NotConfiguredError, QueryServiceError +from tom_targets.models import Target, TargetName + +MOCK_CLASSIFIERS = [ + { + "classifier_id": 10, + "classifier_name": "lc_classifier", + "classifier_version": "hierarchical_random_forest_1.0.0", + "classes": ["SNIa", "SNII", "AGN"], + "class_ids": {"SNIa": 0, "SNII": 1, "AGN": 2}, + }, + { + "classifier_id": 11, + "classifier_name": "stamp_classifier", + "classifier_version": "stamp_classifier_1.0.1", + "classes": ["AGN", "SN", "bogus"], + "class_ids": {"AGN": 0, "SN": 1, "bogus": 2}, + }, +] + +# alerce_tap.classifier JOIN alerce_tap.taxonomy rows (tid=0, ZTF) that group into MOCK_CLASSIFIERS +MOCK_ZTF_TAP_CLASSIFIER_ROWS = [ + {"classifier_id": 10, "classifier_name": "lc_classifier", + "classifier_version": "hierarchical_random_forest_1.0.0", "class_id": 0, "class_name": "SNIa"}, + {"classifier_id": 10, "classifier_name": "lc_classifier", + "classifier_version": "hierarchical_random_forest_1.0.0", "class_id": 1, "class_name": "SNII"}, + {"classifier_id": 10, "classifier_name": "lc_classifier", + "classifier_version": "hierarchical_random_forest_1.0.0", "class_id": 2, "class_name": "AGN"}, + {"classifier_id": 11, "classifier_name": "stamp_classifier", "classifier_version": "stamp_classifier_1.0.1", + "class_id": 0, "class_name": "AGN"}, + {"classifier_id": 11, "classifier_name": "stamp_classifier", "classifier_version": "stamp_classifier_1.0.1", + "class_id": 1, "class_name": "SN"}, + {"classifier_id": 11, "classifier_name": "stamp_classifier", "classifier_version": "stamp_classifier_1.0.1", + "class_id": 2, "class_name": "bogus"}, +] + +# tid=1 (LSST) rows, exercising the LSST-only stamp classifier +MOCK_LSST_TAP_CLASSIFIER_ROWS = [ + {"classifier_id": 20, "classifier_name": "stamp_classifier_rubin_beta", "classifier_version": "1.0.0", + "class_id": 3, "class_name": "SN"}, + {"classifier_id": 20, "classifier_name": "stamp_classifier_rubin_beta", "classifier_version": "1.0.0", + "class_id": 4, "class_name": "bogus"}, +] + + +def _tap_classifier_side_effect(ztf_rows=None, lsst_rows=None): + """ + add_classifiers_fields() now queries TAP once per survey (so every survey's + classifier fields are always on the form -- see its docstring). Route each + tap_service.search() call's mocked response by the `tid` filter in the ADQL query, + rather than a single fixed return_value, so ZTF (tid=0) and LSST (tid=1) get their + own rows regardless of call order. + """ + ztf_rows = MOCK_ZTF_TAP_CLASSIFIER_ROWS if ztf_rows is None else ztf_rows + lsst_rows = MOCK_LSST_TAP_CLASSIFIER_ROWS if lsst_rows is None else lsst_rows + + def _search(query, *args, **kwargs): + return lsst_rows if "tid = 1" in query else ztf_rows + + return _search + + +class TestAlerceForm(TestCase): + def setUp(self): + cache.clear() + alerce_patcher = patch("tom_dataservices.data_services.alerce.alerce") + self.mock_alerce = alerce_patcher.start() + self.addCleanup(alerce_patcher.stop) + tap_patcher = patch("tom_dataservices.data_services.alerce.tap_service") + self.mock_tap_service = tap_patcher.start() + self.addCleanup(tap_patcher.stop) + self.mock_tap_service.search.side_effect = _tap_classifier_side_effect() + + def test_ztf_classifier_fields_added_dynamically(self): + form = AlerceForm(data={"data_service": "ALeRCE"}) + self.assertIn("cfield_ZTF__lc_classifier", form.fields) + self.assertIn("prob_cfield_ZTF__lc_classifier", form.fields) + self.assertEqual( + form.fields["cfield_ZTF__lc_classifier"].choices, + [(None, "")] + [(k, k) for k in ["SNIa", "SNII", "AGN"]], + ) + self.assertIn("cfield_ZTF__stamp_classifier", form.fields) + + def test_lsst_classifier_fields_added_dynamically(self): + form = AlerceForm(data={"data_service": "ALeRCE"}) + self.assertIn("cfield_LSST__stamp_classifier_rubin_beta", form.fields) + self.assertEqual( + form.fields["cfield_LSST__stamp_classifier_rubin_beta"].choices, + [(None, "")] + [(k, k) for k in ["SN", "bogus"]], + ) + + def test_both_surveys_classifier_fields_present_regardless_of_selection(self): + """ + Every survey's classifier fields must be on the form regardless of which + survey is currently selected, so the advanced form partial can show/hide the + right set purely client-side (via Alpine, keyed off $store.alerce.survey) when + the user switches surveys -- with no full form re-render. + """ + form = AlerceForm(data={"data_service": "ALeRCE", "survey": "ZTF"}) + self.assertIn("cfield_ZTF__lc_classifier", form.fields) + self.assertIn("cfield_LSST__stamp_classifier_rubin_beta", form.fields) + + def test_classifiers_cached_after_first_query(self): + AlerceForm(data={"data_service": "ALeRCE"}) + self.assertEqual(self.mock_tap_service.search.call_count, 2) # one per survey + AlerceForm(data={"data_service": "ALeRCE"}) + self.assertEqual(self.mock_tap_service.search.call_count, 2) # served from cache + + def test_get_classifiers_queries_use_tid_per_survey(self): + AlerceForm(data={"data_service": "ALeRCE"}) + queries = [call.args[0] for call in self.mock_tap_service.search.call_args_list] + self.assertTrue(any("c.tid = 0" in q for q in queries)) + self.assertTrue(any("c.tid = 1" in q for q in queries)) + + def test_per_survey_cache_keys_are_independent(self): + AlerceForm(data={"data_service": "ALeRCE"}) + self.assertIsNotNone(cache.get("ds_alerce_classifiers_0")) + self.assertIsNotNone(cache.get("ds_alerce_classifiers_1")) + + def test_no_rest_query_classifiers_call(self): + AlerceForm(data={"data_service": "ALeRCE"}) + self.mock_alerce.query_classifiers.assert_not_called() + + def test_form_renders_without_classifier_fields_when_tap_fails(self): + self.mock_tap_service.search.side_effect = pyvo.dal.DALServiceError("TAP down") + with self.assertLogs("tom_dataservices.data_services.alerce", level="ERROR"): + form = AlerceForm(data={"data_service": "ALeRCE", "survey": "ZTF"}) + self.assertFalse(any(name.startswith("cfield_") for name in form.fields)) + self.assertIn("object_id", form.fields) + self.assertIsNone(cache.get("ds_alerce_classifiers_0")) + + # Not cached, so the next form load retries and recovers + self.mock_tap_service.search.side_effect = _tap_classifier_side_effect() + form = AlerceForm(data={"data_service": "ALeRCE", "survey": "ZTF"}) + self.assertIn("cfield_ZTF__lc_classifier", form.fields) + + def test_clean_bundles_selected_classifiers_for_selected_survey(self): + form = AlerceForm( + data={ + "data_service": "ALeRCE", + "survey": "ZTF", + "cfield_ZTF__lc_classifier": "SNIa", + "prob_cfield_ZTF__lc_classifier": 0.8, + } + ) + self.assertTrue(form.is_valid(), form.errors) + self.assertEqual( + form.cleaned_data["classifiers"], + [{"classifier": "lc_classifier", "class": "SNIa", "probability": 0.8}], + ) + + def test_clean_ignores_stale_hidden_survey_classifier_value(self): + """ + Regression test for what the client-side-only survey toggle makes possible: a + user picks a ZTF classifier, then switches the Survey dropdown to LSST (no page + reload, so the ZTF field's value is untouched) and submits. The ZTF field is + hidden but still present (and still POSTed); clean() must not bundle it since it + doesn't belong to the now-selected survey. + """ + form = AlerceForm( + data={ + "data_service": "ALeRCE", + "survey": "LSST", + "cfield_ZTF__lc_classifier": "SNIa", + "prob_cfield_ZTF__lc_classifier": 0.8, + } + ) + self.assertTrue(form.is_valid(), form.errors) + self.assertEqual(form.cleaned_data["classifiers"], []) + + +class TestGroupTapClassifierRows(TestCase): + def test_groups_by_classifier_name_and_version_preserving_taxonomy_order(self): + grouped = _group_tap_classifier_rows(MOCK_ZTF_TAP_CLASSIFIER_ROWS) + self.assertEqual(grouped, MOCK_CLASSIFIERS) + + def test_empty_rows_returns_empty_list(self): + self.assertEqual(_group_tap_classifier_rows([]), []) + + +class TestBuildQueryParameters(TestCase): + def setUp(self): + self.ds = AlerceDataService() + + def test_ztf_survey_sid_and_name(self): + params = self.ds.build_query_parameters({"survey": "ZTF"}) + self.assertEqual(params["sid"], 0) + self.assertEqual(params["survey"], "ztf") + + def test_lsst_diaobject_sid(self): + params = self.ds.build_query_parameters({"survey": "LSST", "lsst_object_type": "diaObject"}) + self.assertEqual(params["sid"], 1) + + def test_lsst_ssobject_sid(self): + params = self.ds.build_query_parameters({"survey": "LSST", "lsst_object_type": "ssObject"}) + self.assertEqual(params["sid"], 2) + + def test_firstmjd_lastmjd_only_set_when_both_bounds_given(self): + params = self.ds.build_query_parameters( + {"survey": "ZTF", "firstmjd_gt": 59000.0, "firstmjd_lt": 59500.0} + ) + self.assertEqual(params["firstmjd"], [59000.0, 59500.0]) + self.assertNotIn("lastmjd", params) + + params = self.ds.build_query_parameters({"survey": "ZTF", "firstmjd_gt": 59000.0}) + self.assertNotIn("firstmjd", params) + + def test_lastmjd_gt_only_yields_open_ended_lower_bound(self): + """ + Unlike firstmjd, lastmjd supports an open-ended lower bound (e.g. "last + detected after X") -- a common recency filter, per ALeRCE's own LSST TAP + queries notebook, which only bounds lastmjd from below. + """ + params = self.ds.build_query_parameters({"survey": "ZTF", "lastmjd_gt": 59000.0}) + self.assertEqual(params["lastmjd"], [59000.0]) + + def test_lastmjd_gt_and_lt_yields_closed_range(self): + params = self.ds.build_query_parameters( + {"survey": "ZTF", "lastmjd_gt": 59000.0, "lastmjd_lt": 59500.0} + ) + self.assertEqual(params["lastmjd"], [59000.0, 59500.0]) + + def test_lastmjd_absent_when_gt_not_given(self): + params = self.ds.build_query_parameters({"survey": "ZTF", "lastmjd_lt": 59500.0}) + self.assertNotIn("lastmjd", params) + + def test_ndet_min_and_max(self): + params = self.ds.build_query_parameters({"survey": "ZTF", "ndet_min": 3, "ndet_max": 10}) + self.assertEqual(params["ndet"], [3, 10]) + + def test_ndet_max_only_defaults_min_to_zero(self): + params = self.ds.build_query_parameters({"survey": "ZTF", "ndet_max": 10}) + self.assertEqual(params["ndet"], [0, 10]) + + def test_ndet_min_only(self): + params = self.ds.build_query_parameters({"survey": "ZTF", "ndet_min": 3}) + self.assertEqual(params["ndet"], [3]) + + def test_max_results_and_order_become_page_size_and_order(self): + params = self.ds.build_query_parameters( + {"survey": "ZTF", "max_results": 50, "order_by": "lastmjd", "order_mode": "ASC"} + ) + self.assertEqual(params["page_size"], 50) + self.assertEqual(params["order_by"], "lastmjd") + self.assertEqual(params["order_mode"], "ASC") + + def test_order_mode_defaults_to_desc_and_is_omitted_without_order_by(self): + params = self.ds.build_query_parameters({"survey": "ZTF", "order_by": "ndet"}) + self.assertEqual(params["order_mode"], "DESC") + params = self.ds.build_query_parameters({"survey": "ZTF", "order_mode": "ASC"}) + self.assertNotIn("order_by", params) + self.assertNotIn("order_mode", params) + self.assertNotIn("page_size", params) + + def test_deltamjd_max_only_passed_for_lsst(self): + params = self.ds.build_query_parameters({"survey": "LSST", "deltamjd_max": 0.5}) + self.assertEqual(params["deltamjd_max"], 0.5) + params = self.ds.build_query_parameters({"survey": "LSST", "deltamjd_max": 0.0}) + self.assertEqual(params["deltamjd_max"], 0.0) + params = self.ds.build_query_parameters({"survey": "ZTF", "deltamjd_max": 0.5}) + self.assertNotIn("deltamjd_max", params) + + def test_form_rejects_deltamjd_max_for_ztf(self): + with patch("tom_dataservices.data_services.alerce._fetch_classifiers_for_tid", return_value=[]): + ztf = AlerceForm(data={"data_service": "ALeRCE", "survey": "ZTF", "deltamjd_max": 1}) + lsst = AlerceForm(data={"data_service": "ALeRCE", "survey": "LSST", "deltamjd_max": 1}) + self.assertFalse(ztf.is_valid()) + self.assertIn("deltamjd_max", ztf.errors) + self.assertTrue(lsst.is_valid(), lsst.errors) + + def test_form_rejects_out_of_range_max_results_and_unknown_order(self): + with patch("tom_dataservices.data_services.alerce._fetch_classifiers_for_tid", return_value=[]): + form = AlerceForm(data={"data_service": "ALeRCE", "survey": "ZTF", "max_results": 0, + "order_by": "oid; DROP"}) + self.assertFalse(form.is_valid()) + self.assertIn("max_results", form.errors) + self.assertIn("order_by", form.errors) + + def test_ndet_absent_when_neither_given(self): + params = self.ds.build_query_parameters({"survey": "ZTF"}) + self.assertNotIn("ndet", params) + + def test_cone_params_set_only_when_all_present(self): + params = self.ds.build_query_parameters( + {"survey": "ZTF", "ra": 10.0, "dec": 20.0, "radius": 30.0} + ) + self.assertEqual(params["ra"], 10.0) + self.assertEqual(params["dec"], 20.0) + self.assertEqual(params["radius"], 30.0) + + params = self.ds.build_query_parameters({"survey": "ZTF", "ra": 10.0, "dec": 20.0}) + self.assertNotIn("ra", params) + + def test_object_id_becomes_oid(self): + params = self.ds.build_query_parameters({"survey": "ZTF", "object_id": "ZTF18aaaaaa"}) + self.assertEqual(params["oid"], "ZTF18aaaaaa") + + def test_classifiers_defaults_to_empty_list(self): + params = self.ds.build_query_parameters({"survey": "ZTF"}) + self.assertEqual(params["classifiers"], []) + + +class TestBuildTapObjectQuery(TestCase): + def test_top_and_sid(self): + query = _build_tap_object_query({"sid": 1}) + self.assertIn("SELECT TOP 20 * FROM alerce_tap.object", query) + self.assertIn("WHERE sid = 1", query) + + def test_page_size_override(self): + query = _build_tap_object_query({"sid": 2}, page_size=50) + self.assertIn("SELECT TOP 50", query) + + def test_page_size_from_query_parameters(self): + self.assertIn("SELECT TOP 75 ", _build_tap_object_query({"sid": 1, "page_size": 75})) + self.assertIn("SELECT TOP 20 ", _build_tap_object_query({"sid": 1})) + + def test_order_by_maps_ndet_to_n_det(self): + query = _build_tap_object_query({"sid": 1, "ndet": [3], "order_by": "ndet", "order_mode": "DESC"}) + self.assertTrue(query.endswith("AND n_det >= 3 ORDER BY n_det DESC")) + + def test_no_order_clause_by_default(self): + self.assertNotIn("ORDER BY", _build_tap_object_query({"sid": 1})) + + def test_deltamjd_max(self): + self.assertIn("AND deltamjd <= 1.0", _build_tap_object_query({"sid": 1, "deltamjd_max": 1})) + self.assertIn("AND deltamjd <= 0.0", _build_tap_object_query({"sid": 1, "deltamjd_max": 0})) + self.assertNotIn("deltamjd", _build_tap_object_query({"sid": 1})) + + def test_unknown_order_column_and_mode_not_interpolated(self): + query = _build_tap_object_query({"sid": 1, "order_by": "oid; DROP TABLE x", "order_mode": "ASC"}) + self.assertNotIn("ORDER BY", query) + query = _build_tap_object_query({"sid": 1, "order_by": "lastmjd", "order_mode": "ASC; --"}) + self.assertTrue(query.endswith("ORDER BY lastmjd DESC")) + + def test_cone_search_converts_radius_arcsec_to_degrees(self): + query = _build_tap_object_query({"sid": 1, "ra": 305.58, "dec": -18.79, "radius": 3600.0}) + self.assertIn( + "CONTAINS(POINT('ICRS', meanra, meandec), CIRCLE('ICRS', 305.58, -18.79, 1.0))", + query, + ) + + def test_cone_search_absent_when_any_param_missing(self): + query = _build_tap_object_query({"sid": 1, "ra": 305.58, "dec": -18.79}) + self.assertNotIn("CONTAINS", query) + + def test_mjd_ranges(self): + query = _build_tap_object_query({"sid": 1, "firstmjd": [59000.0, 59500.0], "lastmjd": [59100.0, 59600.0]}) + self.assertIn("AND firstmjd >= 59000.0 AND firstmjd <= 59500.0", query) + self.assertIn("AND lastmjd >= 59100.0 AND lastmjd <= 59600.0", query) + + def test_lastmjd_one_sided(self): + query = _build_tap_object_query({"sid": 1, "lastmjd": [59100.0]}) + self.assertIn("AND lastmjd >= 59100.0", query) + self.assertNotIn("lastmjd <=", query) + + def test_ndet_two_sided(self): + query = _build_tap_object_query({"sid": 1, "ndet": [3, 10]}) + self.assertIn("AND n_det >= 3", query) + self.assertIn("AND n_det <= 10", query) + + def test_ndet_one_sided(self): + query = _build_tap_object_query({"sid": 1, "ndet": [3]}) + self.assertIn("AND n_det >= 3", query) + self.assertNotIn("n_det <=", query) + + +class TestBuildTapClassifierQuery(TestCase): + def test_joins_object_and_probability_filtered_by_ids_and_ranking(self): + query = _build_tap_classifier_query({"sid": 1}, classifier_id=20, class_id=3) + self.assertIn("FROM alerce_tap.object AS obj", query) + self.assertIn("JOIN alerce_tap.probability AS prob", query) + self.assertIn("obj.oid = prob.oid AND obj.sid = prob.sid", query) + self.assertIn("obj.sid = 1", query) + self.assertIn("prob.classifier_id = 20", query) + self.assertIn("prob.class_id = 3", query) + self.assertIn("prob.ranking = 1", query) + self.assertIn("ORDER BY prob.probability DESC", query) + self.assertNotIn("probability >=", query) + + def test_probability_threshold_appended_when_given(self): + query = _build_tap_classifier_query({"sid": 1}, classifier_id=20, class_id=3, probability=0.9) + self.assertIn("AND prob.probability >= 0.9", query) + + def test_shared_filters_use_obj_column_prefix(self): + query = _build_tap_classifier_query( + {"sid": 1, "ra": 305.58, "dec": -18.79, "radius": 3600.0, "ndet": [3]}, + classifier_id=20, + class_id=3, + ) + self.assertIn("CONTAINS(POINT('ICRS', obj.meanra, obj.meandec)", query) + self.assertIn("AND obj.n_det >= 3", query) + + def test_asteroid_candidate_query_filters_deltamjd_on_obj(self): + query = _build_tap_classifier_query({"sid": 1, "deltamjd_max": 1.0}, classifier_id=3, class_id=3) + self.assertIn("AND obj.deltamjd <= 1.0", query) + + def test_page_size_and_order_override_probability_default(self): + query = _build_tap_classifier_query( + {"sid": 1, "page_size": 100, "order_by": "firstmjd", "order_mode": "ASC"}, classifier_id=20, class_id=3, + ) + self.assertTrue(query.startswith("SELECT TOP 100 ")) + self.assertTrue(query.endswith(" ORDER BY obj.firstmjd ASC")) + self.assertNotIn("ORDER BY prob.probability", query) + + +class TestQueryService(TestCase): + def setUp(self): + cache.clear() + self.ds = AlerceDataService() + alerce_patcher = patch("tom_dataservices.data_services.alerce.alerce") + self.mock_alerce = alerce_patcher.start() + self.addCleanup(alerce_patcher.stop) + tap_patcher = patch("tom_dataservices.data_services.alerce.tap_service") + self.mock_tap_service = tap_patcher.start() + self.addCleanup(tap_patcher.stop) + + def test_ztf_oid_path_uses_rest_and_strips_sid(self): + # query_objects returns a paginated wrapper, not the object itself + self.mock_alerce.query_objects.return_value = { + "total": 1, "page": 1, "items": [{"oid": "ZTF18aaaaaa", "meanra": 10.0, "deltajd": 3.0}], + } + result = self.ds.query_service({"oid": "ZTF18aaaaaa", "sid": 0, "survey": "ztf"}) + self.mock_alerce.query_objects.assert_called_once() + call_kwargs = self.mock_alerce.query_objects.call_args.kwargs + self.assertNotIn("sid", call_kwargs) + self.assertEqual(result, [{ + "oid": "ZTF18aaaaaa", "meanra": 10.0, "deltamjd": 3.0, "survey": "ztf", + "alerce_url": "https://alerce.online/object/ZTF18aaaaaa", + }]) + + def test_oid_lookups_with_no_match_return_no_results(self): + self.mock_alerce.query_objects.return_value = {"total": 0, "page": 1, "items": []} + self.assertEqual(self.ds.query_service({"oid": "ZTF18zzzzzz", "sid": 0, "survey": "ztf"}), []) + self.mock_tap_service.search.return_value = [] + self.assertEqual(self.ds.query_service({"oid": "1", "sid": 1, "survey": "lsst"}), []) + + def test_lsst_oid_path_uses_tap_and_renames_ndet(self): + row = { + "oid": 12345, + "meanra": np.float64(10.0), + "meandec": np.float64(-5.0), + "n_det": np.int64(7), + } + self.mock_tap_service.search.return_value = [row] + result = self.ds.query_service({"oid": 12345, "sid": 1, "survey": "lsst"}) + + adql = self.mock_tap_service.search.call_args.args[0] + self.assertIn("oid =", adql) + self.assertIn("sid = 1", adql) + self.assertEqual(len(result), 1) + self.assertEqual(result[0]["ndet"], 7) + self.assertNotIn("n_det", result[0]) + self.assertIsInstance(result[0]["ndet"], int) + self.assertIsInstance(result[0]["meanra"], float) + self.assertEqual(result[0]["survey"], "lsst") + self.assertEqual(result[0]["alerce_url"], "https://lsst.alerce.online/object/12345?survey=lsst") + + def test_lsst_non_numeric_oid_raises_query_service_error_without_querying(self): + with self.assertRaises(QueryServiceError): + self.ds.query_service({"oid": "1 OR 1=1", "sid": 1, "survey": "lsst"}) + self.mock_tap_service.search.assert_not_called() + + def test_lsst_designation_resolves_to_ssobject(self): + self.mock_tap_service.search.side_effect = [ + [{"ssobjectid": np.int64(21164706944660568)}], + [{"oid": np.int64(21164706944660568), "sid": np.int16(2), "meanra": np.float64(1.0)}], + ] + # sid=1 (diaObject selected) is overridden: designations are always ssObjects + result = self.ds.query_service({"oid": " 2010 WX64 ", "sid": 1, "survey": "lsst"}) + designation_adql, object_adql = [c.args[0] for c in self.mock_tap_service.search.call_args_list] + self.assertIn("FROM alerce_tap.lsst_mpc_orbits WHERE designation = '2010 WX64'", designation_adql) + self.assertIn("oid = 21164706944660568 AND sid = 2", object_adql) + self.assertEqual(result[0]["oid"], 21164706944660568) + + def test_lsst_unknown_designation_returns_no_results(self): + self.mock_tap_service.search.return_value = [] + self.assertEqual(self.ds.query_service({"oid": "1999 XX999", "sid": 2, "survey": "lsst"}), []) + self.mock_tap_service.search.assert_called_once() + + def test_lsst_designation_with_quotes_rejected_without_querying(self): + with self.assertRaises(QueryServiceError): + self.ds.query_service({"oid": "2010 WX64' OR '1'='1", "sid": 2, "survey": "lsst"}) + self.mock_tap_service.search.assert_not_called() + + def test_tap_errors_raise_query_service_error(self): + queries = [ + {"sid": 1, "survey": "lsst", "classifiers": []}, + {"oid": "12345", "sid": 1, "survey": "lsst"}, + {"oid": "2010 WX64", "sid": 2, "survey": "lsst"}, + ] + for error in (pyvo.dal.DALServiceError("TAP down"), pyvo.dal.DALQueryError("bad ADQL")): + self.mock_tap_service.search.side_effect = error + for query_parameters in queries: + with self.subTest(error=type(error).__name__, query=query_parameters): + with self.assertLogs("tom_dataservices.data_services.alerce", level="ERROR"): + with self.assertRaises(QueryServiceError): + self.ds.query_service(dict(query_parameters)) + + def test_ztf_general_query_unwraps_items_and_annotates_survey(self): + self.mock_alerce.query_objects.return_value = { + "items": [{"oid": "ZTF18aaaaaa"}, {"oid": "ZTF18bbbbbb"}] + } + result = self.ds.query_service({"sid": 0, "survey": "ztf", "classifiers": []}) + self.assertEqual(len(result), 2) + self.assertTrue(all(r["survey"] == "ztf" for r in result)) + + def test_ztf_classifier_loop_queries_once_per_classifier(self): + self.mock_alerce.query_objects.return_value = {"items": [{"oid": "ZTF18aaaaaa"}]} + classifiers = [ + {"classifier": "lc_classifier", "class": "SNIa", "probability": 0.5}, + {"classifier": "stamp_classifier", "class": "AGN", "probability": 0.9}, + ] + result = self.ds.query_service( + {"sid": 0, "survey": "ztf", "classifiers": classifiers} + ) + self.assertEqual(self.mock_alerce.query_objects.call_count, 2) + first_call_kwargs = self.mock_alerce.query_objects.call_args_list[0].kwargs + self.assertEqual(first_call_kwargs["classifier"], "lc_classifier") + self.assertEqual(first_call_kwargs["class_name"], "SNIa") + self.assertEqual(first_call_kwargs["probability"], 0.5) + self.assertEqual(len(result), 2) + + def test_object_not_found_error_raises_query_service_error(self): + self.mock_alerce.query_objects.side_effect = ObjectNotFoundError("nope") + with self.assertRaises(QueryServiceError): + self.ds.query_service({"sid": 0, "survey": "ztf", "classifiers": []}) + + def test_api_error_raises_query_service_error(self): + self.mock_alerce.query_objects.side_effect = APIError("boom") + with self.assertRaises(QueryServiceError): + self.ds.query_service({"sid": 0, "survey": "ztf", "classifiers": []}) + + def test_value_error_raises_query_service_error(self): + self.mock_alerce.query_objects.side_effect = ValueError("bad param") + with self.assertRaises(QueryServiceError): + self.ds.query_service({"sid": 0, "survey": "ztf", "classifiers": []}) + + def test_lsst_general_query_returns_annotated_results(self): + """ + LSST general (non-oid) queries now go through TAP against alerce_tap.object + instead of the REST client (whose query_objects(survey='lsst') returns a + bare list, breaking the ZTF-shaped `.get("items", [])` unwrap). + """ + rows = [ + {"oid": 12345, "meanra": np.float64(10.0), "n_det": np.int64(5)}, + {"oid": 67890, "meanra": np.float64(20.0), "n_det": np.int64(9)}, + ] + self.mock_tap_service.search.return_value = rows + result = self.ds.query_service({"sid": 1, "survey": "lsst", "classifiers": []}) + self.mock_tap_service.search.assert_called_once() + self.assertEqual(len(result), 2) + self.assertTrue(all(r["survey"] == "lsst" for r in result)) + self.assertEqual(result[0]["ndet"], 5) + self.assertNotIn("n_det", result[0]) + + def test_results_annotated_with_explorer_url(self): + self.mock_tap_service.search.return_value = [{"oid": 12345, "meanra": np.float64(10.0)}] + lsst = self.ds.query_service({"sid": 1, "survey": "lsst", "classifiers": []}) + self.assertEqual(lsst[0]["alerce_url"], "https://lsst.alerce.online/object/12345?survey=lsst") + + self.mock_alerce.query_objects.return_value = {"items": [{"oid": "ZTF18aaaaaa"}]} + ztf = self.ds.query_service({"survey": "ztf", "classifiers": []}) + self.assertEqual(ztf[0]["alerce_url"], "https://alerce.online/object/ZTF18aaaaaa") + + def test_results_table_links_oid_to_explorer(self): + html = render_to_string( + AlerceDataService.query_results_table, + {"results": [ + {"id": 1, "oid": 12345, "alerce_url": "https://lsst.alerce.online/object/12345?survey=lsst"}, + {"id": 2, "oid": 67890}, + ]}, + ) + self.assertIn('= 0.9", probability_query) + self.assertEqual(len(result), 1) + self.assertEqual(result[0]["ndet"], 3) + + def test_lsst_classifier_results_carry_class_and_classifier_names(self): + classifiers = [{"classifier": "stamp_classifier_rubin_beta", "class": "SN", "probability": None}, + {"classifier": "stamp_classifier_rubin_beta", "class": "bogus", "probability": None}] + row_sn = {"oid": 1, "probability": np.float64(0.95), "ranking": np.int64(1)} + row_bogus = {"oid": 2, "probability": np.float64(0.80), "ranking": np.int64(1)} + self.mock_tap_service.search.side_effect = [MOCK_LSST_TAP_CLASSIFIER_ROWS, [row_sn], [row_bogus]] + + result = self.ds.query_service({"sid": 1, "survey": "lsst", "classifiers": classifiers}) + + self.assertEqual([(r["oid"], r["class"], r["classifier"]) for r in result], + [(1, "SN", "stamp_classifier_rubin_beta"), (2, "bogus", "stamp_classifier_rubin_beta")]) + html = render_to_string(AlerceDataService.query_results_table, {"results": result}) + self.assertIn("SN [0.95]", html) + self.assertIn("bogus [0.8]", html) + + def test_lsst_classifier_query_ranking_default_when_no_probability_threshold(self): + classifiers = [{"classifier": "stamp_classifier_rubin_beta", "class": "SN", "probability": None}] + self.mock_tap_service.search.side_effect = [MOCK_LSST_TAP_CLASSIFIER_ROWS, []] + + self.ds.query_service({"sid": 1, "survey": "lsst", "classifiers": classifiers}) + + probability_query = self.mock_tap_service.search.call_args_list[1].args[0] + self.assertNotIn("probability >=", probability_query) + self.assertIn("ranking = 1", probability_query) + + def test_lsst_ssobject_classifier_query_raises_query_service_error(self): + """ + Known ssObjects (sid=2) are pre-assigned probability 1 "asteroid" rather + than classified, so classifier queries against sid=2 are rejected. + """ + classifiers = [{"classifier": "stamp_classifier_rubin_beta", "class": "SN", "probability": 0.5}] + with self.assertRaises(QueryServiceError): + self.ds.query_service({"sid": 2, "survey": "lsst", "classifiers": classifiers}) + self.mock_tap_service.search.assert_not_called() + + def test_lsst_classifier_query_unknown_classifier_raises_query_service_error(self): + classifiers = [{"classifier": "nonexistent_classifier", "class": "SN", "probability": 0.5}] + self.mock_tap_service.search.return_value = MOCK_LSST_TAP_CLASSIFIER_ROWS + with self.assertRaises(QueryServiceError): + self.ds.query_service({"sid": 1, "survey": "lsst", "classifiers": classifiers}) + + def test_ztf_general_query_normalizes_deltajd_to_deltamjd(self): + self.mock_alerce.query_objects.return_value = { + "items": [{"oid": "ZTF18aaaaaa", "deltajd": 12.5}] + } + result = self.ds.query_service({"sid": 0, "survey": "ztf", "classifiers": []}) + self.assertEqual(result[0]["deltamjd"], 12.5) + self.assertNotIn("deltajd", result[0]) + + +class TestTargetAndDatumCreation(TestCase): + def setUp(self): + self.ds = AlerceDataService() + + def test_create_target_from_query(self): + target = self.ds.create_target_from_query( + {"oid": "ZTF18aaaaaa", "meanra": 10.0, "meandec": -5.0} + ) + self.assertEqual(target.name, "ZTF18aaaaaa") + self.assertEqual(target.type, "SIDEREAL") + self.assertEqual(target.ra, 10.0) + self.assertEqual(target.dec, -5.0) + + def test_create_target_extras_from_query_excludes_core_fields(self): + extras = self.ds.create_target_extras_from_query( + {"oid": "ZTF18aaaaaa", "meanra": 10.0, "meandec": -5.0, "ndet": 7, "survey": "ztf"} + ) + self.assertNotIn("oid", extras) + self.assertNotIn("meanra", extras) + self.assertNotIn("meandec", extras) + self.assertEqual(extras, {"ndet": 7, "survey": "ztf"}) + + def test_create_reduced_datums_from_ztf_detections_and_non_detections(self): + target = Target.objects.create(name="ZTF18aaaaaa", type="SIDEREAL", ra=10.0, dec=-5.0) + data = { + "detections": [ + {"mjd": 59000.0, "magpsf": 18.5, "sigmapsf": 0.1, "fid": 1}, + ], + "non_detections": [ + {"mjd": 58999.0, "diffmaglim": 20.0, "fid": 2}, + ], + } + reduced_datums = self.ds.create_reduced_datums_from_query(target, data=data) + self.assertEqual(len(reduced_datums), 2) + + def test_requery_with_revised_uncertainties_keeps_existing_datums(self): + """ + A re-queried light curve whose uncertainties have changed is the same measurement: + it must match the stored datums rather than fail the uniqueness constraints (#1669). + """ + target = Target.objects.create(name="ZTF18aaaaaa", type="SIDEREAL", ra=10.0, dec=-5.0) + ztf = {"mjd": 59000.0, "magpsf": 18.5, "sigmapsf": 0.1, "fid": 1} + lsst = {"mjd": 61000.0, "psfFlux": 123.4, "psfFluxErr": 5.0, "band": 1} + non_detection = {"mjd": 58999.0, "diffmaglim": 20.0, "fid": 2} + + first = self.ds.create_reduced_datums_from_query( + target, data={"detections": [ztf, lsst], "non_detections": [non_detection]} + ) + revised = {"detections": [{**ztf, "sigmapsf": 0.2}, {**lsst, "psfFluxErr": 9.0}], + "non_detections": [non_detection]} + second = self.ds.create_reduced_datums_from_query(target, data=revised) + + self.assertEqual([datum.pk for datum in second], [datum.pk for datum in first]) + self.assertEqual(PhotometryReducedDatum.objects.filter(target=target).count(), 3) + self.assertEqual(second[0].brightness_error, 0.1) + self.assertAlmostEqual(second[1].brightness_error, 2.5 / math.log(10) * 5.0 / 123.4) + + def test_rejected_datums_raise_query_service_error(self): + target = Target.objects.create(name="ZTF18aaaaaa", type="SIDEREAL", ra=10.0, dec=-5.0) + cases = { + "detection": {"detections": [{"mjd": 59000.0, "magpsf": 18.5, "sigmapsf": 0.1, "fid": 1}]}, + "non_detection": {"non_detections": [{"mjd": 58999.0, "diffmaglim": 20.0, "fid": 2}]}, + } + for label, data in cases.items(): + with self.subTest(label), \ + patch.object(PhotometryReducedDatum.objects, "get_or_create", + side_effect=ValidationError("duplicate")): + with self.assertRaises(QueryServiceError): + self.ds.create_reduced_datums_from_query(target, data=data) + + def test_lsst_detection_fixture_creates_reduced_datum(self): + """ + LSST detections carry difference flux in nJy (`psfFlux`) and an integer + `band`, not the ZTF REST shape's `magpsf`/`sigmapsf`/`fid`. + """ + target = Target.objects.create(name="LSST12345", type="SIDEREAL", ra=10.0, dec=-5.0) + data = { + "detections": [ + {"mjd": 59000.0, "psfFlux": 123.4, "band": 4}, + ], + "non_detections": [], + } + reduced_datums = self.ds.create_reduced_datums_from_query(target, data=data) + self.assertEqual(len(reduced_datums), 1) + datum = reduced_datums[0] + self.assertAlmostEqual(datum.brightness, 31.4 - 2.5 * math.log10(123.4)) + self.assertIsNone(datum.brightness_error) + self.assertEqual(datum.bandpass, "z") + self.assertEqual(datum.unit, "mag") + self.assertEqual(datum.telescope, "Rubin") + self.assertEqual(datum.instrument, "LSSTCam") + + def test_lsst_detection_flux_error_and_band_name(self): + target = Target.objects.create(name="LSST12345", type="SIDEREAL", ra=10.0, dec=-5.0) + data = { + "detections": [ + {"mjd": 61000.5, "psfFlux": 3631e9 * 1e-8, "psfFluxErr": 3631e9 * 1e-9, + "band": 6, "band_name": "u"}, + ], + } + datum = self.ds.create_reduced_datums_from_query(target, data=data)[0] + # 1e-8 of the 3631 Jy AB reference -> 20 mag; 10% flux error -> ~0.1086 mag + self.assertAlmostEqual(datum.brightness, 20.0, places=2) + self.assertAlmostEqual(datum.brightness_error, 2.5 / math.log(10) * 0.1) + self.assertEqual(datum.bandpass, "u") + + def test_lsst_detection_mjd_is_tai(self): + target = Target.objects.create(name="LSST12345", type="SIDEREAL", ra=10.0, dec=-5.0) + data = {"detections": [{"mjd": 61000.0, "psfFlux": 100.0, "band": 1}]} + datum = self.ds.create_reduced_datums_from_query(target, data=data)[0] + # TAI - UTC = 37 s since 2017 + expected = Time(61000.0, format="mjd", scale="tai").utc.to_datetime(TimezoneInfo()) + self.assertEqual(datum.timestamp, expected) + self.assertEqual(datum.timestamp.second, 23) + + def test_lsst_non_positive_flux_detections_skipped(self): + target = Target.objects.create(name="LSST12345", type="SIDEREAL", ra=10.0, dec=-5.0) + data = { + "detections": [ + {"mjd": 61000.0, "psfFlux": -50.0, "band": 1}, + {"mjd": 61001.0, "psfFlux": 0.0, "band": 1}, + {"mjd": 61002.0, "psfFlux": 100.0, "band": 1}, + ], + } + reduced_datums = self.ds.create_reduced_datums_from_query(target, data=data) + self.assertEqual(len(reduced_datums), 1) + self.assertEqual(reduced_datums[0].bandpass, "g") + + +# alerce_tap.lsst_mpc_orbits rows as returned by TAP (subset of columns). 2020 TE16 has +# 0.0 placeholders for a/mean_anomaly/mean_motion; 2000 SK234 has them populated. +MPC_ORBIT_2020_TE16 = { + "ssobjectid": np.int64(21165806405629509), "designation": "2020 TE16", + "a": np.float64(0.0), "q": np.float64(1.92531105668165), "e": np.float64(0.242376936145473), + "i": np.float64(6.3219861793433), "node": np.float64(234.8308671640585), + "argperi": np.float64(130.788251505259), "peri_time": np.float64(60558.1804330095), + "mean_anomaly": np.float64(0.0), "mean_motion": np.float64(0.0), "epoch_mjd": np.float64(60600.0), + "h": np.float64(19.474), "g": np.float64(0.15), +} +MPC_ORBIT_2000_SK234 = { + "ssobjectid": np.int64(21163607367496779), "designation": "2000 SK234", + "a": np.float64(2.7672962522372817), "q": np.float64(2.09903169657479), "e": np.float64(0.241486452750485), + "i": np.float64(8.314624536688), "node": np.float64(67.0818177412172), + "argperi": np.float64(277.5160290084402), "peri_time": np.float64(60182.875107613), + "mean_anomaly": np.float64(174.9480202569077), "mean_motion": np.float64(0.21410193458476848), + "epoch_mjd": np.float64(61000.0), "h": np.float64(16.297), "g": np.float64(0.15), +} + + +class TestSSObjectTargetCreation(TestCase): + def setUp(self): + self.ds = AlerceDataService() + tap_patcher = patch("tom_dataservices.data_services.alerce.tap_service") + self.mock_tap_service = tap_patcher.start() + self.addCleanup(tap_patcher.stop) + + def _create(self, oid, orbit_rows): + self.mock_tap_service.search.return_value = orbit_rows + return self.ds.create_target_from_query( + {"oid": oid, "sid": 2, "meanra": 151.18, "meandec": 1.91, "survey": "lsst"} + ) + + def test_ssobject_becomes_non_sidereal_minor_planet(self): + target = self._create(21165806405629509, [MPC_ORBIT_2020_TE16]) + adql = self.mock_tap_service.search.call_args.args[0] + self.assertIn("alerce_tap.lsst_mpc_orbits", adql) + self.assertIn("ssObjectId = 21165806405629509", adql) + self.assertEqual(target.name, "21165806405629509") + self.assertEqual(target.type, Target.NON_SIDEREAL) + self.assertEqual(target.scheme, "MPC_MINOR_PLANET") + self.assertIsNone(target.ra) + self.assertEqual(target.epoch_of_elements, 60600.0) + self.assertEqual(target.inclination, 6.3219861793433) + self.assertEqual(target.lng_asc_node, 234.8308671640585) + self.assertEqual(target.arg_of_perihelion, 130.788251505259) + self.assertEqual(target.eccentricity, 0.242376936145473) + self.assertEqual(target.perihdist, 1.92531105668165) + self.assertEqual(target.epoch_of_perihelion, 60558.1804330095) + self.assertEqual(target.abs_mag, 19.474) + self.assertEqual(target.slope, 0.15) + self.assertIsInstance(target.eccentricity, float) + # Derived rather than taken from the 0.0 placeholders + self.assertAlmostEqual(target.semimajor_axis, 1.92531105668165 / (1 - 0.242376936145473)) + self.assertGreater(target.mean_anomaly, 0.0) + + def test_derived_elements_match_alerce_values_when_populated(self): + target = self._create(21163607367496779, [MPC_ORBIT_2000_SK234]) + self.assertAlmostEqual(target.semimajor_axis, MPC_ORBIT_2000_SK234["a"], places=6) + self.assertAlmostEqual(target.mean_daily_motion, MPC_ORBIT_2000_SK234["mean_motion"], places=6) + self.assertAlmostEqual(target.mean_anomaly, MPC_ORBIT_2000_SK234["mean_anomaly"], places=3) + + def test_unbound_orbit_uses_comet_scheme(self): + orbit = dict(MPC_ORBIT_2020_TE16, e=np.float64(1.05)) + target = self._create(21165806405629509, [orbit]) + self.assertEqual(target.scheme, "MPC_COMET") + self.assertEqual(target.perihdist, 1.92531105668165) + self.assertEqual(target.epoch_of_perihelion, 60558.1804330095) + self.assertIsNone(target.semimajor_axis) + self.assertIsNone(target.mean_anomaly) + + def test_ssobject_without_orbit_falls_back_to_sidereal(self): + target = self._create(21165806405629509, []) + self.assertEqual(target.type, "SIDEREAL") + self.assertEqual(target.ra, 151.18) + self.assertEqual(target.dec, 1.91) + + def test_ssobject_orbit_tap_error_falls_back_to_sidereal(self): + self.mock_tap_service.search.side_effect = pyvo.dal.DALServiceError("TAP down") + with self.assertLogs("tom_dataservices.data_services.alerce", level="ERROR"): + target = self.ds.create_target_from_query( + {"oid": 21165806405629509, "sid": 2, "meanra": 151.18, "meandec": 1.91} + ) + self.assertEqual(target.type, "SIDEREAL") + self.assertEqual(target.ra, 151.18) + + def test_diaobject_does_not_query_orbits(self): + target = self.ds.create_target_from_query({"oid": 313853496686280764, "sid": 1, "meanra": 9.35, + "meandec": -42.46}) + self.assertEqual(target.type, "SIDEREAL") + self.mock_tap_service.search.assert_not_called() + + def test_ssobject_target_saves(self): + target = self._create(21165806405629509, [MPC_ORBIT_2020_TE16]) + target.save() + self.assertEqual(Target.objects.get(pk=target.pk).scheme, "MPC_MINOR_PLANET") + + def test_to_target_adds_designation_alias(self): + self.mock_tap_service.search.return_value = [MPC_ORBIT_2020_TE16] + target = self.ds.to_target({"oid": 21165806405629509, "sid": 2, "meanra": 151.18, "meandec": 1.91}) + self.assertEqual(target.name, "21165806405629509") + self.assertEqual(list(target.aliases.values_list("name", flat=True)), ["2020 TE16"]) + + def test_lsst_targets_ingest_photometry_right_after_creation(self): + """ + Mirrors CreateTargetFromQueryView: the instance returned by to_target() goes straight + to query_reduced_data(). TAP returns integer oids; an int Target.name made + build_query_parameters_from_target raise AttributeError (a 500 in the view). + """ + lightcurve = {"detections": [{"mjd": 61000.0, "psfFlux": 100.0, "band": 1}], "non_detections": []} + results = [ + ({"oid": 313853496686280764, "sid": 1, "meanra": 9.35, "meandec": -42.46}, []), + ({"oid": 21165806405629509, "sid": 2, "meanra": 151.18, "meandec": 1.91}, [MPC_ORBIT_2020_TE16]), + ] + with patch("tom_dataservices.data_services.alerce.alerce") as mock_alerce: + mock_alerce.query_lightcurve.return_value = lightcurve + for result, orbit_rows in results: + with self.subTest(sid=result["sid"]): + self.mock_tap_service.search.return_value = orbit_rows + target = self.ds.to_target(result) + self.assertEqual(target.name, str(result["oid"])) + datums = self.ds.to_reduced_datums(target, self.ds.query_reduced_data(target)) + self.assertEqual(len(datums), 1) + self.assertEqual(mock_alerce.query_lightcurve.call_args.kwargs["oid"], str(result["oid"])) + self.assertEqual(mock_alerce.query_lightcurve.call_args.kwargs["survey"], "lsst") + + def test_query_aliases_returns_designation_for_ssobject_target(self): + self.mock_tap_service.search.return_value = [MPC_ORBIT_2020_TE16] + target = Target(name="21165806405629509", type=Target.NON_SIDEREAL) + self.assertEqual(self.ds.query_aliases(target=target), ["2020 TE16"]) + self.assertIn("ssObjectId = 21165806405629509", self.mock_tap_service.search.call_args.args[0]) + + def test_query_aliases_skips_other_targets_without_querying(self): + for target in (None, + Target(name="313853496686280764", type="SIDEREAL"), + Target(name="C/2020 F3", type=Target.NON_SIDEREAL), + Target(name="6478", type=Target.NON_SIDEREAL)): # numbered asteroid, not an ssObjectId + self.assertEqual(self.ds.query_aliases(target=target), []) + self.mock_tap_service.search.assert_not_called() + + def test_query_aliases_include_mpc_number_name_and_secondary_designations(self): + designation_data = { + "name": "Gault", "permid": "6478", "iau_name": "", "orbfit_name": "6478", + "unpacked_primary_provisional_designation": "1988 JC1", + "unpacked_secondary_provisional_designations": ["1995 KC1"], + } + target = Target(name="20890962690584899", type=Target.NON_SIDEREAL) + # TAP may return the jsonb column as a JSON string or already decoded + for mpc_orb_jsonb in (json.dumps({"designation_data": designation_data}), + {"designation_data": designation_data}): + with self.subTest(jsonb_type=type(mpc_orb_jsonb).__name__): + self.mock_tap_service.search.return_value = [ + {"ssobjectid": np.int64(20890962690584899), "designation": "1988 JC1", + "mpc_orb_jsonb": mpc_orb_jsonb} + ] + self.assertEqual(self.ds.query_aliases(target=target), ["1988 JC1", "6478", "Gault", "1995 KC1"]) + + def test_query_aliases_unnamed_or_unparseable_orbit_json_gives_designation_only(self): + target = Target(name="21165806405629509", type=Target.NON_SIDEREAL) + unnamed = {"designation_data": {"name": "", "permid": "", "unpacked_secondary_provisional_designations": []}} + for mpc_orb_jsonb in (json.dumps(unnamed), "{not json", None): + with self.subTest(mpc_orb_jsonb=mpc_orb_jsonb): + self.mock_tap_service.search.return_value = [dict(MPC_ORBIT_2020_TE16, mpc_orb_jsonb=mpc_orb_jsonb)] + self.assertEqual(self.ds.query_aliases(target=target), ["2020 TE16"]) + + def test_query_aliases_empty_when_no_orbit_or_tap_error(self): + target = Target(name="21165806405629509", type=Target.NON_SIDEREAL) + self.mock_tap_service.search.return_value = [] + self.assertEqual(self.ds.query_aliases(target=target), []) + self.mock_tap_service.search.side_effect = pyvo.dal.DALServiceError("TAP down") + with self.assertLogs("tom_dataservices.data_services.alerce", level="ERROR"): + self.assertEqual(self.ds.query_aliases(target=target), []) + + +class TestBuildQueryParametersFromTarget(TestCase): + def setUp(self): + self.ds = AlerceDataService() + tap_patcher = patch("tom_dataservices.data_services.alerce.tap_service") + self.mock_tap_service = tap_patcher.start() + self.addCleanup(tap_patcher.stop) + tns_patcher = patch("tom_dataservices.data_services.alerce.TNSDataService") + self.mock_tns = tns_patcher.start().return_value + self.addCleanup(tns_patcher.stop) + + def _target(self, name, aliases=(), **kwargs): + target = Target.objects.create(name=name, type=kwargs.pop("type", "SIDEREAL"), ra=1.0, dec=2.0, **kwargs) + for alias in aliases: + TargetName.objects.create(target=target, name=alias) + return target + + def test_alerce_id_names_used_directly_without_remote_lookups(self): + for name, survey in (("ZTF24aaiafkl", "ZTF"), ("313853496686280764", "LSST")): + with self.subTest(name=name): + params = self.ds.build_query_parameters_from_target(self._target(name)) + self.assertEqual(params, {"object_id": name, "survey": survey}) + self.mock_tap_service.search.assert_not_called() + self.mock_tns.query_service.assert_not_called() + + def test_survey_id_alias_preferred_over_remote_lookup_of_name(self): + target = self._target("AT2026ziu", aliases=["ZTF26abpyrfi"]) + params = self.ds.build_query_parameters_from_target(target) + self.assertEqual(params, {"object_id": "ZTF26abpyrfi", "survey": "ZTF"}) + self.mock_tns.query_service.assert_not_called() + + def test_tns_name_resolved_through_tns_internal_names(self): + self.mock_tns.query_service.return_value = { + "objname": "2026ziu", "name_prefix": "AT", "internal_names": "ATLAS26xyz, ZTF26abpyrfi", + } + params = self.ds.build_query_parameters_from_target(self._target("AT2026ziu")) + self.assertEqual(params, {"object_id": "ZTF26abpyrfi", "survey": "ZTF"}) + self.assertEqual(self.mock_tns.build_query_parameters.call_args.args[0], {"objname": "2026ziu"}) + self.mock_tap_service.search.assert_not_called() + + def test_tns_unconfigured_or_unknown_falls_back_to_name(self): + target = self._target("SN2026sqf") + for side_effect in (NotConfiguredError("no TNS"), requests.HTTPError("503"), + [{"objname": {"110": {"message": "No results found."}}}]): + with self.subTest(side_effect=side_effect): + self.mock_tns.query_service.side_effect = side_effect + params = self.ds.build_query_parameters_from_target(target) + self.assertEqual(params, {"object_id": "SN2026sqf", "survey": "LSST"}) + + def test_numbered_asteroid_resolved_through_provisional_designation_alias(self): + self.mock_tap_service.search.return_value = [{"ssobjectid": np.int64(20890962690584899)}] + target = self._target("6478", aliases=["Gault", "1988 JC1", "1995 KC1"], type=Target.NON_SIDEREAL) + params = self.ds.build_query_parameters_from_target(target) + self.assertEqual(params, {"object_id": "20890962690584899", "survey": "LSST"}) + # "6478" and "Gault" aren't looked up; the first designation that resolves is used + self.mock_tap_service.search.assert_called_once() + self.assertIn("designation = '1988 JC1'", self.mock_tap_service.search.call_args.args[0]) + self.mock_tns.query_service.assert_not_called() + + def test_unresolved_designation_tries_next_alias(self): + self.mock_tap_service.search.side_effect = [[], [{"ssobjectid": np.int64(123456789012345678)}]] + target = self._target("6478", aliases=["1988 JC1", "1995 KC1"], type=Target.NON_SIDEREAL) + params = self.ds.build_query_parameters_from_target(target) + self.assertEqual(params, {"object_id": "123456789012345678", "survey": "LSST"}) + + def test_tap_error_during_designation_lookup_falls_back_to_name(self): + self.mock_tap_service.search.side_effect = pyvo.dal.DALServiceError("TAP down") + target = self._target("6478", aliases=["1988 JC1"], type=Target.NON_SIDEREAL) + with self.assertLogs("tom_dataservices.data_services.alerce", level="ERROR"): + params = self.ds.build_query_parameters_from_target(target) + self.assertEqual(params, {"object_id": "6478", "survey": "LSST"}) diff --git a/tom_dataservices/tests/data_services/test_mpc.py b/tom_dataservices/tests/data_services/test_mpc.py index 03485bc94..e8d482f27 100644 --- a/tom_dataservices/tests/data_services/test_mpc.py +++ b/tom_dataservices/tests/data_services/test_mpc.py @@ -155,8 +155,8 @@ def setUp(self): def test_query_neo(self): query_parameters = {'desig': '433'} - self.ds.query_service(query_parameters) - self.ds.query_targets(query_parameters) + targets = self.ds.query_targets(query_parameters) + self.target_result = targets[0] target = self.ds.to_target(self.target_result) # Only test things that are not likely to change (much) with time self.assertEqual(target.name, '433') @@ -173,8 +173,8 @@ def test_query_neo(self): def test_query_comet(self): query_parameters = {'desig': 'C/1995 O1'} - self.ds.query_service(query_parameters) - self.ds.query_targets(query_parameters) + targets = self.ds.query_targets(query_parameters) + self.target_result = targets[0] target = self.ds.to_target(self.target_result) target.save(names=getattr(target, 'extra_names', [])) # Only test things that are not likely to change (much) with time