Skip to content

Enhance the ALeRCE broker to add LSST and solar system support via TAP - #1686

Open
talister wants to merge 26 commits into
TOMToolkit:devfrom
talister:enhancement/alerce-add-tap-and-lsst
Open

talister wants to merge 26 commits into
TOMToolkit:devfrom
talister:enhancement/alerce-add-tap-and-lsst

Conversation

@talister

Copy link
Copy Markdown
Collaborator

Summary

Extends the ALeRCE DataService in tom_dataservices so it can search, create targets from, and ingest photometry for Rubin/LSST objects in addition to the existing ZTF support. LSST searches (and the classifier lists for both surveys) go through ALeRCE's TAP service via pyvo, because the REST API can't yet filter LSST objects by classifier, list LSST classifiers, or provide the MPC orbits needed for asteroids (see this response from Alejandra of ALeRCE in Rubin Community) ZTF searches and light curves for both surveys still use the REST API through alerce_client.

Implements #1665 and also fixes #1669

Changes

LSST searches

  • New Survey (ZTF/LSST) and LSST Object Type (diaObject / ssObject) fields. LSST general queries are built as ADQL against alerce_tap.object (cone search, first/last detection MJD, ndet range).
  • diaObject classifier searches join alerce_tap.probability (top-ranked class, minimum probability), and results show the class and probability. ssObject classifier searches raise a clear error, since ALeRCE assigns every known ssObject "asteroid" with probability 1.
  • Object ID accepts a diaObjectId, an ssObjectId, or an MPC designation (e.g. 2010 WX64), which is resolved to an ssObjectId through alerce_tap.lsst_mpc_orbits.
  • New advanced-form fields: Max. Results, Sort By, Sort Order (both surveys), and Max. Detection Time Span (LSST only, for finding new asteroid candidates among diaObjects). lastmjd can now be given as a lower bound only.
  • Classifier fields for both surveys are always on the form, and a shared Alpine store shows the selected survey's set, so switching surveys no longer needs a resubmit. clean() drops classifier values from the non-selected survey.
  • Classifier lists come from TAP and are cached per survey (ds_alerce_classifiers_{tid}).

Targets and photometry

  • LSST photometry is ingested: difference-image flux in nJy is converted to AB magnitudes, bands 1–6 are mapped to ugrizy, and TAI is converted to UTC. Data is tagged Rubin/LSSTCam. Detections with non-positive flux are skipped.
  • ssObject targets are created as non-sidereal targets from ALeRCE's copy of the MPC orbit. Missing orbital elements are derived from q, e and peri time, as in MPCExplorerDataService. If there is no orbit, or the lookup fails, the target falls back to sidereal with a logged warning.
  • Asteroid targets get their MPC provisional designation, permanent number, name and secondary designations as aliases.
  • Update data now resolves the ALeRCE object from the target's name and aliases: ZTF/LSST IDs directly, AT/SN names through TNS internal names (if TNS is configured), and MPC designations through lsst_mpc_orbits. Targets created by TNS or MPC Explorer can therefore pick up ALeRCE photometry.
  • Results and created targets link to the ALeRCE Explorer (alerce.online / lsst.alerce.online).

Bug fixes

  • ZTF object ID lookups returned the paginated wrapper instead of the object (regression from 6539f4e).
  • An LSST ID with no match raised IndexError; it now returns no results.
  • Creating any LSST target returned a 500 and saved it without photometry, because TAP returns integer oids and Target.name was left as an int in memory.
  • Re-querying photometry with revised uncertainties raised an uncaught ValidationError (Alerce DataService can crash when re-querying photometry for a target #1669). Existing datums are now matched on the uniqueness-constraint fields only and left unchanged.
  • TAP outages no longer 500 the search, target-creation or query-form pages; they surface as QueryServiceError messages or graceful fallbacks.
  • TestMPCExplorerDataServiceCanary never assigned self.target_result, so it failed before reaching the live API. This bug predates the branch and is also on dev.

Dependencies and docs

  • pyvo >=1.5,<2 is declared as a direct dependency. It was previously only pulled in through astroquery; the locked version (1.9.1) is unchanged.
  • New docs page docs/data_services/alerce.rst covering ZTF vs LSST querying, diaObject/ssObject searches, asteroid aliases, flux-to-magnitude conversion, and alias-based matching.

Testing

  • 101 tests in tom_dataservices/tests/data_services/test_alerce.py cover form building, query parameters, the REST and TAP paths, error handling, and target/datum creation. The ALeRCE and MPC suites pass locally.
  • LSST classifier and ssObject paths were checked against the live ALeRCE TAP service.
  • End to end in a downstream TOM (FOMO, tomtoolkit 3.1): an LSST search for 2010 WX64 found ssObject 21164706944660568, and creating a target from it gave:
    • a NON_SIDEREAL / MPC_MINOR_PLANET target with its orbit from lsst_mpc_orbits;
    • aliases 2010 WX64 and 728999;
    • 44 LSSTCam detections in g, r, i and z as AB magnitudes (matching ALeRCE's ndet of 44);
    • an ALeRCE Explorer link.
    • ZTF microlensing example from Markus (ZTF24aaiafkl) and ZTF SNe example (ZTF24aamlalc) from ALeRCE notebooks also tested out fine

Some code and some of this PR text generated with Claude robots, end-to-end user testing done with humans

talister and others added 26 commits September 25, 2026 10:42
…ve ones, rename n_det->ndet to match alerce_client
Covers AlerceForm classifier field generation/caching, build_query_parameters,
query_service (ZTF REST + LSST TAP oid paths, error handling), and target/datum
creation. Two @expectedfailure cases pin known bugs (LSST general query
AttributeError, LSST detection KeyError) that later steps fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
query_objects(survey='lsst') on the REST client returns a bare list, which
broke the ZTF-shaped .get("items", []) unwrap. LSST (sid != 0) general queries
now build ADQL against alerce_tap.object via a new _build_tap_object_query
helper (cone search, mjd ranges, ndet range, TOP/sid); LSST classifier queries
raise QueryServiceError until TAP classifier/probability support lands. Also
normalizes ZTF REST's deltajd to deltamjd, matching TAP/firstmjd/lastmjd
naming, and factors n_det->ndet renaming into a shared _normalize_tap_record
helper reused by the oid and general paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Queries alerce_tap.classifier JOIN alerce_tap.taxonomy (filtered by tid,
survey-aware via SURVEY_TID) and groups rows into the shape
add_classifiers_fields() already expects, via the new
_group_tap_classifier_rows helper. Cache key is now per-survey
(ds_alerce_classifiers_{tid}) instead of a single shared key, so ZTF and LSST
classifier lists no longer clobber each other in the 24h cache. No REST
query_classifiers() call remains in this module.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
self.target_result was never assigned in these tests, so both raised
AttributeError before ever reaching the live MPC API assertions -- pre-existing
on dev, unrelated to the ALeRCE TAP migration in this branch's other commits.
query_targets() already returns a list of {mpc_orb, name, aliases} dicts (it
calls query_service() internally), so capture that instead of the previous
redundant standalone query_service() call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AlerceForm.add_classifiers_fields() previously only built classifier fields for
whichever survey the form was constructed with -- ZTF by default on every fresh
page load. Since the Survey dropdown's ZTF/LSST toggle is purely client-side
(Alpine x-show), a user selecting LSST would never see LSST's classifier fields
appear; they'd only show up after a full form resubmit that happened to bind
survey=LSST server-side (e.g. a validation-error redisplay).

Now every survey's classifier fields are always added to the form
(cfield_{survey}__{classifier_name}), and a shared Alpine store
($store.alerce.survey, registered in alerce_simple_form.html and read by
alerce_advanced_form.html via the new cfield_survey template filter) shows only
the selected survey's group -- updating live as the dropdown changes, no
resubmit needed.

Since hidden (non-selected-survey) classifier fields are still present in the
DOM and still get POSTed, AlerceForm.clean() now drops any classifier value
whose embedded survey doesn't match the submitted survey, so a stale ZTF
selection left over from switching to LSST isn't silently bundled into the
query.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Known ssObjects are pre-assigned probability 1 "asteroid" rather than
classified, so sid=2 classifier queries still raise QueryServiceError,
but sid=1 (diaObject) classifier queries now resolve classifier/class
names to their TAP ids and query alerce_tap.probability joined to
alerce_tap.object, matching ALeRCE's own LSST SSO notebook usage
(ranking=1 for top-ranked classification, probability as a minimum
threshold). Verified live against the real ALeRCE TAP service.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrBHYzLY14hzhPAYwhSkAM
lastmjd can now be filtered as "last detected after X" with no upper
bound (query_parameters["lastmjd"] = [gt] only), mirroring how ndet
already supports a min-only bound. firstmjd is unchanged (still
requires both bounds) since only lastmjd was reported as a common
one-sided pattern -- e.g. ALeRCE's own LSST TAP queries notebook
filters lastmjd this way when selecting recently-active objects.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrBHYzLY14hzhPAYwhSkAM
LSST detections carry difference-image flux in nJy (psfFlux/psfFluxErr),
an integer band (1-6, ugrizy) and TAI mjd, rather than ZTF's
magpsf/sigmapsf/fid, so create_reduced_datums_from_query raised KeyError
for LSST targets. Convert them to AB magnitudes (31.4 - 2.5 log10 f),
map bands via band_name or ALeRCE's band lookup table, convert TAI to
UTC, and tag datums with telescope Rubin / instrument LSSTCam.
Detections with non-positive flux have no magnitude and are skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
ssObjects (sid=2) were created as SIDEREAL targets at their mean
position, which is wrong for moving objects. Fetch the object's
alerce_tap.lsst_mpc_orbits record and build a NON_SIDEREAL target.
The table's a/mean_anomaly/mean_motion can be 0.0 placeholders, so,
as in MPCExplorerDataService, derive them from q, e and peri_time.
Bound orbits use MPC_MINOR_PLANET, unbound ones MPC_COMET. Objects
with no stored orbit fall back to SIDEREAL with a logged warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
The LSST oid lookup formatted the user-supplied Object ID straight into
the ADQL WHERE clause. Coerce it to int first; a non-numeric ID now
raises QueryServiceError (via the existing ValueError handling) without
querying TAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
Annotate each query result with an alerce_url for the survey's Explorer
(alerce.online for ZTF, lsst.alerce.online for LSST) and link the
Object ID in the results table to it. The URL is also stored as a
target extra when a target is created from the result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
LSST TAP queries were fixed at TOP 20 with no ORDER BY, so they
returned an arbitrary 20 rows. Add Max. Results, Sort By (last/first
detection, number of detections) and Sort Order fields to the advanced
form. They map to page_size/order_by/order_mode for the ZTF REST
client and to TOP n / ORDER BY for TAP. LSST classifier searches still
sort by probability unless another order is chosen. Only whitelisted
columns and ASC/DESC are interpolated into ADQL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
New moving objects arrive in LSST alerts as diaObjects, not ssObjects.
ALeRCE's SSO notebook finds them with the stamp classifier's asteroid
class plus a short first-to-last detection span. Add a Max. Detection
Time Span field that filters alerce_tap.object.deltamjd. The ZTF REST
API has no equivalent, so the form rejects it for ZTF and it is only
shown when LSST is selected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
ssObject targets are named by their numeric ssObjectId, which is
needed to look up their photometry but is not how anyone refers to an
asteroid. Implement query_aliases to return the designation (e.g.
"2020 TE16") from alerce_tap.lsst_mpc_orbits for non-sidereal targets
with numeric names, so it is added as a TargetName on creation and on
data update. Other targets are skipped without querying TAP, and a TAP
failure is logged rather than failing target creation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
Since 6539f4e, ZTF ID lookups call query_objects(), which returns a
paginated wrapper ({"total", "items", ...}), and the wrapper was used as
the object, giving a result with no oid. Unwrap "items" as the general
query path does. An LSST ID with no match raised IndexError; it now
returns no results, as a ZTF miss does. ID lookups also returned before
results were annotated, so they had no survey or Explorer link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
Accept an asteroid designation (e.g. "2010 WX64") in the Object ID field
for LSST searches. Non-numeric IDs are resolved to an ssObjectId through
alerce_tap.lsst_mpc_orbits and looked up as an ssObject whichever object
type is selected. An unknown designation gives no results. Only
designation characters are accepted, so quotes cannot reach the ADQL.
designation is not indexed in that table, but a lookup takes about 1 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
TAP errors (pyvo DALAccessError, which pyvo also raises for connection
failures and HTTP errors) weren't caught, so an ALeRCE TAP outage
returned a 500 on three pages:

- LSST searches: query_service only caught ZTF REST client errors. TAP
  errors now become QueryServiceError, which the search view shows as
  a message, as it does for ZTF failures.
- Creating targets from ssObjects: a failed orbit lookup now falls back
  to a SIDEREAL target, as when no orbit is stored.
- The query form: a failed classifier lookup now renders the form
  without classifier fields. The failure isn't cached, so the next load
  retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
The LSST classifier TAP query filters alerce_tap.probability by numeric
class_id but never selects the class name, so the results table's
"Class [Probability]" column showed only " [0.95]". Add the searched
class and classifier names to each result, matching the class and
classifier fields ZTF REST results already carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
The ALeRCE DataService imports pyvo for its TAP queries, but pyvo was
only installed as a dependency of astroquery. Declare it directly. The
locked version (1.9.1) is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
TAP returns LSST oids as integers, and create_target_from_query put
them straight into Target.name. The database stores a string, but the
in-memory instance kept the int, and CreateTargetFromQueryView passes
that instance straight to query_reduced_data(). There,
build_query_parameters_from_target calls target.name.startswith(),
which raised AttributeError, so creating any LSST target (diaObject or
ssObject) returned a 500 and saved the target without photometry.
Build the name as a string. ZTF names were already strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
"Update data" from ALeRCE only used the target's name as the object ID,
so it found nothing for targets created elsewhere: a TNS-named SN/AT
target carrying its ZTF ID as an alias, or an MPC Explorer asteroid
target (e.g. "6478" with alias "1988 JC1"). Resolve the ALeRCE object
from the target's name and aliases instead:

- ZTF IDs and LSST IDs (15+ digits, so numbered asteroids aren't
  mistaken for them) are used directly, and are tried first so no
  remote lookup happens when one exists.
- AT/SN names are resolved through the ZTF/LSST IDs among their TNS
  internal names, if TNS is configured.
- MPC provisional, survey and comet designations are resolved to an
  ssObjectId through alerce_tap.lsst_mpc_orbits.

Lookup failures are logged and the next name is tried. If nothing
resolves, the target's name is used as before. The survey is now
inferred from the ID rather than from classifier/survey target extras.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
LSST ssObject targets only got their provisional designation as an
alias, so they couldn't be found by their number or name. Also read
the permanent number, name and secondary provisional designations from
the MPC orbit record (mpc_orb_jsonb designation_data) that ALeRCE
stores, e.g. "1988 JC1", "6478", "Gault" and "1995 KC1" for
ssObject 20890962690584899. Only query for targets named by an LSST ID
(15+ digits), so a numbered-asteroid target such as "6478" no longer
triggers a pointless TAP lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
create_reduced_datums_from_query looked up existing datums on
brightness_error and unit as well as the fields of the
PhotometryReducedDatum uniqueness constraints. A re-queried detection
with a revised uncertainty then missed the stored row, the create failed
full_clean() with a ValidationError, and that escaped the data service,
because only IntegrityError was caught. Non-detections had no error
handling at all.

Look up on the constraint fields only and set brightness_error,
telescope, unit and source_name through defaults, so a re-query leaves
stored datums unchanged. Report either error as a QueryServiceError for
detections and non-detections alike.

Fixes TOMToolkit#1669

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
Add a data services page covering why LSST searches use ALeRCE's TAP
service, diaObject and ssObject searches, target creation and asteroid
aliases, LSST flux-to-magnitude conversion, and how targets from other
sources are matched to ALeRCE objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
@jchate6 jchate6 moved this to Needs Review in TOM Toolkit Sep 25, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs Review

Development

Successfully merging this pull request may close these issues.

Alerce DataService can crash when re-querying photometry for a target

2 participants