Repository navigation
Conversation
…ve ones, rename n_det->ndet to match alerce_client
Covers AlerceForm classifier field generation/caching, build_query_parameters, query_service (ZTF REST + LSST TAP oid paths, error handling), and target/datum creation. Two @expectedfailure cases pin known bugs (LSST general query AttributeError, LSST detection KeyError) that later steps fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
query_objects(survey='lsst') on the REST client returns a bare list, which
broke the ZTF-shaped .get("items", []) unwrap. LSST (sid != 0) general queries
now build ADQL against alerce_tap.object via a new _build_tap_object_query
helper (cone search, mjd ranges, ndet range, TOP/sid); LSST classifier queries
raise QueryServiceError until TAP classifier/probability support lands. Also
normalizes ZTF REST's deltajd to deltamjd, matching TAP/firstmjd/lastmjd
naming, and factors n_det->ndet renaming into a shared _normalize_tap_record
helper reused by the oid and general paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Queries alerce_tap.classifier JOIN alerce_tap.taxonomy (filtered by tid,
survey-aware via SURVEY_TID) and groups rows into the shape
add_classifiers_fields() already expects, via the new
_group_tap_classifier_rows helper. Cache key is now per-survey
(ds_alerce_classifiers_{tid}) instead of a single shared key, so ZTF and LSST
classifier lists no longer clobber each other in the 24h cache. No REST
query_classifiers() call remains in this module.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
self.target_result was never assigned in these tests, so both raised
AttributeError before ever reaching the live MPC API assertions -- pre-existing
on dev, unrelated to the ALeRCE TAP migration in this branch's other commits.
query_targets() already returns a list of {mpc_orb, name, aliases} dicts (it
calls query_service() internally), so capture that instead of the previous
redundant standalone query_service() call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AlerceForm.add_classifiers_fields() previously only built classifier fields for
whichever survey the form was constructed with -- ZTF by default on every fresh
page load. Since the Survey dropdown's ZTF/LSST toggle is purely client-side
(Alpine x-show), a user selecting LSST would never see LSST's classifier fields
appear; they'd only show up after a full form resubmit that happened to bind
survey=LSST server-side (e.g. a validation-error redisplay).
Now every survey's classifier fields are always added to the form
(cfield_{survey}__{classifier_name}), and a shared Alpine store
($store.alerce.survey, registered in alerce_simple_form.html and read by
alerce_advanced_form.html via the new cfield_survey template filter) shows only
the selected survey's group -- updating live as the dropdown changes, no
resubmit needed.
Since hidden (non-selected-survey) classifier fields are still present in the
DOM and still get POSTed, AlerceForm.clean() now drops any classifier value
whose embedded survey doesn't match the submitted survey, so a stale ZTF
selection left over from switching to LSST isn't silently bundled into the
query.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Known ssObjects are pre-assigned probability 1 "asteroid" rather than classified, so sid=2 classifier queries still raise QueryServiceError, but sid=1 (diaObject) classifier queries now resolve classifier/class names to their TAP ids and query alerce_tap.probability joined to alerce_tap.object, matching ALeRCE's own LSST SSO notebook usage (ranking=1 for top-ranked classification, probability as a minimum threshold). Verified live against the real ALeRCE TAP service. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrBHYzLY14hzhPAYwhSkAM
lastmjd can now be filtered as "last detected after X" with no upper bound (query_parameters["lastmjd"] = [gt] only), mirroring how ndet already supports a min-only bound. firstmjd is unchanged (still requires both bounds) since only lastmjd was reported as a common one-sided pattern -- e.g. ALeRCE's own LSST TAP queries notebook filters lastmjd this way when selecting recently-active objects. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrBHYzLY14hzhPAYwhSkAM
LSST detections carry difference-image flux in nJy (psfFlux/psfFluxErr), an integer band (1-6, ugrizy) and TAI mjd, rather than ZTF's magpsf/sigmapsf/fid, so create_reduced_datums_from_query raised KeyError for LSST targets. Convert them to AB magnitudes (31.4 - 2.5 log10 f), map bands via band_name or ALeRCE's band lookup table, convert TAI to UTC, and tag datums with telescope Rubin / instrument LSSTCam. Detections with non-positive flux have no magnitude and are skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
ssObjects (sid=2) were created as SIDEREAL targets at their mean position, which is wrong for moving objects. Fetch the object's alerce_tap.lsst_mpc_orbits record and build a NON_SIDEREAL target. The table's a/mean_anomaly/mean_motion can be 0.0 placeholders, so, as in MPCExplorerDataService, derive them from q, e and peri_time. Bound orbits use MPC_MINOR_PLANET, unbound ones MPC_COMET. Objects with no stored orbit fall back to SIDEREAL with a logged warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
The LSST oid lookup formatted the user-supplied Object ID straight into the ADQL WHERE clause. Coerce it to int first; a non-numeric ID now raises QueryServiceError (via the existing ValueError handling) without querying TAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
Annotate each query result with an alerce_url for the survey's Explorer (alerce.online for ZTF, lsst.alerce.online for LSST) and link the Object ID in the results table to it. The URL is also stored as a target extra when a target is created from the result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
LSST TAP queries were fixed at TOP 20 with no ORDER BY, so they returned an arbitrary 20 rows. Add Max. Results, Sort By (last/first detection, number of detections) and Sort Order fields to the advanced form. They map to page_size/order_by/order_mode for the ZTF REST client and to TOP n / ORDER BY for TAP. LSST classifier searches still sort by probability unless another order is chosen. Only whitelisted columns and ASC/DESC are interpolated into ADQL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
New moving objects arrive in LSST alerts as diaObjects, not ssObjects. ALeRCE's SSO notebook finds them with the stamp classifier's asteroid class plus a short first-to-last detection span. Add a Max. Detection Time Span field that filters alerce_tap.object.deltamjd. The ZTF REST API has no equivalent, so the form rejects it for ZTF and it is only shown when LSST is selected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
ssObject targets are named by their numeric ssObjectId, which is needed to look up their photometry but is not how anyone refers to an asteroid. Implement query_aliases to return the designation (e.g. "2020 TE16") from alerce_tap.lsst_mpc_orbits for non-sidereal targets with numeric names, so it is added as a TargetName on creation and on data update. Other targets are skipped without querying TAP, and a TAP failure is logged rather than failing target creation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
Since 6539f4e, ZTF ID lookups call query_objects(), which returns a paginated wrapper ({"total", "items", ...}), and the wrapper was used as the object, giving a result with no oid. Unwrap "items" as the general query path does. An LSST ID with no match raised IndexError; it now returns no results, as a ZTF miss does. ID lookups also returned before results were annotated, so they had no survey or Explorer link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
Accept an asteroid designation (e.g. "2010 WX64") in the Object ID field for LSST searches. Non-numeric IDs are resolved to an ssObjectId through alerce_tap.lsst_mpc_orbits and looked up as an ssObject whichever object type is selected. An unknown designation gives no results. Only designation characters are accepted, so quotes cannot reach the ADQL. designation is not indexed in that table, but a lookup takes about 1 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpjq1sabk4eExET2eAKjs5
TAP errors (pyvo DALAccessError, which pyvo also raises for connection failures and HTTP errors) weren't caught, so an ALeRCE TAP outage returned a 500 on three pages: - LSST searches: query_service only caught ZTF REST client errors. TAP errors now become QueryServiceError, which the search view shows as a message, as it does for ZTF failures. - Creating targets from ssObjects: a failed orbit lookup now falls back to a SIDEREAL target, as when no orbit is stored. - The query form: a failed classifier lookup now renders the form without classifier fields. The failure isn't cached, so the next load retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
The LSST classifier TAP query filters alerce_tap.probability by numeric class_id but never selects the class name, so the results table's "Class [Probability]" column showed only " [0.95]". Add the searched class and classifier names to each result, matching the class and classifier fields ZTF REST results already carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
The ALeRCE DataService imports pyvo for its TAP queries, but pyvo was only installed as a dependency of astroquery. Declare it directly. The locked version (1.9.1) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
TAP returns LSST oids as integers, and create_target_from_query put them straight into Target.name. The database stores a string, but the in-memory instance kept the int, and CreateTargetFromQueryView passes that instance straight to query_reduced_data(). There, build_query_parameters_from_target calls target.name.startswith(), which raised AttributeError, so creating any LSST target (diaObject or ssObject) returned a 500 and saved the target without photometry. Build the name as a string. ZTF names were already strings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
"Update data" from ALeRCE only used the target's name as the object ID, so it found nothing for targets created elsewhere: a TNS-named SN/AT target carrying its ZTF ID as an alias, or an MPC Explorer asteroid target (e.g. "6478" with alias "1988 JC1"). Resolve the ALeRCE object from the target's name and aliases instead: - ZTF IDs and LSST IDs (15+ digits, so numbered asteroids aren't mistaken for them) are used directly, and are tried first so no remote lookup happens when one exists. - AT/SN names are resolved through the ZTF/LSST IDs among their TNS internal names, if TNS is configured. - MPC provisional, survey and comet designations are resolved to an ssObjectId through alerce_tap.lsst_mpc_orbits. Lookup failures are logged and the next name is tried. If nothing resolves, the target's name is used as before. The survey is now inferred from the ID rather than from classifier/survey target extras. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
LSST ssObject targets only got their provisional designation as an alias, so they couldn't be found by their number or name. Also read the permanent number, name and secondary provisional designations from the MPC orbit record (mpc_orb_jsonb designation_data) that ALeRCE stores, e.g. "1988 JC1", "6478", "Gault" and "1995 KC1" for ssObject 20890962690584899. Only query for targets named by an LSST ID (15+ digits), so a numbered-asteroid target such as "6478" no longer triggers a pointless TAP lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
create_reduced_datums_from_query looked up existing datums on brightness_error and unit as well as the fields of the PhotometryReducedDatum uniqueness constraints. A re-queried detection with a revised uncertainty then missed the stored row, the create failed full_clean() with a ValidationError, and that escaped the data service, because only IntegrityError was caught. Non-detections had no error handling at all. Look up on the constraint fields only and set brightness_error, telescope, unit and source_name through defaults, so a re-query leaves stored datums unchanged. Report either error as a QueryServiceError for detections and non-detections alike. Fixes TOMToolkit#1669 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
Add a data services page covering why LSST searches use ALeRCE's TAP service, diaObject and ssObject searches, target creation and asteroid aliases, LSST flux-to-magnitude conversion, and how targets from other sources are matched to ALeRCE objects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdFmrCcTbUcGsdWgQVCCBJ
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the ALeRCE DataService in
tom_dataservicesso it can search, create targets from, and ingest photometry for Rubin/LSST objects in addition to the existing ZTF support. LSST searches (and the classifier lists for both surveys) go through ALeRCE's TAP service viapyvo, because the REST API can't yet filter LSST objects by classifier, list LSST classifiers, or provide the MPC orbits needed for asteroids (see this response from Alejandra of ALeRCE in Rubin Community) ZTF searches and light curves for both surveys still use the REST API throughalerce_client.Implements #1665 and also fixes #1669
Changes
LSST searches
diaObject/ssObject) fields. LSST general queries are built as ADQL againstalerce_tap.object(cone search, first/last detection MJD, ndet range).diaObjectclassifier searches joinalerce_tap.probability(top-ranked class, minimum probability), and results show the class and probability.ssObjectclassifier searches raise a clear error, since ALeRCE assigns every known ssObject "asteroid" with probability 1.2010 WX64), which is resolved to an ssObjectId throughalerce_tap.lsst_mpc_orbits.lastmjdcan now be given as a lower bound only.clean()drops classifier values from the non-selected survey.ds_alerce_classifiers_{tid}).Targets and photometry
ssObjecttargets are created as non-sidereal targets from ALeRCE's copy of the MPC orbit. Missing orbital elements are derived from q, e and peri time, as inMPCExplorerDataService. If there is no orbit, or the lookup fails, the target falls back to sidereal with a logged warning.lsst_mpc_orbits. Targets created by TNS or MPC Explorer can therefore pick up ALeRCE photometry.alerce.online/lsst.alerce.online).Bug fixes
IndexError; it now returns no results.Target.namewas left as an int in memory.ValidationError(Alerce DataService can crash when re-querying photometry for a target #1669). Existing datums are now matched on the uniqueness-constraint fields only and left unchanged.QueryServiceErrormessages or graceful fallbacks.TestMPCExplorerDataServiceCanarynever assignedself.target_result, so it failed before reaching the live API. This bug predates the branch and is also ondev.Dependencies and docs
pyvo >=1.5,<2is declared as a direct dependency. It was previously only pulled in throughastroquery; the locked version (1.9.1) is unchanged.docs/data_services/alerce.rstcovering ZTF vs LSST querying, diaObject/ssObject searches, asteroid aliases, flux-to-magnitude conversion, and alias-based matching.Testing
tom_dataservices/tests/data_services/test_alerce.pycover form building, query parameters, the REST and TAP paths, error handling, and target/datum creation. The ALeRCE and MPC suites pass locally.2010 WX64found ssObject21164706944660568, and creating a target from it gave:NON_SIDEREAL/MPC_MINOR_PLANETtarget with its orbit fromlsst_mpc_orbits;2010 WX64and728999;ndetof 44);Some code and some of this PR text generated with Claude robots, end-to-end user testing done with humans