Skip to content

Adding unfied planet mapmaker using for MF-SATs - #1706

Draft
yoshinori-0778 wants to merge 48 commits into
masterfrom
add_planet_mapmaker
Draft

Adding unfied planet mapmaker using for MF-SATs#1706
yoshinori-0778 wants to merge 48 commits into
masterfrom
add_planet_mapmaker

Conversation

@yoshinori-0778

Copy link
Copy Markdown
Contributor

Adding general functions in coords.planets.py and functions related to planet mapmaking in mapmaking.planet_mapmaker.py.

Now is a very very draft, but I confirmed that a planet map was made. Will arrange these more appropriately.

@yoshinori-0778
yoshinori-0778 requested a review from mhasself August 4, 2026 06:14
Comment thread sotodlib/mapmaking/planet_mapmaker.py Outdated
"""
# This is based on TOD fits summarized here: https://docs.google.com/presentation/d/1DPBeMX9ZPHwQyi9-s5orfRZIYcRLXaejFWOtlsflgUU/edit?usp=sharing
# amplitude below is in arcmin, phase in radian
amp_satp1 = {'f090': {'ws0': 0.3945,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these sorts of hard-codings I would move to a version-controlled configuration file.

@mhasself mhasself left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all this work! As requested, here is some general guidance (but not an exhaustive review yet):

  • Please include an example config file as part of documentation.
  • Format code and docs to less than 120 columns.
  • There are also some "whitespace" violations -- might be worth just running a source formatter on the new files you're contributing (but don't mass-reformat the existing files like planets.py!)
  • Consider making the deflection/wobble code a little more general, and moving into coords module somewhere.
  • You have functions for getting and archiving and processing PWV. Are those needed, if instead we can provide PWV through obsdb? I think it's good to minimize the number of different ways we pull data from APEX / Toco radiometer.
  • Use "single" instead of "each" for functions that process one of something (e.g. get_each_instrument_P). "Each" implies that it is looping, internally, over a bunch of things.
  • Remove default paths from planet_mapmaker.py funcs

Much of the new code in planets.py is basically dealing with the fact that SlowSource is not really good enough, at this point. It would be nice to take your functions calc_planet_* and instead create a replacement for SlowSource, e.g. AzElSource, that provides an az, el interpolator for any kind of source with sufficient precision for beams work. (Basically I want to get rid of SlowSource in favor of something that works better, without a huge performance penalty for simple cases.)

tags = obs['tags']
subs = 'ws'
if configs['query'].get('all_wafers', False):
obs_wafers = [f'ws{i}' for i in range(7)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps use obs['wafer_slots_list'] for this.

# add function for overwrite
for wafer in obs_wafers:
if configs['overwrite']:
irunlist = {'obs_id':obs_id, 'wafer_info': {'wafer_slot': wafer, 'wafer.bandpass': band}}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Construct this data one time (instead of duplicating 3 times).

@@ -0,0 +1,138 @@
# To make atmic planet map in detector-centered coordinate system

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a full submodule docstring """ ... """ here, and explain in more detail what the script does. It's also a good place to put in an example of a configuration file. The docstring here is easy to have rendered in readthedocs, so it can be the "one place" this script is explained.

'det_weights': det_weights,
}

def xieta0(q):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, xieta decomposition won't work (as you say in docstring), but I think this would?:

 phi, theta, gamma = so3g.proj.quat.decompose_iso(q)
 return so3g.proj.rotation_iso(phi, theta, 0.)

@mhasself

mhasself commented Sep 2, 2026

Copy link
Copy Markdown
Member

Thanks for all this work! As requested, here is some general guidance (but not an exhaustive review yet):
...

One more thing: utility functions (such as the ones you've added in coords.planets) should have unit tests. (It's not necessary to duplicate all the calculations; rather the tests just make sure there aren't typos and that different ways of asking for the same thing will yield the same answer.)

@yoshinori-0778

Copy link
Copy Markdown
Contributor Author

THanks for the comments @mhasself

Please include an example config file as part of documentation.
Could you more clarify where I should inculde? Should I add example config files in this mapmaking directory?

You have functions for getting and archiving and processing PWV. Are those needed, if instead we can provide PWV through obsdb? I think it's good to minimize the number of different ways we pull data from APEX / Toco radiometer
If there is PWV database, we do not need this. Is that available somewhere? I am happy to incorporate it.

@yoshinori-0778

Copy link
Copy Markdown
Contributor Author

Future task for me: run all combination of detector/boresignt/planet center and wafer/detector based deflection correction.

@kmharrington

Copy link
Copy Markdown
Member

You have functions for getting and archiving and processing PWV. Are those needed, if instead we can provide PWV through obsdb? I think it's good to minimize the number of different ways we pull data from APEX / Toco radiometer
If there is PWV database, we do not need this. Is that available somewhere? I am happy to incorporate it.

@mhasself A note on this. I discovered recently that the extended obsdbs running in your user account do not include calibration observations. It would be good to have those expanded and to make them official. Otherwise we'll keep having parallel implementations pop up.

Yoshinori Sueno and others added 11 commits September 7, 2026 01:03
* Make a lightweight script for bootstrapping a dev env

* Get conda dependencies from soconda

* Rename file

* Add init file for data directory to remove warnings

* Bump macos github runner to latest
* fix bug of nseg

* reduce warning of calc_psd and calc_wn

* fix

* fix
* Add on_mismatch to merge and wrap

add test

update

update docstring

* keep default behaviors

* Add intersect option
* Change fitting method

* Change fit method configurable

* Fix default option
* fix default log level

* fix in main func
Co-authored-by: Michael McCrackan <mmccrack@login32.chn.perlmutter.nersc.gov>
* add glitched tod fraction select

* update docstring

* update docstring

* updates

* fix typos

* default max_t_frac

---------

Co-authored-by: Michael McCrackan <mmccrack@login32.chn.perlmutter.nersc.gov>
* allow select:False to actually not cut data

* apply change throughout

---------

Co-authored-by: Michael McCrackan <mmccrack@login32.chn.perlmutter.nersc.gov>
* add option to save det_cal nan flags in preprocessing
---------

Co-authored-by: Michael McCrackan <mmccrack@login32.chn.perlmutter.nersc.gov>
* asymmetric turnaround buffers

* add missing axis

* smooth az

* az_buffer for scanspeed method

* update test

* address comments

* fix typos
17-sugiyama and others added 26 commits September 7, 2026 04:02
This PR is for the minor bug fix of get_psd_mask in fft_ops.
* Allow 1D binning and fix edge bins

* debug

* Add test

* reduce redundancy of 1d and 2d

* fix base_valid

* update tests

* fix docstring

* Fix bin_centers for non-uniform bins etc
* Added core profiling module and used across data-package scripts

* add profiling to update_obsdb

* Import Path

---------

Co-authored-by: Michael McCrackan <mmccrack@login14.chn.perlmutter.nersc.gov>
* Add script for export of specific observation IDs

When debugging code on a small number of observations, it can
be useful to copy that data to a local system.  This adds a script
to do that export.

* Add documentation.  Add small fixes to support LAT data and preprocessing.
* Changed local median signal calculation to global median

* Delete 1 unused import leftover from previous testing.

* Delete the extra line mentioned in Reijo's comment.

* update how HK database delays are processed (#1644)

* update how HK database delays are processed

* make sure stop is not None

* ignore the suprsync folders

---------

Co-authored-by: Tran Tsan <ttsan@login25.chn.perlmutter.nersc.gov>
Co-authored-by: Tran Tsan <ttsan@login37.chn.perlmutter.nersc.gov>
Co-authored-by: Katie Harrington <katie.megan.harrington@gmail.com>
* use valid_data in metrics

---------

Co-authored-by: Michael McCrackan <mmccrack@login32.chn.perlmutter.nersc.gov>
* Update preproc documentation.

* Fix link render.

* Fix test failure in Hessian.

* Reformat the sidebar sections.

* Address Michael's comments

* Fix docs build warnings across all modules.

* Add details about running first time vs later times.
* Add Jbolo calculation to dPhi0dT calculation

* Remove reference detector parameters

* Fix typo

* Add support for LF SAT using older bolocalc version

* Move JBolo calculation out of detector loop

* Safeguard against reference issues

* Check for saturated detectors

* Nest jbolo import inside try block

* Handle saturated detectors

* Add comment clarifying JBolo version

---------

Co-authored-by: Toby Satterthwaite <tpsatt@dhcp-visitor-enr-112-189.slac.stanford.edu>
Co-authored-by: Toby Satterthwaite <tpsatt@Tobys-MacBook-Pro-3.local>
Co-authored-by: Toby Satterthwaite <tpsatt@dhcp-visitor-enr-113-180.slac.stanford.edu>
Co-authored-by: Toby Satterthwaite <tpsatt@mac.mynetworksettings.com>
* allow other types of non-monotonic fields

* add hk suprsync to the list

* revert dependency change

* missed an important return statement

* missed an import

* cleanup hk suprsync as well
#1689)

Updated get_turnaround_flags so that it can catch stationary observations that may trickle through. Introduces and az_throw_threshold and logs when below threshold.
* Add FileNotFoundError to transient errors list (#1720)

* Enhance get_cal_obsids with max_days_before parameter (#1720)

Added max_days_before parameter to get_cal_obsids function to allow dynamic adjustment of the time range for calibration operations.

* Find bgmap max 20 days before (#1720)

* Use configurable max_days_before in bgmap loading (#1720)
* Fix axis error in circle fit parameters wiregrid.py

* Add circular_mean function for wire grid (#1699)

* Modify location of wrapping (#1699)
* fix plotting bug and increase x tick count for quality reports

---------

Co-authored-by: Michael McCrackan <mmccrackan@compute11.simonsobs.org>
* fix qr bug and add index file support.

---------

Co-authored-by: Michael McCrackan <mmccrackan@compute11.simonsobs.org>
* add bandpass select to preproc noise

---------

Co-authored-by: Michael McCrackan <mmccrack@login38.chn.perlmutter.nersc.gov>
* this check is more annoying than helpful sometimes

* allow this through CLI
* improve type handling
---------

Co-authored-by: Michael McCrackan <mmccrack@login38.chn.perlmutter.nersc.gov>
Co-authored-by: Yoshinori Sueno <ys5857@tiger3.princeton.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.