Self-contained before/after camera position visualization (#95, #43)

[bpurinton]

What & why

First cut at issue #95 ("visualize bundle adjustment results"), focused on the part @ we discussed: seeing where a camera moved (before → after), not just the residuals on the ground. Also delivers the quiver ideas from #43.

Key design choice: this is self-contained on a bundle_adjust output folder — it does not require the user to supply the pre-BA original camera files (which aren't co-located in a BA folder). Contrast with the existing csm_camera_plot, which needs original + optimized camera pairs.

How it works

New ReadBundleAdjustCameras + PlotBundleAdjustCameras in bundle_adjust.py read three products ASP writes into the BA folder:

• *.adjust — the rigid adjustment: ECEF translation T + rotation quaternion.
• *.adjusted_state.json — the optimized CSM state, used to locate each camera center.
• *camera_offsets.txt (optional) — ASP's authoritative horizontal/vertical camera-center change in local NED.

Per ASP's documented .adjust convention (a world point projects the same in the original camera as R*(P−C)+C+T in the adjusted, with C = camera center for pixel (0,0)), the translation T is the exact bulk camera-center shift at the anchor pixel. I verified against the example data that T's vertical component matches camera_offsets.txt closely, while horizontal differs by the rotation lever-arm — so the tool uses camera_offsets.txt for authoritative magnitudes (bars) and T for the shift direction (quiver). Falls back to T for magnitudes when camera_offsets.txt is absent (flagged by offsets_from_asp).

Three views (#95 + #43)

summary_plot() combines:

1. Position-change map quiver — horizontal camera-center shift arrows, vertical change as RdBu color, with a true-scale quiver key.
2. Per-camera center-displacement bars — horizontal & vertical change per camera.
3. Orientation-change quiver — yaw/roll as arrow direction, pitch as color.

Also a bundle_adjust_cameras CLI (points at a BA folder; no original cameras needed).

Validation

• Ran on real example data (atlanta stereo pair, ucsd, aster jitter) — figures look right; verified the .adjust reconstruction against camera_offsets.txt ground truth.
• Handles both .adjust naming conventions (<base>.adjust and <base>.adjusted_state.adjust).
• New tests (reader with/without offsets, .adjust parsing, id normalization, the three plot methods, error path) + camera_offsets.txt fixture. Full suite: 396 passed.

Notes / open questions for review

• This is intentionally a first, self-contained slice to react to — happy to wire it into the PDF report pipeline (report_pipeline.py) as a follow-up, and/or add the ellipsoid-relative / NED-vector refinements from Ideas for 3D quiver plots for camera optimization #43.
• The example BA runs here are position-dominated, so orientation changes are near-zero (panel 3 is mostly a placeholder on this data); it'll be more informative on jitter-solve runs.
• Multi-camera (N-scene) BA will make the map quiver much richer than the 2-camera stereo pairs shown.

Closes part of #95; addresses #43.

[bpurinton]

Very WIP, and continuing to iterate, but getting some interesting ideas from initial outputs:

[bpurinton]

Perhaps some progress on design. Trying to capture the horizontal + vertical shifts along with the change in angular orientation:

[bpurinton]

Bundle adjustment vs. jitter correction (context for this PR)

Short answer: your intuition is exactly right. Bundle adjustment applies one rigid position + orientation correction to the whole camera; jitter correction adjusts the camera's position and orientation line-by-line along the trajectory. ASP's own docs put it almost word-for-word the way you did — from jitter_solve (docs):

jitter_solve … adjusts each individual camera position and orientation in the linescan model … This is quite analogous to what bundle_adjust does, except that the latter tool has just a single position and orientation per camera, instead of a sequence of them.

Why a pushbroom camera can be corrected with "one shift"

The thing that confused you is the key insight: a linescan/pushbroom camera model already contains the full flight trajectory — a time-sampled sequence of positions (ephemeris) and orientations (attitude quaternions). In the CSM state file these are the m_positions and m_quaternions arrays; each image line is imaged from its own interpolated position/orientation along that sequence.

bundle_adjust does not touch those samples individually. It solves for just six numbers per camera — one translation T (3) + one rotation R (3, internally axis-angle, stored as a quaternion) — and applies that same rigid transform to the entire stored trajectory — it moves/rotates all the pose samples together as a rigid body. So it corrects the bulk, low-frequency error: the whole camera's absolute position and pointing being off (which is what produces the DEM offsets/tilts and misregistration between overlapping DEMs that BA is designed to fix). What it cannot change is the internal shape of the trajectory — the relative wiggle from one line to the next.

The .adjust file is literally just that rigid transform (.adjust format): a translation T (x, y, z, meters) and a rotation R (quaternion w, x, y, z), applied about the camera center C for pixel (0,0):

P' = R * (P - C) + C + T

The rotation is around the camera center C for pixel (0, 0) (for a linescan camera the camera center depends on the pixel). These are applied on top of initial cameras.

(That single R/T per camera is exactly what this PR's bundle_adjust_cameras tool reads and visualizes — one center shift + one roll/pitch/yaw change per camera. More on that below.)

What jitter actually is

"Jitter" is the high-frequency part BA leaves behind. From jitter_solve (§ Camera constraints):

Jitter is believed to be caused by vibrations in the linescan camera as it acquires the image. If that is the case, the camera positions are likely accurate, and can be constrained to not move much, while the orientations can move more.

So during the ~1–10 s it takes to scan the scene, micro-vibrations (reaction wheels, thermal snap, structural resonance) make the pointing oscillate faster than the smooth, sparsely-sampled ephemeris/attitude model captures. Over a pushbroom scan those attitude wiggles imprint a wavy, oscillatory distortion into the image and the resulting DEM (the "washboard"/banding artifacts you see in the ASTER and CTX jitter examples). A single rigid R/T — no matter how well chosen — literally has no degrees of freedom to represent a wiggle, which is precisely why jitter is a separate problem.

ASP even says this directly in the bundle-adjustment docs, next to a post-BA intersection-error figure: "The remaining wavy pattern is due to jitter, that ASP has a solver for." BA does its job (removes the bulk error), and whatever wavy residual is left over is, by definition, the jitter.

What jitter_solve does differently

jitter_solve optimizes the per-pose positions and orientations along the trajectory — in the source it literally writes back into the m_positions and m_quaternions sample arrays of the CSM linescan model, pose by pose (JitterSolveCostFuns.cc) — so it can bend/warp the trajectory to remove the line-to-line perturbation. A few practical consequences from the docs:

• It is normally run after bundle adjustment, taking BA's output as its starting point (jitter_solve <images> <cameras> <input adjustments>). BA removes the bulk error first; jitter cleans up the residual oscillation. (It can be skipped-into directly if the input cameras are already accurate and well-aligned, but that's the exception.)
• Because it has so many free parameters, it's easy to over-fit, so it must be constrained. The docs "strongly recommend" always using anchor points (--num-anchor-points + --anchor-dem) to pin orientation, plus --camera-position-uncertainty to keep positions near their (trusted) values, and optionally smoothness/roll/yaw weights. Without these the solution "can become unstable." A reference-DEM constraint is also usually needed, because "a long-wavelength perturbation consistently applied to all obtained camera trajectories may work just as well" (the low-frequency modes are ambiguous).
• The poses are often resampled first (--num-lines-per-position / --num-lines-per-orientation) — e.g. Pleiades samples position every 30 s while the whole image takes ~1.6 s, so you need to densify before you can even represent jitter; CTX is the opposite and needs thinning.
• Output is the full optimized CSM state (.adjusted_state.json), not a .adjust file — because the correction is no longer expressible as a single rigid transform (".adjust files should not be used as they save the adjustments only").

One-line mental model

	bundle_adjust	jitter_solve
Free parameters per camera	1 position + 1 orientation (rigid R,T)	a sequence of positions + orientations (per pose/line)
Error it fixes	bulk / low-frequency (absolute position & pointing)	high-frequency along-track oscillation (jitter)
Output	.adjust (T + quaternion)	full .adjusted_state.json CSM state
Typical order	first	after BA, heavily constrained

Relevance to this PR

This is actually why the two viz layers in asp_plot look different:

• bundle_adjust_cameras (this PR) correctly models each camera's change as one center shift + one roll/pitch/yaw change — a single arrow / single body-axis cartoon per camera — because that's all a BA .adjust is. On a position-dominated BA run the orientation change is ~1e-4°, which is why the cartoon prints the number instead of scaling an arrow.
• csm_camera.py / csm_camera_summary_plot is the right tool for jitter: it plots the position and roll/pitch/yaw differences as a function of image line along the trajectory, so you can actually see the oscillation that jitter_solve removed. So the two tools are complementary: per-camera rigid summary (BA) vs. along-track per-pose profile (jitter).

Sources: ASP docs bundle_adjustment, bundle_adjust (§ Format of .adjust files), jitter_solve (overview, § Camera constraints, § Anchor points, § Resampling the poses).

[bpurinton]

Added the nice writeup above in #154 (comment) as a new docs section in the stereopipeline-quickstart:

uw-cryo/stereopipeline-quickstart@2ce1afd

https://stereopipeline-quickstart.readthedocs.io/en/latest/concepts/bundle-adjustment.html#what-bundle-adjustment-cannot-fix-jitter