[skip benchmarks] Vggt verified pipeline Code Changes - #1118
Draft
kathirgounder wants to merge 93 commits into
Draft
[skip benchmarks] Vggt verified pipeline Code Changes#1118kathirgounder wants to merge 93 commits into
kathirgounder wants to merge 93 commits into
Conversation
When use_view_graph_calibration=True, refines VGGT's predicted intrinsics via
joint Fetzer optimization over the frontend's F-matrix edges (existing
calibrate_view_graph from gtsfm/view_graph_estimator/view_graph_calibration.py)
before the cluster BA. VGGT's predicted poses are unchanged; only the
intrinsics handed to BA are swapped from VGGT-predicted to view-graph-refined.
Useful for uncalibrated phototourism scenes where VGGT's predicted focals
may be unreliable.
Default off (use_view_graph_calibration: false) → behavior identical to master.
## Diff
- `_refine_vggt_intrinsics_via_view_graph` helper at module level: rescales
VGGT's predicted intrinsics from VGGT pixel space to original image coords,
feeds them as initial estimate to `calibrate_view_graph`, returns refined
intrinsics in original image coords.
- `_build_gtsfm_data_from_vggt_depth` accepts optional `refined_intrinsics` —
when provided, uses them for the BA cameras instead of rescaling VGGT's.
Depth lookup is unaffected (uses VGGT pixel coords directly).
- `ClusterVGGTWithFrontend.__init__` adds `use_view_graph_calibration: bool = False`.
- `create_computation_graph` adds one conditional dispatch block.
- 3 yamls (vggt_sift_frontend_megaloc, _phototourism, vggt_unified_frontend_megaloc)
expose the flag with default false.
## How to enable
```yaml
cluster_optimizer:
optimizer:
_target_: gtsfm.cluster_optimizer.VggtWithFrontend
...
use_view_graph_calibration: true
```
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the retri stage merged in borglab#1111 (BundleAdjustmentOptimizer) at the cluster-BA layer so VGGT-frontend pipelines also recover tracks dropped earlier. After the initial cluster BA + filter, optionally re-triangulates the union-find 2D tracks against the post-BA cameras (multi_view_retriangulate_from_2d_tracks) and runs a second cluster BA on the augmented set. Default off (use_multi_view_retriangulation: false) → no behavior change. ## Diff - `_run_cluster_ba` (shared by ClusterVGGT and ClusterVGGTWithFrontend) gains `tracks_2d` and `use_multi_view_retriangulation` kwargs. When the flag is on and tracks_2d is supplied, runs retri + a second BA + post-BA reproj filter. - `ClusterVGGTWithFrontend.__init__` adds `use_multi_view_retriangulation: bool = False` and passes the union-find tracks_2d_graph through to `_run_cluster_ba`. - 3 yamls (vggt_sift_frontend_megaloc, _phototourism, vggt_unified_frontend_megaloc) expose the flag with default false. ## How to enable ```yaml cluster_optimizer: optimizer: _target_: gtsfm.cluster_optimizer.VggtWithFrontend ... use_multi_view_retriangulation: true ``` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-calibration Add optional view-graph (joint intrinsic) calibration to ClusterVGGTWithFrontend
…e retriangulation Gated behind a single SceneOptimizer flag `use_verified_pipeline` (default off, so existing configs are byte-for-byte unchanged). When enabled, SceneOptimizer.run: 1. Runs a global two-view verification pass over the full MegaLoc retrieval graph, reusing the per-cluster frontend chain (ClusterMVO._run_correspondence_generator -> _pad_keypoints_list -> _run_two_view_estimation, which already filters to TwoViewResult.valid()). This populates the per-pair caches, so subsequent per-cluster frontends cache-hit. 2. Partitions METIS on the verified subgraph (edges that survive verification) instead of the raw retrieval graph, and logs verified-vs-retrieval edge counts + how many cameras the largest-CC extraction would drop. 3. Builds global 2D tracks from the verified correspondences (get_2d_tracks) and, after the hierarchical merge, retriangulates them against the merged VGGT poses, runs BA (jointly refining structure + poses), and writes the result to results/merged_retriangulated/ with its own metrics -- alongside the unchanged results/merged/ for A/B comparison. Reuses existing machinery only (multi_view_retriangulate_from_2d_tracks, CppDsfTracksEstimator via get_2d_tracks, BundleAdjustmentOptions.run_simple_ba); new code is confined to scene_optimizer.py (one helper + the flag branch). Adds vggt_sift_frontend_megaloc_phototourism_verified.yaml (phototourism config + use_verified_pipeline: true) as the A/B treatment arm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erified) - Fix instantiation crash: merging_options.ba_options.use_pose_prior (removed from BundleAdjustmentOptions) -> use_pose_prior_all_cameras (preserves intent: soft prior anchoring every merged camera to its own pose during merge BA). - Detector: SIFTDetectorDescriptor (OpenCV) -> ColmapSIFTDetectorDescriptor. - MegaLoc retriever: num_matched 15 -> 100, min_score 0.5 -> 0.15 (denser graph, matches the peak megaloc_sift_gp_single_pt config). - METIS: min_cameras_to_partition 12 -> 30, max_cameras 40 -> 70 (larger clusters, shallower tree). Applied identically to both arms so the A/B differs only by use_verified_pipeline. Full config audit (25 _target_ blocks vs class signatures) found use_pose_prior was the only invalid key; all others validate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0/0.15 gate Brings the Brussels phototourism configs (baseline + verified) to the peak two-view frontend from the gp-glomap-parity lineage (commits da03cb8 "PoseLib verifier...", 5ed1955 "Brussels AUC@5 0.730->0.795"): - verifier: Ransac -> PoseLibVerifier (estimation_threshold_px=2). mapping-vggt's PoseLibVerifier is a functional superset of gp-glomap-parity's peak verifier (poselib.estimate_relative_pose: 5-point + LO-RANSAC + 2-view bundle), so no port needed. Dropped the Ransac-only use_intrinsics_in_verification key. (Peak uses PoseLib, not scipy -- scipy was removed in c9ed098.) - ColmapSIFT max_keypoints 5000 -> 8192. - inlier_support_processor 15/0.1 -> 30/0.15 (GLOMAP-matched edge-quality gate). - Declare poselib>=2.0 in pyproject (PoseLibVerifier imports it at module load). Note: not using the fetzer-only estimate_calibration_geometry variant (that param does not exist on mapping-vggt's PoseLibVerifier). Applied identically to both arms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…: true) Turns on mapping-vggt's existing scipy Fetzer focal-length refiner (view_graph_estimator/view_graph_calibration.py: scipy.optimize.least_squares on E=K2^T F K1 singular-value residuals, Cauchy loss). It recomputes F from the PoseLib-verified correspondences and refines per-camera focal lengths; VGGT poses are untouched, principal point/distortion fixed. Independent of the verifier and distinct from the gtsam Fetzer SelfCalibrationFactor (which is not on this branch). Left ba_options.use_calibration_prior = false (cluster + merge): the refined focal is used as the BA initialization; BA may still adjust it. Applied to both arms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d pipeline) In the verified pipeline, clusters now reuse the global verified two-view + SIFT tracks (filtered to each cluster's cameras) instead of re-running a per-cluster correspondence + two-view frontend, and build the cluster BA's 3D structure by triangulating those SIFT tracks against the VGGT poses (multi_view_retriangulate_from_2d_tracks) instead of lifting per-pixel VGGT depth. Fixes the trackless-camera problem: cluster-local frontends left many cameras with few/no tracks, and the VGGT-depth 3D init was multi-view-inconsistent so the pre-BA reprojection filter (14px) dropped most tracks (survival 0-76% per cluster). Global SIFT tracks give richer per-camera coverage; triangulation yields consistent structure that clears the filter. - ClusterContext gains an optional precomputed_global_frontend bundle (scattered futures: padded keypoints, verified two-view results, global 2D tracks). - SceneOptimizer scatters these once (broadcast) and threads them into every ClusterContext when use_verified_pipeline is on. - ClusterVGGTWithFrontend branches on the bundle: filters global tracks to the cluster's cameras (_filter_tracks_to_cameras), filters verified two-view to cluster edges for the scipy focal calibration (_filter_two_view_to_cameras), skips the per-cluster frontend, and builds BA input via _build_gtsfm_data_via_triangulation. Baseline arm unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…broadcast broadcast=True replicated the global keypoints + verified two-view results + ~80 MiB track list onto every worker, OOMing the node (KilledWorker -> driver dies at handle.metrics.result()). Scatter once (broadcast=False); workers fetch on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… edges calibrate_view_graph's no-edges early return did 'return list(initial_intrinsics)', yielding camera-index keys instead of the intrinsics dict. Downstream then indexed it as intrinsics -> Cal3Bundler(pose, <int>) TypeError, crashing any cluster with no valid F-edges. Surfaces under use_gt_intrinsics=false (weaker verification). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ated structure client.scatter of the global frontend crashed dask right after the cluster tree. A cluster's BA only uses within-cluster measurements, which the per-cluster frontend already produces (cache-hit from the global verification pass), so the global-track plumbing wasn't worth the scatter fragility. Revert to the per-cluster frontend for cluster tracks; keep the triangulated 3D init (the actual fix for the trackless-camera / pre-BA-reproj attrition) behind a new use_triangulated_structure flag (true in both phototourism configs). The global two-view verification stays (verified-graph partition + post-merge retriangulation, which takes the concrete global_tracks_2d again). - cluster_vggt_with_frontend: always per-cluster frontend; 3D init triangulate-vs-VGGT-depth via use_triangulated_structure; drop the _filter_* helpers. - scene_optimizer: remove scatter + PrecomputedGlobalFrontend; post-merge retri takes concrete tracks. - cluster_optimizer_base: drop the precomputed_global_frontend field + bundle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…& merge BAs Use VGGT only for poses, Fetzer only for focals, and stop every BA from re-optimizing the focal away from its Fetzer value. Global Fetzer (verified pipeline): - compute_global_view_graph_intrinsics() runs one Fetzer optimization over the full verified view graph (heuristic init, never VGGT focals), so camera i gets a single F_global[i] identical in every cluster. - ClusterContext.global_refined_intrinsics carries it; ClusterVGGTWithFrontend gains use_global_view_graph_calibration (flag/property/__repr__/selection, precedence global -> per-cluster -> raw VGGT); SceneOptimizer computes it once. - Also fixes the broken view-graph-calibration logger (get_logger()). Two-tier focal anchoring (both phototourism configs): - cluster BA: use_calibration_prior=true, focal_sigma=5px -- pin focals at the Fetzer value so per-cluster BA optimizes poses, not focals. Without this each cluster drifts the same camera's focal differently (focal/depth ambiguity), so parent vs child separator-camera focals diverge -> noisier Sim3 merge. - merge BA: use_calibration_prior=true, focal_sigma=10px -- looser, where poses are globally reconciled. Note: cluster ba_options is in __repr__ (cache key), so this invalidates the cluster cache -- a full recompute, which global Fetzer requires anyway. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…low fix)
The triangulated-structure path inflated each cluster to 11k-15k tracks (e.g.
C=14887, C_3_1=11090) -> 85-200s BAs on huge factor graphs -> worker OOM and
cascading dask FutureCancelledError ("lost dependencies" + nanny kill), with no
proven gain over Akshay's VGGT-depth baseline (219 cameras / AUC@3 0.675).
Flip use_triangulated_structure: true -> false in the verified config, reverting
per-cluster 3D init to lifting VGGT predicted depth (the known-good, low-memory
path). The focal-flow fix is unaffected: _build_gtsfm_data_from_vggt_depth applies
the global Fetzer focals identically (refined_intrinsics), and the cluster/merge
calibration priors anchor at that focal regardless of build path. Post-merge global
retriangulation (use_verified_pipeline) and global Fetzer are kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-cluster frontend ran correspondence generation and two-view estimation as Dask tasks that themselves opened worker_client() and submitted per-image / per-pair sub-tasks, then gathered them (cluster_mvo._run_correspondence_generator /_run_two_view_estimation). On a single worker that nested gather holds the entire frontend working set resident at once -- every image's keypoints+descriptors, every pair's putative correspondences, every TwoViewResult -- as live futures. Over the global verification graph (14585 pairs / 236 images) that balloons to tens of GB and, when the worker tips, the nested gather cannot recover (no other worker holds the deps) -> FutureCancelledError "lost dependencies", fatal. It was always the same line. Replace the nested submission with inline execution, mirroring the existing synchronous ColmapCorrespondenceGenerator pattern: - DetDescCorrespondenceGenerator.generate_correspondences_inline(images, vg): detect once per image, match once per pair, plain loops over the cache-backed primitives (+ base-class stub). - two_view_estimator.create_two_view_results_inline(...): same per-pair kwargs as create_two_view_estimator_futures, but call run_2view directly in a loop. - cluster_mvo: both _run_* methods drop worker_client(); images now arrive as a normal Dask dependency (a list of image futures, auto-materialized by delayed) instead of being re-gathered inside a nested client. scene_optimizer global-verification call site updated to match. Computed-and-discarded item by item, so peak memory is bounded to one cluster's features rather than the whole graph. Method bodies don't change the optimizer repr, so existing cluster caches stay valid (no forced recompute). Frontend is now serial within a task (no per-pair Dask parallelism); cache-cold runs are slightly slower. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… measurements _build_gtsfm_data_from_vggt_depth kept a SIFT measurement only if VGGT per-pixel depth_confidence > 0 at the keypoint, so cameras whose keypoints fell in low-confidence VGGT regions (low-texture/transient content, or portrait images center-cropped to 518) got zero tracks at construction and were dropped at merge/retri. On Brussels this lost 5 connected cameras (idx 5/104/160/206/207) that have abundant verified edges and are richly tracked in GLOMAP. Decouple the gate: every verified SIFT measurement now enters the track; only conf>0 depth anchors the 3D-point init. The existing per-measurement 14px pre-BA reproj filter then prunes pose-inconsistent observations, so recovery self-selects on pose quality — cams with sound VGGT poses (5: 0.30deg, 104: 0.16deg vs GLOMAP) return with real BA-refined tracks, while bad-pose cams (206 ~10deg) correctly stay out. Track count is unchanged (same confident-anchor gate); only measurement count grows, so no factor-graph blow-up like the reverted global use_triangulated_structure path (ede5a9b). __repr__ gains an /allkpts cache token (when use_triangulated_structure=false) to force the cluster recompute. Also documents a reverted global-Fetzer median-fill of unrefined focals: the cameras Fetzer can't refine skew high-focal (true f/maxdim ~1.2) where the 1.2 heuristic is already accurate, so median-fill hurt 3 cams to help 1 (215). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er frontend) The verified pipeline ran the SIFT frontend twice: once globally (the two-view verification, which produces padded keypoints + v_corr_idxs_dict), then again per cluster. The second pass dominated wall-clock -- not from recomputing SIFT (it cache-hits) but from serially deserializing 3 disk caches (detector + matcher + two-view) over hundreds of pairs, redundantly across heavily-overlapping clusters (~60s/cluster x ~40 clusters). A cluster build only needs tracks_2d (it extracts v_corr, runs get_2d_tracks, and discards the two-view relative poses/F/configs -- VGGT supplies poses). The global v_corr + keypoints already live in the main process, where each cluster's create_computation_graph also runs. So when reuse_global_correspondences is set: subset the global v_corr to the cluster's edges and build tracks_2d EAGERLY in the main process, then pass it into the VGGT build and skip the per-cluster frontend entirely. Only the resulting per-cluster tracks_2d (~1 MiB) is embedded in the dask graph -- no scatter (which OOM'd before), no full-dict embedding. Per-edge v_corr is identical to the per-cluster frontend output (same edges, same two-view, same heuristic intrinsics), so this is a pure speedup; verify by flag-on-vs-off track-count + AUC parity. Plumbed via two new ClusterContext fields (mirroring global_refined_intrinsics); gated by the reuse_global_correspondences flag with a /gcorr __repr__ cache token so the cluster cache invalidates once and flag-on/off get distinct keys. Falls back to the per-cluster frontend when the globals are absent (non-verified runs). Enabled in the verified config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cameras with good VGGT poses but zero VGGT depth-confidence (e.g. Brussels 104/207) get no 3D track at per-cluster construction (the confident-anchor gate) and are dropped at the merge. Capture those filter-dropped good-pose cameras with their merged poses and inject them into the post-merge retriangulation, which is depth-confidence-independent: it re-triangulates their existing global 2D tracks against the good poses. Cameras that still fail to gain a >=3-view track are cleanly dropped (final filter retain decoupled from keep_all_cameras). Gated behind recover_trackless_cameras_in_retriangulation (off by default; on in the verified config). Adds a per-camera recovery diagnostic (recovered, retri_tracks, global tracks touching, max views). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ports VGGT-Omega support from PR borglab#1116 (hkhanuja) — the VggtOmegaGeometryTransformer, the thirdparty/vggt-omega submodule (@39a0cb8), the gated-weights download flag, and the baseline omega optimizer/config — and runs omega geometry through OUR verified optimizer (ClusterVGGTWithFrontend, via its injected geometry_transformer) so it inherits global Fetzer focals, correspondence reuse, and the post-merge trackless-camera recovery (rather than Harneet's optimizer, which lacks those). Code wiring (2 small fixes): (1) cluster_vggt_with_frontend.py guards the two transformer.config dereferences (__init__ loader_kwargs + __repr__ cache key) so a transformer without a .config (omega) instantiates and caches cleanly; (2) vggt_omega_geometry_transformer.py adds a per-worker model singleton so the 1B weights load once per worker when the optimizer drives predict() with model=None. New config vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml: the verified config with the geometry transformer swapped to omega and model_cache_key=false (forces omega self-load; null would silently load VANILLA VGGT). Controlled A/B vs the 0.6968/230-cam VGGT-verified run — only the geometry predictor changes. With omega's tighter poses, the trackless-recovery is the lever expected to finally recover 104/207. Requires the gated cc-by-nc-4.0 omega weights (HF facebook/VGGT-Omega) + 'git submodule update --init --recursive'. CUDA-only. Omega module imports are lazy (registry + Hydra), so non-omega runs are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The omega run was feeding VGGT-Omega images preprocessed by the VGGT loader (518px/14-aligned, pad-only) because _load_vggt_inputs hardcoded load_image_batch_vggt_loader. Omega expects its own preprocessing (512px/16-patch-aligned, aspect-cropped) and the depth-lift indexes the model's depth map via the loader's original_coords, so the mismatch ran omega out-of-distribution and likely mis-indexed its depth (measured: pre-BA track survival 75% vs VGGT's 81% on the same cluster). Fix: add GeometryTransformer.load_image_batch (default = the VGGT loader); VggtOmegaGeometryTransformer overrides it to use load_image_batch_vggt_omega_loader (mode=balanced; omega's modes are balanced/max_size, not VGGT's crop/pad). _load_vggt_inputs now dispatches via transformer.load_image_batch, and both call sites (ClusterVGGT, ClusterVGGTWithFrontend) pass self.geometry_transformer. Provably a no-op for VGGT: VggtGeometryTransformer inherits the base, which is the exact prior load_image_batch_vggt_loader(loader, indices, mode=input_mode) call. Only an omega transformer changes the loader. NOTE: the prior (mis-preprocessed) omega per-cluster cache must be cleared before rerunning, since this fix does not change the optimizer repr/cache key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Decomposes the 16-commit branch into 6 self-contained feature units (verified pipeline, global Fetzer, peak frontend, correspondence reuse, trackless recovery, VGGT-Omega) mapping 1:1 to the clean branches we'll land into master, with per-unit commits/files/design notes, the config-flag reference, the Brussels A/B result (0.7191 AUC@3 / 231 cams), and known follow-ups (104/207 PnP). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per Harneet's PR review. In _build_gtsfm_data_from_vggt_depth, keypoints in the cropped-away margins (under crop / omega aspect-crop input modes) map outside the VGGT dense map. The old np.clip(..., 0, W_vggt-1) clamped them to the border pixel and read THAT pixel's depth/confidence — anchoring the track's 3D point from an unrelated edge location, and letting border garbage count toward the >=min_track_length confident-anchor gate. Now we bounds-check and 'continue', so only genuinely in-bounds depths anchor the point. Safe: all_measurements.append moves above the pixel computation, so the BA-constraint set is unchanged (every measurement still added); only the depth-anchor set drops border-clamped garbage. In-bounds keypoints are byte-identical (clip was a no-op when in-bounds). NOTE: build output changes for cropped images -> clear the cluster_optimizer cache before re-running. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tabase.db GTSFM's Dask SIFT frontend OOMs on large scenes (St Peter's: ~2.5k imgs, 123k edges -> worker killed at the global two-view gather). This reads the verified frontend from a COLMAP database.db built offline (GPU SIFT + vocab-tree matching), bypassing the Dask frontend. New: ColmapDBRetriever (gtsfm.retriever.ColmapDB) returns the db's geometrically-verified pairs; scripts/build_colmap_db.sh builds a db per scene; vggt_omega_..._colmapdb.yaml wires retriever+ColmapCorrespondenceGenerator into the verified pipeline (db path defaults to <dataset_dir>/database.db). Reuses the orphaned ColmapCorrespondenceGenerator (db keypoints + verified two_view_geometries, with resolution rescale). Surgical bypass in scene_optimizer.py: a new CorrespondenceGeneratorBase.produces_verified_correspondences flag (True only for ColmapCorrespondenceGenerator) gates a branch that reads keypoints + v_corr from the db in the MAIN PROCESS — no Dask two-view estimation, no client.gather of all per-edge results (the OOM). Lines 337-366 (verified graph -> tracks -> reuse -> Fetzer) run unchanged; non-colmapdb configs are byte-identical (flag defaults False). VERIFY-FIRST on St Peter's (reproj is the canary): (1) db keypoints must land in the loader's resolution frame (ColmapCorrespondenceGenerator scales by image.width/camera.width); (2) for pairs where COLMAP image_id order != GTSFM filename order, confirm two_view_geometry.inlier_matches columns aren't swapped. Bad reproj => one of these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The verified-pipeline global two-view pass gathered all ~N full TwoViewResults
(three per-point BA reports + putative idxs each) into the client, which
OOM-killed the client process on RAM-limited single nodes (e.g. PACE 1xH200).
Only v_corr_idxs is consumed downstream on this path (relative poses come from
VGGT per cluster), so collapse to the {(i1,i2): ndarray} dict inside the delayed
graph via _extract_v_corr_idxs_dict and gather only that. run_2view still runs
identically on the worker, so TwoViewEstimatorCacher/DB writes and valid()
filtering are unchanged; downstream consumers get the same v_corr_idxs_dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lean-gather fix (prev commit) bounded the CLIENT, but the verified-pipeline
global pass still built the full {edge: TwoViewResult} dict on ONE worker before
reducing — ~48k results x three per-point BA reports each (10-48GB) OOM-killed the
worker on large scenes (St Peter's, 48k edges). Add create_v_corr_idxs_inline +
ClusterMVO._run_two_view_v_corr_idxs: run run_2view per pair but keep only each
valid edge's v_corr_idxs and drop the heavy TwoViewResult immediately, so the
worker holds ~one result at a time instead of all N. run_2view still executes
identically (TwoViewEstimatorCacher/DB writes + valid() filtering unchanged);
_run_two_view_estimation and the per-cluster path are untouched (backward-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…k worker) The verified-pipeline global frontend ran as one monolithic delayed task on one worker, with the main process blocking on a single client.gather for the entire multi-hour run. On large scenes (St Peter's, ~2500 imgs / 48k edges) the worker was repeatedly dropped mid-run with 'lost dependencies' + register-client comm churn. Confirmed NOT memory: node has 2TB RAM, frontend footprint ~15-20GB, and it died identically at 90GB and 128GB worker limits with no dask memory/GIL/segfault warning in the log. It is a scheduler<->worker comm/coordination failure over the long run. Fix: mirror the COLMAP-DB branch and run the frontend inline in the main process -- gather the downsampled images once, then call _run_correspondence_generator + _pad_keypoints_list + _run_two_view_v_corr_idxs directly (they are plain, no worker_client). No worker runs the frontend, so there is nothing for a comm hiccup to drop. Streaming v_corr reduction + TwoViewEstimatorCacher/DB writes unchanged; memory-trivial on a 2TB node; still serial (cache makes reruns fast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… timeouts)
St Peter's (~2500 imgs) died in the image-loading phase with CommClosedError on the
client<->scheduler comm: get_image_futures submitted 2500 images in a tight
client.submit loop, which starves the client event loop between submits so it can't
drain the scheduler's key-in-memory replies -> the batched comm backs up and closes
mid-submission (seen at loader-get-image-1401), then the downstream gather hangs on
never-submitted futures.
- loader_base.get_image_futures: bulk-submit via client.map (one update-graph message)
instead of a per-index submit loop. Preserves keys (loader-get-image-{idx}) + worker
pinning. Verified client.map accepts a list key on distributed 2025.9.1.
- runner: raise distributed.comm.timeouts.connect/tcp 30s -> 300s so transient event-loop
stalls (bulk loads, the in-process frontend) don't tear down comms mid-run.
Complements the inline-frontend fix (1e1cc46): that removed the worker-side coordination,
this removes the client-side image-load storm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline global frontend goes dark for tens of minutes with no signal. Add periodic progress logs (count, elapsed, rate, ETA) to the three serial loops so the multi-hour run is observable: - det_desc_correspondence_generator.generate_correspondences_inline: detection (every 250 imgs) + matching (every 5000 pairs). Adds a module logger (had none). - two_view_estimator.create_v_corr_idxs_inline: two-view (every 2000 pairs, + valid count). Logging only; no behavior change. Restart is cheap — the per-item cachers replay completed detection/matching/two-view instantly, so you get live progress from where it left off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bscription) BFMatcher parallelizes over cv2's global thread pool (default: all cores), so every concurrent apply_matcher task spawned a full-node pool — with 5-16 dask workers, 5-16x oversubscription explains both the slow matching stage and why adding workers made it WORSE. Same bug family as pycolmap SIFT num_threads=-1 (818f7b4). Set in match() since __init__ doesn't re-run in worker processes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cher Drop-in alternative to the OpenCV TwoWayMatcher with identical semantics — validated on real production SIFT descriptors (Brussels cache): match-set Jaccard 1.0000 across all tested pairs. One distance GEMM + topk per direction; ratio test applied in BOTH directions before the mutual intersection, exactly mirroring the OpenCV path (the unidirectional version disagreed at Jaccard 0.35 on real data — caught by the cache A/B). 2.2x faster than OpenCV even on 1 CPU thread (854 vs 1873 ms/pair at ~10k kps); ~2ms/pair on datacenter GPUs, which are idle during the matching stage. CPU fallback pins torch to 1 thread (dask workers are the parallelism). Opt-in via config/CLI: ...matcher.matcher_obj._target_=gtsfm.frontend.matcher.torch_twoway_matcher.TorchTwoWayMatcher Note: new matcher repr => matcher/two-view caches recompute (cheap at GPU speed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… verified config
Roman Forum audit: the [0.25, 4] band executed 18 lawful children (~1,200
cameras including a 258-cam subtree at scale 4.06; others at 4.7-12.5x and
0.07-0.25x). The band's premise ('metric clusters, scale ~ 1') does not hold:
VGGT normalizes scene scale per batch, so clusters with heterogeneous spatial
extents legitimately need large scale corrections at the seat. The detonations
the band was built for were 1e8+ — [0.02, 50] still rejects those by orders of
magnitude while admitting lawful sprawling-scene seats. Both guard sites
(pre-solve pair prefilter + post-solve) now read MergingOptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The band's premise (metric clusters, solved scale ~ 1) is false: VGGT normalizes scene scale per batch, so lawful seats on heterogeneous scenes routinely solve at 4-12x (Roman Forum: 18 lawful children executed, ~1,200 cams). Genuinely diverged seats remain covered by the structureless/0-track guards, the correspondence floor, the pose-pinned robust merge BA, and the post-merge reprojection filters; NaN scales still drop (fail the comparison). Set a finite band in MergingOptions to restore legacy behavior. Pre-team-grind audit: 27/28 targeted tests pass (1 = known pre-existing flake in combined runs only, passes in isolation); all edited modules compile; torch matcher remains opt-in; vanilla IMC config inherits only (a) gated Fetzer and (b) band removal — both validated against 1DSfM gold GT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For multi-node parallel experiments sharing one cache: - All six cachers now resolve their root via cache_utils.get_cache_root(), honoring the GTSFM_CACHE_ROOT env var (default: <repo>/cache, unchanged). Dask workers inherit the launcher's environment, so exporting the var at launch covers the whole worker fleet. - write_to_bz2_file is now atomic (unique tmp + os.replace): concurrent same-key writers and killed workers can no longer leave torn .pbz2 entries (observed in production caches). Readers already self-heal on corruption (delete + recompute); now they never see partial files in the first place. Verified: default root unchanged; override reaches cacher module constants in fresh processes; atomic round-trip; corrupted-entry self-heal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ double/Vector1 storage The closed-form focal factor appears as SelfCalibrationFactor in current gtsam-develop and FetzerFactor in pre-rename builds; older builds store the focal variable as Vector1 (optimize() throws GenericValue<double>-vs-Matrix). _solve_focals now resolves the class by either name and retries with Vector1 storage on failure, so any gtsam vintage with the factor gets the gtsam path automatically (scipy fallback unchanged otherwise). Validated end-to-end in the gtsfm-v1 env (FetzerFactor + Vector1): identical solve to the manual PR-parameter run on 6,035 gated British Museum edges. On those edges vs IMC GT intrinsics: gtsam 3.28% median focal err / 52% >3% vs scipy 4.06% / 62% — modest but consistent solver win. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Config-reachable sigma for the PriorFactorPose3 added by use_pose_prior_*, anchored at the initialization poses. Near-zero (1e-3) freezes poses so the cluster BA refines only structure+calibration — the self-calibration recipe validated offline on British Museum (Fetzer 8% focal error -> 1% after pose-frozen focal-free cluster BA; full-tail replay AUC@10 0.429 -> 0.677). Default (0.1) matches the previous ctor default: zero behavior change. NOTE: BundleAdjustmentOptions repr feeds the cluster cache key, so pulling this commit invalidates existing cluster caches (one-time VGGT recompute). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The calibration/pose priors that protect the incremental merges throttle the full-scene retri BA. retri_free_ba drops them (GNC on) for a fully-free final solve, GLOMAP-style; retri_iterations alternates retriangulation with the BA. Offline BM replay: full AUC@3 0.232->0.546 (baseline clusters) and 0.504->0.574 (selfcal clusters) on identical merged inputs; @10 0.477->0.794 / 0.770->0.818. Defaults (false, 1) preserve existing behavior exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Defaults (false, 1) = current behavior; keys present so campaign runs can override without '+'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GNC ablation on BM: +0.01 AUC for ~2x solve time — the win is from removing the priors (0.770->0.806 @10 constructed with the config's plain GM kernel). GNC stays reachable via merging_options.ba_options.use_gnc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Frustums: smaller (0.01 -> 0.006 of scene extent) and near-black (Color3(0.1) instead of orange), 1DSfM/Snavely wireframe look. Camera-center dots also darkened + shrunk to match. - New H hotkey toggles a "clean view" that hides BOTH overlays (the top-left stats card #sceneStats and the bottom control bar #hud) for an unobstructed screenshot; press H again to restore. Wired through _applyStatsVisibility / _applyHudMode so it survives scene reloads and stats/hud state changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build overrides VGGT/Omega's predicted intrinsics with the view-graph focals, making them unrecoverable from any export. Capture them (rescaled to original resolution) before the override and write vggt_predicted_intrinsics.json next to the vggt/ exports. No cache-key change; cached clusters simply skip the sidecar. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Bring the pipeline-viz branch's modern viewer in as a second, standalone viewer alongside ./viz (which is untouched). The branch version only handled pipeline traces (manifest.json + stage_*) and was missing its templates/index.html, so it couldn't run at all. Adapt it to render STATIC reconstructions: - server.py: find_runs() discovers static reconstructions (any dir with points3D.txt, excluding trace stage_* dirs) in addition to pipeline traces; the manifest endpoint synthesizes a single-stage manifest for static recons; default --base is now `results` (same data ./viz uses). Port 5174. - templates/index.html: NEW — the DOM shell viewer.js expects (stats card, help card, control bar) styled to the existing Snavely-inspired style.css. Trace-only controls (mode/play/timeline/stage) carry data-role="trace-only". - viewer.js: build stage URLs that tolerate an empty subdir (static = dir root); hide the trace-only controls when a run has a single stage; add an H hotkey to hide all overlays for clean money shots (matches ./viz). Server path verified end-to-end locally (discovery -> synthesized manifest -> data serving -> index render, all 200s). Run: ./pipeline-viz --base <results>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The colmapdb variant had drifted from the golden settings: depth-lift structure (use_triangulated_structure=false), floater-injecting trackless recovery (true), and missing calibration_source (class default fetzer, golden is exif passthrough). Also mirrors the retri_free_ba/retri_iterations keys and tail flags so per-run overrides match the verified config's shape. Frontend swap (ColmapDB retriever + correspondence reader) is now the ONLY functional difference between the two configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The E/F-only filter silently dropped PLANAR/PANORAMIC/PLANAR_OR_PANORAMIC two-view geometries — ~1.3k of Wilson's ToL EG pairs are H-verified (flat walls, rotation-heavy viewpoints) and GLOMAP consumes them, so same-graph head-to-head runs were eating a leaner diet than the baseline. Low-parallax structure from these pairs is handled by the existing triangulation and angle filters downstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t H-verified pairs Two fixes for same-graph runs on DBs built with --image_path above the images dir (names stored as 'Scene/images/x.jpg'): 1. read_image(loader_basename) missed, returning an invalid image_id (uint32 -1) that crashed pycolmap's read_two_view_geometry with 'image_id1 < kMaxNumImages (4294967295 vs 2147483647)'. Lookups now go through a basename-keyed map (mirrors ColmapDBRetriever). 2. The matches reader had its own E/F-only config filter, which would have silently dropped homography-verified pairs (configs 4-6) even after the retriever started admitting them. Widened to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Negating only the up row of the scene rotation gave det=-1 — a chiral mirror that rendered reconstructions backwards. Negate up+forward rows (true rotation) and adjust the reframe target accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carry cameras dropped by post-BA track filters up the merge tree OUTSIDE the scenes: each child's annex is re-seated by the Sim3 relating its scene to the merged node (same rail as the gaussian-splat carry), fresh drops override, re-registered cameras purge. The retri stage's own drops join at the root; the annex is re-seated onto the final scene and exported as results/annex_posed_only (poses only, zero tracks). Inert by construction: the annex never enters a Sim3 solve, duplicate resolution, or BA — unlike keep_all_cameras=true, which re-admits stale cameras into merges and measurably poisons the core (ToL keep_all A/B: core median 1.18m -> 18.35m, 19 rigidly-offset blocks). Recovery census on the golden ToL tree predicts the annex recovers ~90 reference cameras <10m (last-alive median 7.0m) on top of the untouched 434-matched / 1.18m core. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From the adversarial review of ef3773f: 1. Retri-stage drops that never entered the final BA graph hold poses in the root-merge gauge, not the final frame — tag them (never_baed via min_tracks_per_camera membership) and route them through the root->final fSr re-seat with the tree annex instead of treating them as final-frame (which also let them clobber correctly re-seated tree copies when the recovery-injection flag is on). 2. Early-bailout paths (BA exception, no-track prune, BA-off) silently discarded the subtree's accumulated annex while returning a valid scene — carry the children's annexes there too (_annex_on_bailout). Core solve untouched; both fixes are annex-output-only. 17/17 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
parseImages dropped blank lines before pairing pose/points lines, so a camera with an empty POINTS2D line (annex_posed_only exports, keep_all models) shifted the pairing and every second such camera vanished from the frustum count. Keep blank lines, resync on a blank pose slot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… drag-and-drop ready New results/final_reconstruction folder emitted at the end of every verified run (flag export_final_reconstruction, default on): the post-retri scene (root merged fallback), angle-filtered per min_triangulation_angle_deg, tracks colorized by sampling the source image at each track's first measurement (one image in memory at a time), and the prior-backed annex cameras riding along as posed-only frustums — no local post-processing needed before viz/analysis. Pure output: runs last, deep-copies tracks (colorize mutates r/g/b in place), never touches the solve; non-fatal on failure. Adversarially reviewed (3 lenses); coordinate/resolution/RGB conventions verified against utils.images.get_average_point_color and the loader keypoint frame. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… radius guard Roman Forum exposed the failure mode: a child annex carried through a Sim3 fit solved on few / near-collinear shared cameras gets a garbage scale and the whole block explodes numerically (RF annex tier: 537/537 matched cams >20m, coordinates up to 1e90; the core was untouched — inertness held). Carry now requires >=4 shared anchors, non-degenerate anchor geometry (second principal extent >=5% of the first), and the solved transform to reproduce its anchors (median residual <=25% of the merged camera spread); otherwise the child's annex is dropped with a log. The export additionally drops any annex camera beyond 15x the core's robust radius (or non-finite). Bounded failure by design: a dropped annex block is absent, never exploded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-seat Plain Similarity3.Align is least-squares and gets poisoned by stray cameras with extreme coordinates in either pose map — RF displaced its entire 627-camera annex block coherently through one bad root->final fSr. Both annex fits now use sim3_from_Pose3_maps_robust; the anchor/residual/radius guards from 08b1d24 remain as backstops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h length quadratically Alamo's 63-deep partition tree exceeded PATH_MAX just creating output dirs (each level repeats the full accumulated cluster label). Levels <= 8 keep the classic C_i_j_... names (existing drops and tooling unchanged); deeper levels use sibling-unique C<depth>_<idx> components — depth-63 total path drops from ~5,000 chars to ~500. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ne radius Method-level policy (part of the trust doctrine): a camera the model cannot place within its own extent is not a reportable pose. Applied to the retriangulated export (and, via it, final_reconstruction) — cameras beyond 15x the 95th-percentile radius or at non-finite coordinates are removed along with their track measurements. RF's raw combined export carried a single ~1,400km core stray that alone dragged the mean position error from 9.6m to 1,151m; with the policy, our means sit ~10m on all three 1DSfM scenes vs GLOMAP's 28-594m unbounded tails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… bar width - Frustum scale is now slider-controlled (Cam, 0.004-0.06 x viewRange, default 0.015 vs the old fixed 0.03) with in-place line-system rebuild. - The centered control bar sized itself to the longest run path via runSelect, extending past both screen edges and hiding the point-size slider; cap the bar at viewport width, ellipsize runSelect, allow wrap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…redicted focals The loader substitutes a 1.2x maxdim heuristic when EXIF is missing, and the exif passthrough was pinning that at sigma=5px like a real EXIF focal — 19-40% of 1DSfM cameras per scene (RF init-level: 281/1,486), with the heuristic ~17% off on average (the BM lesson per camera: pinned focal error becomes center warp). EXIF-less cameras are now omitted from the global calibration dict; the cluster build's existing precedence then serves the geometry model's predicted focal (VGGT-Omega ~1-2% median error). Trust EXIF where it exists, the learned prior where it doesn't. Cache caution: calibration content is not in the cluster cache keys — use on fresh scenes (Piazza del Popolo, Yorkminster) or clear cluster caches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…llback) 2125833 made the global calibration dict sparse (EXIF-less cameras omitted), but both BA-input builders did a hard refined_intrinsics[idx] lookup on every cluster camera — KeyError on the first EXIF-less camera. Per-camera fallback now takes the documented path: rescale the model's predicted intrinsics, same as when no refinement dict is supplied. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cameras Frustums were uniformly red, which made GLOMAP's (correct) camera clusters read as flagged tiers and left our core/annex indistinguishable. Cameras are now tiered by observation count: cores and untiered models render neutral slate; zero-observation (annex) cameras render red — so red only appears where the model itself declares low evidence, matching the paper's figures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The paper settled on red-equals-camera as the convention; tier commentary was dropped from captions, so two-tone rendering caused cross-figure inconsistency. hasObs stays parsed for future use. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just making a Draft PR so it's easier to look at all the experimental changes in one place, and I can slowly bring in just the right code for master