VidMap

Exploiting Temporal Structure for Video-Based Structure-from-Motion

Zador Pataki · Paul-Edouard Sarlin · Marc Pollefeys

ECCV 2026

VidMap is an offline Structure-from-Motion system for video. It combines temporal tracks, loop closures, metric depth, and global optimization to estimate camera poses, camera intrinsics, and a sparse 3D map.

Quick Start

🔧 Setup — Install dependencies and prepare the environment.
🚀 Run VidMap — Reconstruct a video or image sequence.
📷 Camera rigs, panoramas, and fisheye — Configure your cameras.
👀 Visualization — Explore your reconstruction in the browser or Rerun.
🛠️ Pipeline Configurations — Customize the frontend and mapping settings.
⚡ Speed up VidMap — Find settings to make extraction and mapping faster.
📈 Benchmarks — Prepare datasets and evaluate reconstruction accuracy.

Setup

We provide the Python package vidmap. From a clean Python environment, clone VidMap and its pinned model dependencies:

git clone --recursive https://github.com/cvg/vidmap.git && cd vidmap

Build and install COLMAP and its PyCOLMAP bindings from source. Last tested with: COLMAP b1972bc2. For GPU mapper acceleration, build them against Ceres 2.3 or newer with CUDA and cuDSS instead (see how to enable).

Then install PyTorch, xFormers, and VidMap:

pip install -e .

The frontend was last tested on Linux x86-64 with Python 3.10, PyTorch 2.14.0, TorchVision 0.29.0 and xFormers 0.0.35 on NVIDIA GPUs. Fast loading of cached compiled models requires PyTorch 2.10+.

The first frontend run automatically downloads approximately 9 GB of model checkpoints and caches approximately 1.2 GB of compiled models.

Optional visualization setup. Install Rerun:

pip install rerun-sdk

Execution

The repository includes a silent example video. Run the frontend to build its mapper inputs, then run mapping:

INPUT_DATA=./assets/example_video.mp4
OUTPUT_DIR=./example_output/
MAPPER_INPUTS="$OUTPUT_DIR/mapper_inputs"

python -m vidmap.frontend "$INPUT_DATA" \
  --output "$OUTPUT_DIR"

python -m vidmap.map \
  --mapper-inputs "$MAPPER_INPUTS" \
  --output "$OUTPUT_DIR"

Alternatively, run frontend and mapping together:

python -m vidmap.run \
  --input_data "$INPUT_DATA" \
  --output "$OUTPUT_DIR"

Alternatively, --input_data may point to a directory of extracted frames.

Both mapping commands write the final COLMAP model under $OUTPUT_DIR/rec. Resolved frontend_config.yaml and mapping_config.yaml are written directly under $OUTPUT_DIR.

See files and folders for image layouts, frame associations, configuration files and run artifacts.

Camera rigs, panoramas, and fisheye

Input	Uncalibrated mono	Calibrated mono	Uncalibrated rig	Calibrated rig
Pinhole	✓	✓	✗	✓
Distorted perspective	✓	✓²	✗	✓²
Fisheye	✓¹	✓²	✗	✓²
Stitched 360° panorama	N/A	✓	N/A	✗

Distorted perspective models: SIMPLE_RADIAL, RADIAL, OPENCV, FULL_OPENCV, SIMPLE_DIVISION, DIVISION. See COLMAP camera models for model definitions and parameter order.

Native OPENCV_FISHEYE, forward-facing rays only.
Undistorted to pinhole before mapping. Calibrated rigs may mix camera models.

Visualization

Browser viewer

Open the vidmap-viewer in a Chromium-based browser and choose a run folder containing rec/ and mapper_inputs/. The viewer requires internet access.

[Embed a reconstruction - click to expand]

python -m vidmap.visualization.html --run-dir "$OUTPUT_DIR"

This embeds the reconstruction in $OUTPUT_DIR/vidmap-viewer-embedded.html, so it opens without selecting a run folder.

Rerun playback

To render optimization playback, record solver states during mapping:

python -m vidmap.map \
  --mapper-inputs "$MAPPER_INPUTS" \
  --output "$OUTPUT_DIR" \
  --save-playback-trace

This writes the trace to $OUTPUT_DIR/playback_trace/.

Create solver-playback and flythrough recordings. Local images, frame timing, and missing ground truth are detected automatically:

VISUALS=./example_visuals/

python -m vidmap.visualization.rerun.playback "$OUTPUT_DIR" \
  --view isometric \
  --output "$VISUALS/solver-playback.rrd"

python -m vidmap.visualization.rerun.flythrough "$OUTPUT_DIR" \
  --reconstruction final \
  --view follow \
  --output "$VISUALS/flythrough.rrd"

[Flythroughs with lifted depth maps - click to expand]

Depth lift replaces the current sparse highlight with the RGB-colored full depth prediction. Retain those predictions during reconstruction with --cache-depth-maps, then create an RRD:

python -m vidmap.visualization.rerun.flythrough "$OUTPUT_DIR" \
  --depth-lift \
  --output "$VISUALS/depth-flythrough.rrd"

If the first frontend run did not cache full depth maps, backfill them without repeating tracking or verification:

python -m vidmap.frontend "$INPUT_DATA" \
  --output "$OUTPUT_DIR" \
  --mapper-inputs "$OUTPUT_DIR/mapper_inputs" \
  --cache-depth-maps

Open the recordings with Rerun:

rerun "$VISUALS/solver-playback.rrd"
rerun "$VISUALS/flythrough.rrd"

Don't forget to try VidMap on your favorite videos from the web.

Configuration

VidMap extends COLMAP's global mapping pipeline, for which we provide easily adjustable hyperparameters via configs.

VidMap uses Hydra and OmegaConf for fine-grained configuration with typed defaults. Frontend configs control tracks, keyframes, depth, and calibration, while mapping configs control global positioning and bundle adjustment. Frontend settings follow the pipeline itself: keyframe matching, selection, and features are grouped together; track matching and propagation form a separate stage.

Print all available frontend and mapping defaults:

from vidmap.configuration import FrontendConfig, MappingConfig, summarize_cfg

print(summarize_cfg(FrontendConfig()))
print(summarize_cfg(MappingConfig()))

[Custom configuration - click to expand]

Each YAML file overrides only the relevant defaults and can inherit another configuration with defaults:.

For example, define a compatible frontend and mapping pair:

# vidmap/configs/frontend/uncalib/custom.yaml
defaults:
  - base

keyframes:
  selection:
    max_normalized_keypoint_drift: 0.09

# vidmap/configs/mapping/uncalib/custom.yaml
defaults:
  - base

mapper:
  gp:
    first_pass:
      loss_lc_geometry:
        weight: 0.3

Select custom configs and adjust either stage inline with Hydra overrides:

python -m vidmap.run \
  --input_data "$INPUT_DATA" \
  --output "$OUTPUT_DIR" \
  --frontend-conf uncalib/custom \
  --mapping-conf uncalib/custom \
  frontend.keyframes.selection.max_normalized_keypoint_drift=0.08 \
  mapping.mapper.gp.first_pass.loss_lc_geometry.weight=0.4

[GPU acceleration - click to expand]

Enable GPU acceleration for global positioning and bundle adjustment (requires Ceres 2.3 or newer with GPU support; see Setup):

python -m vidmap.run \
  --input_data "$INPUT_DATA" \
  --output "$OUTPUT_DIR" \
  mapping.mapper.gp.solver_backend.use_cuda=true \
  mapping.mapper.ba.solver_backend.use_cuda=true

Recommended configurations

default: The default reconstruction pipeline, based on the method described in the paper.
smooth_trajectory: Experimental pipeline. Adds keyframe-coordinate trajectory smoothing to the default pipeline for improved robustness on long sequences, as explored on external datasets.

Speed up VidMap

Reducing the number of tracked points, using coarse-only keyframing, and using fewer propagation hops can speed up VidMap. Running global positioning on CUDA reduced total mapping time by about 13% on a tested LaMAR HGE sequence. The benefit of GPU acceleration for bundle adjustment is less clear.

python -m vidmap.run \
  --input_data "$INPUT_DATA" \
  --output "$OUTPUT_DIR" \
  frontend.tracks.propagation.max_kps=1000 \
  frontend.keyframes.matching.coarse_only=true \
  'frontend.tracks.propagation.multiflow_hops=[1,2,4,8]' \
  mapping.mapper.gp.solver_backend.use_cuda=true \
  mapping.mapper.ba.solver_backend.use_cuda=true

The table below provides insight into the accuracy–runtime tradeoffs. Extraction savings approximately exclude startup by subtracting the recorded first calls.

Setting	Stage measured	Time saved	Mean accuracy drop
Hops `[1,2,4,8]`	Extraction	~9.9%	~0%
Hops `[1,2,4]`	Extraction	~15.3%	0.385%
Hops `[1,2]`	Extraction	~21.3%	1.563%
Coarse keyframing	Extraction	~9.0%	0.053%
1,000 points, balanced	Mapping	~40%	0.632%
500 points, balanced	Mapping	~66%	2.067%

Benchmarks

Prepare data

By default, prepared data, caches, and experiments live under ./local/{datasets,cache,experiments}/<dataset>/, relative to the directory where the command is run. To use other paths, create ./vidmap-paths.toml or set VIDMAP_<DATASET>_{DATA,CACHE,EXP,TESTSETS}_DIR.

Then select and prepare a benchmark dataset:

DATASET=lamar  # or: euroc, eth3d_slam, crocodl
python -m vidmap.datasets.prepare --dataset "$DATASET"

This downloads and validates the supported evaluation inventory, then converts it into the canonical images/, rec/, and testset layout. LaMAR intentionally prepares only the images referenced by its benchmark testset: sessions and images absent from that testset are not retained. Downloads resume after interruption and completed sequences are reused. Validated source archives are retained by default; pass --delete-downloads to remove them after successful preparation.

[Expected prepared dataset layouts - click to expand]

Storage for the supported evaluation inventories:

Dataset	Download	Prepared
LaMAR	45.2 GB	24.0 GB
EuRoC	24.7 GB	4.7 GB
ETH3D-SLAM	18.6 GB	18.6 GB
CroCoDL	15.0 GB	14.0 GB

Download sizes are the checksum-pinned source archives; prepared sizes cover the canonical dataset after conversion. LaMAR archives are session-level, so the download includes unused frames inside each required session ZIP; only the 94,489 testset images are retained. Retaining downloads requires roughly the sum of both columns.

After preparation, the selected dataset is available under ./local/datasets/<dataset>/, with its benchmark definitions under ./local/testsets/<dataset>/:

local/
├── datasets/
│   ├── lamar/
│   │   └── scene: <CAB|HGE|LIN>/
│   │       ├── images/
│   │       │   └── cam0/<capture>/<timestamp>.jpg  # e.g. ios_2022-06-30_15.55.53/130156492755.jpg
│   │       ├── rec/
│   │       └── preparation.json
│   ├── euroc/
│   │   └── scene: <machine_hall-MH_01_easy|machine_hall-MH_02_easy|...>/
│   │       ├── images/
│   │       │   └── cam0/<timestamp>.png  # e.g. 1403636580863555584.png
│   │       ├── rec/
│   │       └── preparation.json
│   ├── eth3d_slam/
│   │   └── scene: <plant_1|plant_2|...>/
│   │       ├── images/
│   │       │   └── cam0/<timestamp>.<ext>  # e.g. 5865.976533.png
│   │       ├── rec/
│   │       ├── unposed_images.json
│   │       └── preparation.json
│   └── crocodl/  # Unlike LaMAR, CroCoDL has GT per session, not per scene.
│       └── <location>/  # e.g. ARCHE_D2
│           └── <session>/  # e.g. ios_2023-07-12_20.46.14_000
│               ├── images/
│               │   └── cam0/<timestamp>.jpg  # e.g. 4971459477.jpg
│               ├── rec/
│               └── metadata.json
└── testsets/
    ├── lamar/<scene>/<mode>.yaml  # e.g. HGE/sample.yaml
    ├── euroc/<sequence>/all.yaml  # e.g. machine_hall-MH_01_easy/all.yaml
    ├── eth3d_slam/<sequence>/all.yaml  # e.g. plant_1/all.yaml
    └── crocodl/<ios-location>/<mode>.yaml  # e.g. ios-ARCHE_D2/sample-100.yaml

Run and evaluate

# Use uncalib/base for datasets other than LaMAR.
FRONTEND_CONF=uncalib/gtkf-base
MAPPING_CONF=uncalib/base
# Omit --name below to derive the output name from the config pair.
OUTPUT_NAME=default

python -m vidmap.run_for_benchmark \
  --dataset "$DATASET" \
  --frontend-conf "$FRONTEND_CONF" \
  --mapping-conf "$MAPPING_CONF" \
  --name "$OUTPUT_NAME"

Inline Hydra overrides use the same frontend.<field>=<value> and mapping.<field>=<value> prefixes as vidmap.run.

The command reconstructs and evaluates every selected target, then prints the final aggregate report across all completed trajectories. Add --run-only to skip only this final report; per-trajectory evaluation still runs.

[LaMAR - click to expand]

Evaluate the default sample benchmark, a representative subset of each ground-truth trajectory as outlined in the paper. LaMAR selects wate_auc by default.

python -m vidmap.run_for_benchmark \
  --dataset lamar \
  --frontend-conf uncalib/gtkf-base \
  --mapping-conf uncalib/base

[EuRoC - click to expand]

Evaluate the complete EuRoC benchmark. EuRoC selects pose_auc by default.

python -m vidmap.run_for_benchmark \
  --dataset euroc \
  --frontend-conf uncalib/base \
  --mapping-conf uncalib/base

[ETH3D-SLAM - click to expand]

Evaluate the complete ETH3D-SLAM benchmark. ETH3D-SLAM selects pose_auc by default.

python -m vidmap.run_for_benchmark \
  --dataset eth3d_slam \
  --frontend-conf uncalib/base \
  --mapping-conf uncalib/base

[CroCoDL - click to expand]

Evaluate the complete CroCoDL benchmark. CroCoDL selects wate_auc by default.

python -m vidmap.run_for_benchmark \
  --dataset crocodl \
  --frontend-conf uncalib/base \
  --mapping-conf uncalib/base

[Results - click to expand]

WATE-AUC with late principal point refinement, at 5% error thresholds (%, higher is better). Errors are pooled within each location, then eligible locations are averaged equally:

Dataset	10 m @ 0.5 m	25 m @ 1.25 m	50 m @ 2.5 m	100 m @ 5 m	Full @ 5%
LaMAR (sample-300, 50 sequences)	92.26	92.66	91.16	90.77	90.23
CroCoDL (64 sequences)	95.10	96.55	97.22	97.56	97.18

Pose AUC, averaged equally across sequences (%, higher is better):

Dataset	5 cm	10 cm	50 cm	1 m	10 m
EuRoC (11 sequences)	28.86	60.02	91.96	95.98	99.60
ETH3D-SLAM (55 sequences)	63.96	74.59	89.19	93.25	98.91

[Optionally re-evaluate saved results - click to expand]

Recompute evaluation reports without rerunning reconstruction:

python -m vidmap.evaluate_results \
  --dataset "$DATASET" \
  --name "$OUTPUT_NAME"

BibTeX citation

If you use any ideas from the paper or code from this repository, please consider citing:

@inproceedings{pataki2026vidmap,
  author    = {Zador Pataki and
               Paul-Edouard Sarlin and
               Marc Pollefeys},
  title     = {{VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion}},
  booktitle = {ECCV},
  year      = {2026}
}

Acknowledgments

VidMap builds on COLMAP, MP-SfM, and hloc. For the video visualizations, dense matches were densely triangulated using ImLoc.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
docs		docs
extensions/colmap		extensions/colmap
tests		tests
third_party		third_party
vidmap		vidmap
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitmodules		.gitmodules
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml

Navigation Menu

Repository navigation

About

Repository files navigation

VidMap

Exploiting Temporal Structure for Video-Based Structure-from-Motion

Quick Start

Setup

Execution

Camera rigs, panoramas, and fisheye

Visualization

Browser viewer

Rerun playback

Configuration

Recommended configurations

Speed up VidMap

Benchmarks

Prepare data

Run and evaluate

BibTeX citation

Acknowledgments

About

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

VidMap

Exploiting Temporal Structure for Video-Based Structure-from-Motion

Quick Start

Setup

Execution

Camera rigs, panoramas, and fisheye

Visualization

Browser viewer

Rerun playback

Configuration

Recommended configurations

Speed up VidMap

Benchmarks

Prepare data

Run and evaluate

BibTeX citation

Acknowledgments

About

Resources

Stars

Watchers

Forks

Contributors

Languages