Geo4D: 4D reconstruction trained only on synthetic

ArquivoArchive

CategoriasCategories

Posts recentesRecent Posts

TagsTags

4D reconstruction AI and the brain AI ethics AI infrastructure AI transparency BEDLAM bias ChatGPT class imbalance computer vision dataset augmentation depth fundamentals generative models ghost AI Infinigen introduction long-tail recognition MIT study monocular video object removal opinion paper analysis pre-training procedural generation quality inspection real-data limit representation learning representativeness robotics semantic segmentation sim-to-real SMPL SURREAL synthetic data synthetic humans synthetic images tech giants video

Redes sociaisSocial Links

Bileki

Atualizado em Updated

PAPER ANALYSIS

Geo4D (ICCV 2025) reconstructs 4D scenes from monocular video, trained only on synthetic data. See the numbers on real video and where it struggles.

Geo4D (Jiang, Zheng, Laina, Larlus and Vedaldi, Oxford and Naver Labs Europe; ICCV 2025) reconstructs the 3D geometry of a moving scene from single-camera video, and it is trained only on synthetic data. Read it with the right caveat: the paper shows a specific result on specific benchmarks, not a rule that synthetic beats real.

What Geo4D does

It starts from a pre-trained video diffusion model (DynamiCrafter) and adapts it to predict geometry instead of pixels. For each 16-frame clip, the model predicts three complementary representations: 3D point maps, disparity maps (relative depth) and camera ray maps. A multi-modal alignment algorithm fuses the three, and a sliding window (stride 4 in the main result) stitches the clips into one coherent reconstruction of the whole video.

The adaptation uses five synthetic datasets: BEDLAM, TartanAir, PointOdyssey, Spring and Virtual KITTI 2, about 1.6 million frames in total (TartanAir has 1,000K, BEDLAM 380K). Training took about one week on 4 NVIDIA H100 GPUs.

What the results show

On video depth estimation (Table 1, MonST3R protocol, one scale per video), Geo4D came out ahead of the baselines on all three benchmarks, two of them real video:

  • KITTI (driving, real): Abs Rel 0.086 versus 0.104 for DepthCrafter and MonST3R; δ<1.25 of 93.7% versus 89.6% and 89.5%.
  • Bonn (dynamic indoor, real): Abs Rel 0.059 versus 0.071 for DepthCrafter and 0.063 for MonST3R. On δ<1.25 it ties DepthCrafter at 97.2%.
  • Sintel (synthetic): Abs Rel 0.205 versus 0.270 for DepthCrafter. The paper reports 24.0% lower on Sintel and 17.3% lower on KITTI than DepthCrafter, which starts from the same video generator.

On camera pose (Table 2) the result is mixed, and the paper itself calls it comparable: rotation (RPE-R) is the best on both sets, Sintel (0.547 versus 0.615 for CasualSAM) and TUM-dynamics (0.635 versus 1.217 for MonST3R). Translation error (ATE and RPE-T) is worse than MonST3R and CasualSAM on both; on Sintel, for example, ATE is 0.185 versus 0.108 for MonST3R.

The ablation (Table 3, on Sintel) shows that training with all three modalities beats point maps alone, and that fusing the three at inference beats using any single one.

The limit

  • “Only synthetic” applies to the 4D adaptation. The starting video generator was pre-trained on large-scale real video, and the paper credits it for much of the generalization. The experiment of training the same model on real data isn’t in the paper.
  • Scope: depth and pose in video, on Sintel, Bonn, KITTI and TUM-dynamics, with 50 to 110 frames per sequence for depth and 14 dynamic Sintel sequences for pose. Detection, segmentation and other tasks are out of scope.
  • The authors name three limitations: the point-map encoder/decoder is still not fully accurate and caps quality; the method struggles with large focal-length changes and extreme camera motion, likely because the training data has almost no focal variation; and it only takes monocular video.
  • Inference cost: 1.89 s per frame at stride 4 (MonST3R takes 2.41 s under their setting).

What changes with a controlled scene

This section is our reading, not the paper’s. On geometric tasks, the label (depth, pose, correspondence) comes straight from the scene, free of sensor noise, which is what makes training at scale feasible without collecting annotated video. The limitation the authors cite also points to what a controlled scene can fix: if focal changes and extreme motion are missing, they can be generated on purpose. The caveat is that simulation embeds bias too: a result on Sintel, which is synthetic, is no substitute for testing on your application’s real video.

What to take to your project

  • For dense geometric labels, synthetic is a serious starting point, but here it adds to a generator pre-trained on real video. It is not “only synthetic” end to end.
  • Check what the benchmark measures. Geo4D’s gain is in depth and rotation; camera translation trailed other methods.
  • Map what’s missing in your training data (focal length, motion, scene type) and use the controlled scene to cover exactly that, measuring on a small real set.

Dealing with a depth, pose or motion problem? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

,
, , ,
  1. JiliPH

    I enjoyed reading this article. Thanks for sharing your insights.

Leave a Reply

Your email address will not be published. Required fields are marked *