ROSE: removing objects in video with 3D data

ArquivoArchive

CategoriasCategories

Posts recentesRecent Posts

TagsTags

4D reconstruction AI and the brain AI ethics AI infrastructure AI transparency BEDLAM bias ChatGPT class imbalance computer vision dataset augmentation depth fundamentals generative models ghost AI Infinigen introduction long-tail recognition MIT study monocular video object removal opinion paper analysis pre-training procedural generation quality inspection real-data limit representation learning representativeness robotics semantic segmentation sim-to-real SMPL SURREAL synthetic data synthetic humans synthetic images tech giants video

Redes sociaisSocial Links

Bileki

Atualizado em Updated

PAPER ANALYSIS

ROSE removes objects from video along with their shadows and reflections, trained on 16,678 rendered 3D pairs. See the numbers and what they don’t prove.

Removing an object from a video isn’t just erasing it. You also have to erase what it caused: shadow, reflection, light on the wall, translucency. ROSE (Remove Objects with Side Effects in Videos), by Miao et al., tackles this with data the real world almost never provides: the same video with and without the object. To get it, the authors rendered the pairs in 3D.

What ROSE does

The data. The authors used Unreal Engine 5.3. They started from 28 3D environments, which became 450 scenes, and rendered 90-frame videos at 1920×1080. For each video, the pipeline produces three aligned versions: with the object, without it, and the object mask. In the end, there are 16,678 video pairs for training. The mask comes from a shader in the engine itself, with no segmentation model.

The categories. Besides common objects, the paper covers five side effects: light source, mirror, reflection, shadow and translucency.

The model. ROSE is a Diffusion Transformer based on Wan2.1 (1.3 billion parameters). Two ideas set it apart:

  • Erasing with reference. Instead of cutting the object out and filling the hole, the model receives the whole video, so it can find the shadow and reflection that need to go.
  • Difference mask. An extra module learns to predict which regions change between the with-object and without-object pair, and this enters training as supervision.

What the results show

The numbers below compare ROSE with ProPainter and DiffuEraser, two of the five methods used as references. PSNR and SSIM: higher is better. LPIPS: lower is better.

BenchmarkMetricROSEProPainterDiffuEraser
Synthetic (60 videos)PSNR31.1227.2026.50
Synthetic (60 videos)LPIPS0.0770.0950.128
Real video (DAVIS, paired)PSNR31.3432.8129.97
Real video (DAVIS, paired)LPIPS0.0920.1220.128
  • On the synthetic benchmark, ROSE wins every category on PSNR, including the hard cases. On mirrors, for example, it’s 28.35 against DiffuEraser’s 22.12.
  • On real video, the picture is mixed. The authors conclude that ROSE “generalizes well” to real videos. On the paired table, though, ProPainter has the highest PSNR (32.81 vs 31.34), while ROSE leads on SSIM and LPIPS.
  • Without ground truth, results also split. In VBench evaluation on unpaired real videos, ROSE leads on background and subject consistency, but DiffuEraser has better image quality.
  • What mattered most in the design. In the ablation, passing the whole video as reference was the biggest gain (PSNR from 27.28 to 30.84 on the synthetic benchmark).

The limit: the benchmark is in the training’s family

  • The biggest advantage is on the test closest to training. The 60-video synthetic benchmark comes from the same pipeline as training. It’s where ROSE pulls furthest ahead, and where generalization is least proven.
  • The “real” test is assembled. The paired one on DAVIS is made by pasting an object into another video, with the original as ground truth. The authors justify the benchmark because existing ones lacked pairs with realistic physics and didn’t cover side effects. It’s a good test, but not the same as shadows and reflections filmed for real.
  • Data and architecture come together. The comparison methods weren’t trained on the same data, and ROSE starts from a recent base model. The paper doesn’t separate how much of the gain comes from synthetic data and how much from the architecture.
  • Limits the authors cite. Flicker under large motion, and inference time that grows with video length.

What changes with a controlled scene

ROSE shows well what simulation delivers:

  • The with/without pair is born in the scene. The same render, with and without the object, with the shadow computed from the light.
  • The mask is exact, produced by the engine, without going through a segmentation model.
  • Parameters stay recorded: light, material, trajectory. You can generate the case that was missing after the model fails.

The limit is what the paper shows: the final test must be on real video, where the gap appears. And 450 scenes are still 450 scenes. The method lies in deciding which effects to cover and how much variation the task needs. The same design (train on a synthetic scene, transfer to the real one) shows up in MIT CSAIL’s LucidSim, in robotics.

What to take to your project

  • List your task’s side effects. Does removing, adding or moving an object involve shadow, reflection or occlusion? If so, an object-only label isn’t enough.
  • Be wary of a test that resembles training. Gains on the synthetic benchmark don’t guarantee the same on your real video.
  • Compare on the same data. To know what synthetic brought, train your baseline with the same volume of data.

Dealing with side effects that are hard to annotate? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

,
, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *