LucidSim: generated images to train a robot

ArquivoArchive

CategoriasCategories

Posts recentesRecent Posts

TagsTags

4D reconstruction AI and the brain AI ethics AI infrastructure AI transparency BEDLAM bias ChatGPT class imbalance computer vision dataset augmentation depth fundamentals generative models ghost AI Infinigen introduction long-tail recognition MIT study monocular video object removal opinion paper analysis pre-training procedural generation quality inspection real-data limit representation learning representativeness robotics semantic segmentation sim-to-real SMPL SURREAL synthetic data synthetic humans synthetic images tech giants video

Redes sociaisSocial Links

Bileki

Atualizado em Updated

PAPER ANALYSIS

MIT CSAIL’s LucidSim (CoRL 2024) trained a quadruped robot in simulation only, using AI-generated images. See the results, real and simulated.

LucidSim (Yu et al., MIT CSAIL, CoRL 2024) trains a quadruped robot for "visual parkour" in simulation, with no real training images of the robot. What makes the work interesting is the hybrid recipe: physics comes from a simulator, and each image’s appearance from a generative model. The authors themselves call it a preliminary proof of concept.

What LucidSim does

The task: a robot with a single color camera climbs stairs, jumps hurdles, and chases a ball or a cone. The pipeline has four parts:

  • Physics and geometry: the MuJoCo simulator runs the scene and outputs a depth image and semantic masks at every frame. The terrain is simple, hand-designed and inherited from earlier work.
  • Image: a diffusion model (Stable Diffusion XL Turbo), guided by ControlNet with the depth and masks, generates the appearance from text. The prompts are generated in bulk by an LLM (about a thousand per task), varying weather, time of day and setting.
  • Video: only the first frame of each sequence is generated. The next six are obtained by warping it with optical flow computed from the geometry, which speeds up generation by about 6.5x.
  • Learning: a teacher with privileged access to the terrain height map labels the actions. The visual policy learns first by imitation and then in DAgger cycles (three iterations), with data collected by the policy itself.

What the results show

  • In the evaluation simulator, built from scanned real scenes (Gaussian Splatting), 50 runs per scene and three scenes per task: on hurdles, LucidSim reached 90.7%, 93.5% and 96.5% of goals, against 56.5%, 52.5% and 44.0% for domain randomization. On stairs the result was mixed: 87.0% versus 95.5% on bricks, a tie on concrete (81.0% and 81.5%), and 83.7% versus 71.7% on marble. In chase-soccer, randomization won in one scene (97.3% versus 88.7%).
  • On the real robot (Unitree Go1, budget webcam, onboard inference), with few trials per task: cone 100% (20 trials), soccer 85% (20), dark hurdle 86.7% (15), light hurdle 73.3% (15), stairs 100% (10). Domain randomization scored 70%, 35%, 26.7%, 40% and 50% on the same tasks.
  • Against a depth baseline, the real-world result was mixed: the baseline scored 80% (cone) and 65% (soccer), below LucidSim; 86.7% and 80% on hurdles, tying on the dark one and ahead on the light one; and 100% on stairs, a tie. The authors describe the results as comparable.
  • What weighed most was training on the policy’s own data. Scaling only the teacher’s data saturated quickly, and the DAgger cycles brought most of the performance.

The authors say the robot trained with LucidSim recognizes a black-and-white soccer ball and also balls of other colors. They read this as an effect of the diversity of the generated images, an explanation from the paper and not an isolated test.

The limit: hybrid, small and preliminary

  • “No real data” holds for the policy. The generators (diffusion, ControlNet with estimated depth maps) were trained on real images. What the paper claims is that the policy never saw real images of the robot.
  • Few real-world trials. Between 10 and 20 per task, with no confidence intervals. The 100% results come from 10 and 20 trials.
  • The gain depends on the comparison. Against domain randomization it’s clear on hurdles and smaller on stairs. Against the depth baseline it doesn’t show on real hurdles and stairs (tie or below).
  • Fixed geometry. The authors left terrain variation out of scope, and strong geometric conditioning reduces image detail, a trade-off they describe.
  • Short memory. The policy sees seven frames (140 ms), and the authors note that jumping wide gaps would need about 400 ms.

What changes with a controlled scene

This section is our reading. In LucidSim, appearance comes from a generator and geometry from the simulator, two systems that ControlNet tries to align. In a declared 3D scene, the image and the geometric label come out of the same scene, so there’s no divergence between them by construction. In exchange, appearance depends on what was modeled, and the variety a generator gives for free has to be built: materials, lighting, backgrounds. It also isn’t a direct substitute for tasks where the policy needs physics, such as contact and friction, where the physics simulator remains the central piece.

Geo4D follows the reasoning of geometric labels coming from the scene, on another task, and Infinigen shows one way to create scene variety by rules.

What to take to your project

  • Separate appearance and physics when choosing a tool. If the task depends on both, check they’re aligned or measure the divergence.
  • Say which baseline you compare against. Domain randomization, depth input and real data are different questions, and LucidSim’s results change with each.
  • Test early on the real hardware, with the trial count in view. Ten trials show where the simulation fails, but they don’t give you a success rate.

Dealing with a robot that has to work outside the lab? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

,
, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *