Infinigen: procedural generation of 3D worlds

PAPER ANALYSIS

Infinigen (Princeton, CVPR 2023) builds natural 3D scenes from rules alone, with no external assets. What the paper measured, what it didn’t, and the cost.

Infinigen, by Raistrick and colleagues (Princeton, CVPR 2023), is a procedural generator of 3D scenes of the natural world: everything, from shape to texture, comes from randomized mathematical rules, with no external assets. It’s an open and valuable resource. The useful question for anyone planning to use it is what the paper actually measured.

What Infinigen does

• Entirely procedural. Every asset is generated from scratch by randomized rules, with no third-party model or texture libraries. The authors contrast this with most synthetic datasets, which use static asset libraries.
• The natural world. Plants, animals, terrain, rocks, water, and phenomena such as fire, cloud, rain and snow. The choice is deliberate: the authors say existing synthetic datasets mostly cover urban, driving and indoor scenes.
• Real geometry. Details such as surface roughness are actual geometry, not texture tricks, which allows exact geometric labels.
• Labels from the scene. Depth, normals, occlusion boundaries, instance segmentation, 2D and 3D boxes and optical flow.
• Open. Built on Blender, BSD license. According to Table 2, there are 182 generators and about 1,070 interpretable parameters, a number the authors treat as a proxy for diversity and admit may be an overestimate, since the parameters aren’t all independent.

What the results show

The paper validates the data with a stereo test: 30 thousand Infinigen image pairs, training the RAFT-Stereo network from scratch, compared with the same network trained on other synthetic datasets.

• On Infinigen’s own scenes (400 independent pairs, no assets shared with training), the Infinigen-trained model had the lowest error (Bad 3.0 of 5.53%), against 6.50% for TartanAir, 7.84% for SceneFlow and 25.28% for InStereo2K. The authors warn that this only works as a measure if performance on Infinigen is a good proxy for the real world.
• On real photos of nature, the evaluation is qualitative: no real depth benchmark exists for natural scenes, and high-precision 3D scanning of a natural environment is impractical. The authors show disparity maps for images they photographed with a ZED 2 stereo camera and say the model trained only on Infinigen works well zero-shot, and better than those trained on other sets.
• On the Middlebury benchmark (indoor scenes full of human-made objects), the result is mixed, and the authors say so: 29.7% average error, behind SceneFlow (25.0%) and close to InStereo2K (29.5%). On the only natural scene in the set (Jadeplant), Infinigen did best: 35.2%, against 41.3% for SceneFlow and 43.3% for FallingThings.
• Cost of generating: on average, 3.5 wall-clock hours for one 1080p image pair, with about one hour of GPU and 24 GB of memory, on 2 Xeon CPUs and 1 GPU. More CPUs cut the time. The average complete scene has 16 million polygons.

The limit: one test, and the cost

• The validation is a single stereo experiment, and the paper’s main body has no limitations section. The caveats appear in captions and the appendix. The results don’t cover detection, segmentation or the other tasks cited as possible uses, and the gain measured on real photos is qualitative, with images chosen by the authors.
• A model trained only on nature does worse indoors. The authors themselves explain: natural objects have few planar, textureless surfaces, which Middlebury demands.
• Evaluating on Infinigen’s own scenes favors Infinigen. The result shows the distribution differs from those of other datasets, as the authors conclude, and that it’s a good complement, not a replacement.
• Simple ecological relations. The generator uses rules such as “cacti grow in deserts,” but complex relations, such as predator and prey or plants avoiding shade, aren’t modeled, according to the appendix.
• The hypothesis that the natural world alone suffices to train foundation models is a conjecture by the authors, untested in the paper.
• Cost. Hours per image pair is a real cost to plan for.

What changes with a controlled scene

This section is our reading. Infinigen solves diversity of natural scenes by rules. When the problem is a specific object of a part, a line or a camera, what matters is representing that object and the conditions it appears in. That’s solved with a scene built for the case, and procedural variety can come in as background or as part of the pipeline, as an open tool rather than a competitor.

The same goes for choosing what to measure. The paper makes clear that it wins where coverage was poor (natural scenes) and loses where the object is human-made. An industrial inspection dataset sits on the side where Infinigen, by construction, has no coverage.

Geo4D has a similar line of reasoning, geometric labels from the scene and synthetic-only training, on another task.

What to take to your project

• Check where the generator has coverage. A procedural generator of nature helps with nature, and the paper itself shows worse results in indoor settings.
• Check how the gain was measured. Here, on the generator’s own images and on real photos judged qualitatively. Measure in your domain, with a benchmark of your own.
• Put the cost of generating in the plan. Hours per high-resolution image change the size of the dataset you can produce.

---

Dealing with a specific object or environment that general generators don’t cover? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

• Raistrick, A. et al. Infinite Photorealistic Worlds using Procedural Generation. arXiv:2306.09310v2; CVPR 2023.
• Infinigen project: code, videos and pre-generated data.