PAPER ANALYSIS
An MIT study (ICLR 2022) trained only on GAN images: below real data on ImageNet, above it on cars. What the study does and doesn’t tell you.
In 2022, MIT researchers asked: if generative models are good enough, do we still need datasets? The question often gets read as "simulation versus generative AI," but the study doesn’t compare those. It compares learning from GAN-generated images against learning from real photos. That distinction changes what you can take from it.
What the study did
The paper (Jahanian, Puig, Tian and Isola, ICLR 2022) treats the generator as a black box with no access to the data it was trained on. The core idea: neighboring points in a generator’s latent space produce images of the same content in different "views." Those views serve as positive pairs for contrastive learning, in place of the usual pixel-space transformations. The authors tested two generators: BigBiGAN (ImageNet) and StyleGAN2 (LSUN cars, 893K real photos, FID 2.32). Representation quality was measured with a linear classifier on the learned features.
What the results show
- ImageNet (BigBiGAN, no labels): 43.90% top-1 accuracy with real data versus 42.58% with the best generated setup (Gaussian views plus pixel augmentations). Generated comes close but stays below.
- Class-conditional ImageNet (BigGAN, 1000-class setting): 50.84% real versus 49.25% for the best generated. In the smaller ImageNet100 setting the gap is wider: 81.18% real versus 70.16%.
- Cars (StyleGAN2): transfer to Stanford Cars goes from 39.68% (real car photos) to 49.79% (generated, Gaussian views plus augmentations). On ImageNet it’s a practical tie: 30.30% versus 30.06%.
- More samples help, but little: the gain grows sub-logarithmically with the number of generated images. Distance between views follows an inverted-U curve, peaking at standard deviation 0.2.
- Mixing: in the appendix (Figure 9, ImageNet100, unconditional generation), replacing real photos with generated ones lowers accuracy as real images decrease: from about 67% with real data only to about 58% with generated data only. The authors point to 5% generated as a favorable point, where the curve is practically flat.
The authors summarize it this way: representations learned from large quantities of generated data rival, and sometimes outperform, those learned from real data. The numbers show what "sometimes" means in practice: the case that wins is cars, where the generator reproduces real photos almost indistinguishably (FID 2.32).
The limit: the generator learned from the real
- The origin is still real. StyleGAN2 was trained on 893K real car photos. The study assumes the researcher has no access to that data, not that it doesn’t exist. “Trained only on generated data” describes the last step, not the whole chain.
- One task, few generators. It’s representation learning evaluated by linear classification, on two 2019–2020 generators. It isn’t a detection or segmentation result, or a production pipeline, and today’s generators are different.
- Bias and privacy, per the authors. The ethics statement lists as risks that generators can reveal the data they were trained on and amplify biases, and that this calls for auditing.
- What the study didn’t test: images rendered from a scene. The text itself cites simulation and rendering as the traditional source of synthetic data and says this work looks at something else. Any comparison between the two goes beyond what the paper measured.
What changes with a controlled scene
This is our reading, not a result from the paper. In a declared 3D scene, the label (mask, depth, pose, box) comes out of the same scene that renders the image, rather than depending on a model that estimates it. What you create also doesn’t start from someone else’s real corpus: it starts from a list of what the project needs, including rare cases such as a pedestrian in an unusual place. The cost is different. A controlled scene requires modeling and a bridge to real appearance (the domain gap), and it can carry the bias of whoever designed it. For tasks where the visual variety of photos is what matters and the label is just the image class, as in the MIT study, a generator does the job well.
For the reasoning seen in a measured result, our Geo4D analysis covers a model trained only on synthetic scenes and applied to real video, and the 2017 segmentation paper shows the gain from augmenting a real dataset with synthetic images.
What to take to your project
- Read the method before the headline. “Only synthetic data” can mean a generator trained on millions of real photos, and the result depends on how faithful it is to that distribution.
- Ask where the label comes from. If the task needs exact geometric labels, a declared scene delivers them; for a class per image, a class-conditional generator can too.
- Measure in your domain. A result on ImageNet and LSUN cars doesn’t transfer by itself to your camera, your defect or your environment.
Dealing with a task that depends on exact labels? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.
Sources
- Jahanian, A.; Puig, X.; Tian, Y.; Isola, P. Generative Models as a Data Source for Multiview Representation Learning. ICLR 2022.
- MIT News. When it comes to AI, can we ditch the datasets?. March 15, 2022.
Leave a Reply