Semantic segmentation with synthetic images (2017)

ArquivoArchive

CategoriasCategories

Posts recentesRecent Posts

TagsTags

4D reconstruction AI and the brain AI ethics AI infrastructure AI transparency BEDLAM bias ChatGPT class imbalance computer vision dataset augmentation depth fundamentals generative models ghost AI Infinigen introduction long-tail recognition MIT study monocular video object removal opinion paper analysis pre-training procedural generation quality inspection real-data limit representation learning representativeness robotics semantic segmentation sim-to-real SMPL SURREAL synthetic data synthetic humans synthetic images tech giants video

Redes sociaisSocial Links

Bileki

Atualizado em Updated

PAPER ANALYSIS

In 2017, 100 Blender images per class raised PASCAL VOC segmentation from 52.80% to 55.47% mIoU. What that proves and what it leaves open to question.

In 2017, Goyal, Rajpura, Bojinov and Hegde showed that adding Blender-rendered images to a semantic segmentation dataset improved results on PASCAL VOC 2012. The gain was modest and uneven across classes, and the experiment is small. That’s exactly why it’s a good case to read carefully.

What the paper did

The goal was to reduce the cost of pixel-level annotation. The authors started from two ingredients:

  • Real images with weak annotation: 10 thousand PASCAL VOC images with bounding boxes, converted to approximate masks by CRF (a refinement method). This is the “Weak(10k)” set.
  • Synthetic images: 2 thousand images rendered in Blender (Cycles engine), 100 per class, with ShapeNet 3D models over SUN database backgrounds. Objects enter with random scale, rotation and position under directional lighting. The mask comes out of the scene itself.

The model is FCN-8s, pretrained on ImageNet. It was first fine-tuned on the 10 thousand weakly annotated images (the baseline). Then only a few final layers were fine-tuned on the synthetic images, at a lower learning rate, to mostly capture object shape. Evaluation uses the PASCAL VOC 2012 validation set (1,449 images).

What the results show

  • Mean: the weak-annotation baseline scored 52.80% mIoU. With the 2 thousand synthetic images, 55.47% (+2.67 points).
  • By class, the gain is uneven. Car rose from 51.15% to 61.83% and bottle from 47.25% to 55.52%, plus potted plant, sofa and dining table. TV fell from 27.99% to 22.06% and person from 66.01% to 63.43%. According to the authors, classes with few available 3D models (bicycle, dog, person, TV) did worse.
  • Training with one class only can help or hurt. With 100 synthetic car images, the mean went to 53.38% and car to 69.31%. With 100 aeroplane images, the mean fell to 46.42%, below the baseline.
  • Combining with strong annotation: with the 1.5 thousand pixel-annotated images plus the weak and synthetic ones, the result was 58.27%. Strong annotation plus synthetic, without the weak set, gave 5.08%, which the authors attribute to the negative effect of non-photorealistic images on the strong annotation.
  • For reference, the authors cite 62.2% mIoU on the PASCAL test set for FCN-8s trained on the augmented 8.5 thousand-image strong-annotation dataset, and consider 55.47% reasonable with 12 thousand images in total.

The limit: a small experiment

  • The baseline isn’t “just real data.” The starting point already uses approximate labels generated by CRF. The gain is over that baseline, and the CRF mask edges are part of what the synthetic images helped clean up, according to the authors.
  • A 2015 architecture, one dataset, one training run. The paper reports no repeats with different seeds and no confidence interval, and the difference in the mean is 2.67 points.
  • Per-class results are irregular. Three of the 21 classes fell (TV, person and, by 0.18 points, dog). The other 18 rose, several by less than two points, and the biggest gains are in a few classes (car, bottle, potted plant, motorbike).
  • Evaluation is on the validation set, the same one used to compare the variants.
  • The authors acknowledge the limits of realism: the images aren’t photorealistic, and lack rich textures, relevant backgrounds or multiple classes per image. They suggest richer models, relevant backgrounds, relative scale and occlusions as future work.

What changes with a controlled scene

This section is our reading. The paper already uses what makes a controlled scene useful: the mask is born with the image, with no later estimation. What can be done differently today:

  • Control coverage per class, with the proportion declared, instead of 100 equal images for every class. The per-class results show why this matters.
  • Make the choice of 3D model and background explicit, since the authors attribute the worst results to classes with few models.
  • Record the parameters, so the same set can be regenerated and compared.

None of this guarantees a gain. The 2017 result shows the gain depends on the class, the model and the baseline, and only a test on your real set tells whether it shows up in your case.

The same reading care applies to the MIT study on GAN-generated data, which doesn’t compare simulation and generative AI, as it is often said to.

What to take to your project

  • Measure by class, not just the mean. Here the mean rose 2.67 points, while person and TV got worse.
  • Test synthetic data per class before adding everything. A hundred images of one class raised the mean in one case and dropped it in another.
  • Be careful when combining with strong annotation. The paper shows one combination with a very poor result. Test the data mix in your pipeline before assuming that adding helps.

Want to test dataset augmentation with synthetic images in your case? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

,
, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *