Introduction to synthetic data generation

ArquivoArchive

CategoriasCategories

Posts recentesRecent Posts

TagsTags

4D reconstruction AI and the brain AI ethics AI infrastructure AI transparency BEDLAM bias ChatGPT class imbalance computer vision dataset augmentation depth fundamentals generative models ghost AI Infinigen introduction long-tail recognition MIT study monocular video object removal opinion paper analysis pre-training procedural generation quality inspection real-data limit representation learning representativeness robotics semantic segmentation sim-to-real SMPL SURREAL synthetic data synthetic humans synthetic images tech giants video

Redes sociaisSocial Links

Bileki

Atualizado em Updated

FUNDAMENTOS

What synthetic data is, what the literature shows, when it pays off and where it falls short. A first-timer’s guide for computer vision teams.

Synthetic data is images and annotations generated by computer instead of collected from the real world. This post is a first-timer’s map: what it is, what the literature has shown, when it pays off and where it doesn’t help.

What synthetic data is

Instead of photographing the world, you build the scene (objects, light, camera, background) and render it. Boxes, masks, depth and pose can come out of the scene itself, together with the pixel. There are three ways to get training data, and the difference between them decides almost everything:

  • Real collection: the world decides what’s in the scene, and annotation comes afterward, by hand, with cost and noise.
  • Generative AI: a model produces plausible images. The class label can come from the prompt, but boxes and masks have to be estimated afterward and may not match the pixel.
  • 3D simulation: you decide what’s in the scene, and the label is born with the render. It’s specified, not inferred.

[image: compare-1024×285.webp] Figure 1. Real and synthetic: on the left, a photo of a car; on the right, the same model in a 3D scene in Blender. Source: SynthVision (3D render); photo: [credit to confirm].

When it pays off, and what the literature shows

The signs that it’s worth it are objective:

  • The critical scenario is rare in real collection.
  • The object is dangerous, expensive or infeasible to capture.
  • Manual annotation is the main cost.
  • The project needs systematic variations (light, angle, weather) that no collection covers.
  • The model needs a dense label (depth, flow, pose).

Some published results, each with its own scope:

  • Urban driving. In SYNTHIA, mixing synthetic images into real training raised average per-class accuracy on CamVid from 46.3% to 56.5% (T-Net) and from 52.8% to 72.1% (FCN).
  • Tracking. In Virtual KITTI, pre-training on virtual clones raised the DP-MCF tracker’s MOTA on real videos from 71.9% to 76.7%.
  • Rare classes. SynBalance took accuracy on ImageNet-LT’s rarest classes from 12.6% to 21.4%.
  • Humans. In BEDLAM, a model trained on synthetic only tied with one trained on real images on 3DPW (46.6 mm vs 46.4 mm error).

The limit: the label is the product, and real is the test

Image beauty isn’t the criterion. What teaches the model is the image-label pair: a photorealistic image without a trustworthy annotation teaches little, and a decent image with exact labels teaches a lot.

And in the works above, synthetic doesn’t replace real on its own:

  • In SynBalance, training on synthetic only gave 28.1%, against 42.0% for the real-data baseline.
  • In Virtual KITTI, the authors write that training on virtual data only “is not enough”; the best result came from virtual pre-training followed by real training.
  • In SYNTHIA, global accuracy dropped in one case (FCN on U-LabelMe, from 79.4% to 76.2%).

There are also two limits realism doesn’t solve: simulation also embeds bias (the choice of what to simulate is human; it can be audited, but it doesn’t disappear), and the realism that matters is your task’s, not the pretty picture’s. An expensive detail that doesn’t move the metric can go.

What changes with a controlled scene

In a controlled scene you change one factor at a time. In Virtual KITTI, the same video was rendered with fog, and the MDP tracker lost 57.4 points of MOTA compared with the fog-free version. In collected data, isolating an effect like that is much harder.

In practice, the working loop has three steps:

  • Define the task and the label format your team consumes (COCO, masks, keypoints, depth).
  • Simulate the problem (the scene, the object, the variations that matter) and generate the first dataset.
  • Train and measure on the real domain, on your model, and return to simulation to cover the failures that appeared.

SynthVision mainly supplies the data to teams that already have their own model and training pipeline. When it makes sense, it also develops the full project or acts as a consultancy. The decisions (what to generate, which variables to vary) stay with the team; generation and labeling are automatic. For an industrial case, see quality inspection with synthetic data.

What to take to your project

  • Start from the label question. What answer does the model need to give (box, mask, depth)? If the label is the bottleneck, synthetic enters the conversation early.
  • Choose the method by the problem. Simulation for a specified scene, generative AI for visual variety, real collection for the final test. All three coexist.
  • Always validate on real data. Real data remains the test that decides whether synthetic helped.

Want to start a synthetic data project without starting wrong? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

,
, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *