Synthetic humans: from SURREAL to BEDLAM

ArquivoArchive

CategoriasCategories

Posts recentesRecent Posts

TagsTags

4D reconstruction AI and the brain AI ethics AI infrastructure AI transparency BEDLAM bias ChatGPT class imbalance computer vision dataset augmentation depth fundamentals generative models ghost AI Infinigen introduction long-tail recognition MIT study monocular video object removal opinion paper analysis pre-training procedural generation quality inspection real-data limit representation learning representativeness robotics semantic segmentation sim-to-real SMPL SURREAL synthetic data synthetic humans synthetic images tech giants video

Redes sociaisSocial Links

Bileki

Atualizado em Updated

PAPER ANALYSIS

From SURREAL (6.5M frames) to BEDLAM, Michael Black’s line of work shows when synthetic humans match real data, when they don’t, and what’s missing.

Michael J. Black’s group at the Max Planck Institute for Intelligent Systems has one of the longest lines of research on synthetic humans: from the SMPL body model to SURREAL, AMASS and BEDLAM. The numbers in two of these works are worth reading. They show that synthetic data can match real data on some tasks, not all, and they say what was left out.

What the line of research did

  • SMPL (2015). A parametric model of the human body. Pose and shape become numbers, and the body becomes a 3D object that can be animated.
  • SURREAL (2017). Made together with Inria. In Learning from Synthetic Humans, the authors rendered 6,536,752 frames, in 67,582 clips, from 2,607 motion-capture sequences and 145 subjects. Each frame comes with 2D and 3D pose, depth, body-part segmentation and optical flow. Clothing (930 textures) and backgrounds (about 400K LSUN images) vary per clip, and the motions come from the CMU MoCap database.
  • AMASS (2019). A motion-capture archive that unifies 15 mocap datasets in a common format: over 40 hours, over 300 subjects and over 11,000 motions.
  • BEDLAM (CVPR 2023). A synthetic dataset of about 380K RGB frames, with 1 to 10 people per image. It uses 111 simulated outfits, 1,691 clothing textures, 27 hairstyles, 100 skin textures and 2,311 AMASS motions, in scenes rendered in Unreal Engine 5.

What the results show

SURREAL, training on synthetic only and testing on real data:

  • On Human3.6M segmentation, synthetic came close to real but stayed below: 46.35% IoU against 49.61% for training on real data only. Training on synthetic and fine-tuning on real gave 57.07%.
  • On Freiburg Sitting People, synthetic alone (40.10%) beat real alone (28.77%). The authors explain that the real-only model, with few subjects and little clothing variety, learned skin color more than body parts.

BEDLAM, training on synthetic only:

  • On 3DPW, BEDLAM-CLIFF’s pose error (PA-MPJPE) was 46.6 mm, against 46.4 mm for the same method trained on real images. Practically a tie.
  • On RICH it was better: 51.2 mm against 55.7 mm.
  • The authors highlight that even the simple HMR method, trained on BEDLAM, came close to CLIFF, and suggest better data can matter more than architecture. Fine-tuning on the 3DPW training set lowered BEDLAM-CLIFF’s error from 46.6 to 43.0 mm.

The pattern is similar in both: synthetic is worth a lot, but a final adjustment with real data still helps.

The limit: what each result covers

  • Each result is for one task and one benchmark. A tie on 3DPW and a win on RICH don’t become a rule for other domains. BEDLAM’s “real-only” CLIFF was trained by the authors themselves, with the same setup.
  • SURREAL’s test is simple. The authors acknowledge that Freiburg and Human3.6M have little background clutter and few subjects. On Human3.6M, the ground truth wasn’t hand-annotated: it comes from fitting an SMPL body to motion-capture data.
  • What the authors list as limits: SURREAL doesn’t model cloth deformation and integrates backgrounds poorly with scene lighting. BEDLAM has few sitting and lying poses, doesn’t model interaction between people or with objects, has no hair physics, and no children or bodies with amputations. Some commercial assets couldn’t be used because their licenses forbid training neural networks.
  • Coverage is what the lab can model. Bodies advanced because a body model existed. For parts, soil, fabric or tools, that model still has to be built, and that isn’t in the papers.

What changes with a controlled scene

What SURREAL and BEDLAM do is build the scene and extract the label from it:

  • The label is born in the scene. Pose, depth and mask come from the same place as the pixel.
  • Variation is declared. Clothing, camera, light and background change by parameter, and the authors of both papers chose which to vary.
  • Scale is a compute budget, not a manual-annotation budget.

What the authors themselves show is the other side: what wasn’t declared in the scene (children, interaction, hair) doesn’t exist in training. That’s why the method matters as much as the generator: deciding what to cover, how much realism each point needs, and how to validate against real data. For an industrial example of the same reasoning, see our analysis of quality inspection with synthetic data.

What to take to your project

  • Ask what the benchmark covers. “Matched real on 3DPW” is a fact about 3DPW, not about your problem.
  • List what your scene leaves out. The papers list their own gaps; do the same before training.
  • Keep a final adjustment with real data. In both works, it still added something.

Does your sector still lack a ready-made model like SMPL? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.

Sources

,
, , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *