FUNDAMENTOS
At NeurIPS 2024, Sutskever said pre-training as we know it will end: there’s only one internet. What the talk says, and what it doesn’t say about vision.
In December 2024, at the NeurIPS conference in Vancouver, Ilya Sutskever, OpenAI cofounder and founder of Safe Superintelligence, said that "pre-training as we know it will unquestionably end." The line is widely quoted in discussions about the future of training data. It’s worth separating what he said from what tends to get built on top of it.
What the talk says
According to The Verge’s coverage of December 13, 2024, Sutskever said onstage:
"Pre-training as we know it will unquestionably end."
"Pre-training" here is the first phase of building a language model, when it learns patterns from large amounts of unlabeled data, typically text from the internet, books and other sources. He also said the industry is running out of new data sources and compared the situation to fossil fuels: just as oil is a finite resource, the internet contains a finite amount of human-generated content. The most quoted line:
"We’ve achieved peak data and there’ll be no more. We have to deal with the data that we have. There’s only one internet."
A slide from the presentation sums up the argument: compute is growing (better hardware, better algorithms, larger clusters), while "data is not growing: we have but one internet; the fossil fuel of AI." The report also notes that he believes existing data can still take development further, and that next-generation models will be "agentic in real ways" and able to reason. What that means for where training data comes from is an open question, and the talk, as reported, doesn’t answer it.
What the talk doesn’t cover
- The subject is pre-training of language models, with text. Computer vision and image data weren’t the topic, as far as the report shows.
- It’s a prediction, made onstage at a conference, not a measured result or a study with data.
- The line isn’t about quality or domain. “Peak data” speaks to the volume of public text. It says nothing about the data missing for a specific task.
What changes when the data stock is the bottleneck
This section is our reasoning, not the talk’s. In computer vision, the problem is usually more specific than volume: the rare class that almost never shows up, the extreme condition, the defect that happens once a week. That’s not a global data limit, it’s a coverage limit of your set, and it exists even when there’s plenty of data in the world.
Two points in this line of reasoning call for care:
- Synthetic data isn’t a single source. A generator trained on real data depends on the real-data stock (our reading of the MIT study shows this in one case). A built scene doesn’t depend on it, but it depends on whoever models it.
- Real data is still the test. Synthetic data feeds training; only a test on real data shows whether it worked in your domain.
What to take to your project
- Separate volume from coverage. Is your bottleneck too little data, or a missing type of data? The answer changes the solution.
- Read the source before citing it. The talk is about language pre-training. Extending it to your vision task is a hypothesis to test.
- Measure on real data. Any mix of synthetic and real data should be evaluated on a real set that didn’t take part in training.
Is your project’s bottleneck the dataset? Tell us about it in a diagnostic call. If synthetic data isn’t the answer, we’ll say so.
Sources
- Robison, K. OpenAI cofounder Ilya Sutskever says the way AI is built is about to change. The Verge, December 13, 2024.
Leave a Reply