AI companies are struggling to build more advanced models

AI models are experiencing a gym bro’s worst fear – the plateau. Three of the leading AI companies are now seeing diminishing returns from their costly efforts to build newer models, according to Bloomberg.

• Orion, a new model OpenAI is working on, is so far not considered to be as big a step up as GPT-4 was from GPT-3.5, the system that originally powered the company’s flagship chatbot.
• An upcoming iteration of Google’s Gemini software isn’t living up to internal expectations, sources told Bloomberg.
• The timetable for Anthropic’s Claude 3.5 Opus model, meant to be an improvement over its 3 Opus model, has been pushed back.

All about the data

AI models devour data like Guy Fieri with sandwiches – and it’s become increasingly difficult to find new, untapped sources of high-quality, human-made training data to build more advanced AI systems. Orion’s unsatisfactory coding performance was due in part to the lack of sufficient training data, Bloomberg reports.

Scraping the web for data was sufficient to train models up to this point. But the AI training approach is changing.

Companies are negotiating data partnerships with publishers and universities (OpenAI in particular). And some tech companies are hiring people with graduate degrees that can label data related to their own subject expertise, such as math and coding. The use of synthetic data, AI-generated artificial data that mimics real-world data, is also being explored, though researchers have consistently found that training AI systems on this data causes the model’s output quality to worsen.

👀 Looking ahead…Artificial general intelligence, which has been touted by many in the industry (especially OpenAI CEO Sam Altman) as being right around the corner, may not be so close after all. Cue the talk of agents – AI models that can perform multi-step tasks on behalf of users. But those have a long way to go, too. In several research papers, LLMs only achieved a ~15% success rate on more complex, outcome-based tasks.