AI Inbreeding - AI trained with AI-generated data (opportunities and risks)
We all know the game of silent mail. This game shows how quickly wrong tendencies can be reinforced and the actual message can end up being completely consumed. It's the same principle with AI
Training new AI models is expensive. According to Dam Altman (CEO of OpenAI), the training of GPT-4 alone cost more than 100 million US dollars. It is not only the computing power required that is a decisive factor, but also the data used to train the models. In line with the "shit-in shit-out" principle, AI models must also ensure that the training data is of high quality.
Of particular interest is the development that AIs are increasingly being trained with data that has itself been generated by other AIs. This phenomenon raises many questions and poses both opportunities and significant problems that are worth looking at in more detail.
First of all, it is important to understand why AI-supported data generation is relevant in the first place. The sheer amount of data available on the internet and in various digital formats has increased exponentially. AI models require large amounts of high-quality data in order to make accurate predictions and recognize patterns. AI-generated data can help fulfill these needs by providing synthetic data that can supplement or even replace real-world data in many cases.
Take the development of language models, for example. They rely on analyzing large amounts of text to generate human-like language. AI can generate texts that show realistic and varied speech patterns. This enables faster development and testing of new models.
Problems of synthetic data
Despite these benefits, however, there are serious risks and challenges associated with the use of AI-generated data. A key problem is the quality and diversity of this data. When AIs are trained using AI-generated data, there is a risk that preconceived notions or biases are reinforced. This is because the AI cannot make judgments on its own; it only reproduces and reinforces the information it has been trained with. So if the underlying AIs produce biased or erroneous data, this will also be transferred to the new AI models.
Another major problem is the dependence on AIs that learn from other AIs. This is where the artificial intelligence cycle becomes particularly problematic. If one model is based on another, increasing the likelihood of errors or biases accumulating over generations, this can lead to a significant loss of trust in the technology. The cycle could end at a point where AIs no longer know where important information came from and how it was generated. This phenomenon is often jokingly referred to as AI madness.
All of us are familiar with the game of silent mail. This game shows how quickly wrong tendencies can intensify and the actual message can end up being completely consumed. With AI, it's the same principle.
To summarize, the combination of AIs and AI-generated data presents both opportunities and risks. The ability to work with synthetic data can encourage innovative approaches, but the ethical and qualitative issues involved should not be ignored. To reap the benefits of this technology without falling into the "AI madness", it is crucial to develop robust quality controls and transparency mechanisms. This is the only way to ensure that the next generation of AI models remains trustworthy and reliable.
With the increasing use of AI-generated data, do you think we are in danger of losing control of the truth and slipping into an endless spiral of bias and error, or is it all just doom and gloom?
Sources:
Contributors to Wikimedia projects
BBVA Switzerland
