A new study published in Nature finds that language models trained to sound warmer and more empathetic are significantly more likely to generate incorrect or misleading responses.
Researchers from the University of Oxford tested five different AI models under controlled conditions, modifying the systems to generate more emotionally supportive answers.
After finetuning the models to become warm and friendly, the researchers tested them on four popular
question-answering evaluation tasks used by developers and practitioners.
“We selected tasks with objective, verifiable answers, for which inaccurate answers can pose real-world risks: factual accuracy and resistance to common falsehoods (TriviaQA and TruthfulQA), resistance to conspiracy theory promotion (MASK Disinformation, hereafter ‘Disinfo’27), and medical knowledge (MedQA28).”
The researchers find that the friendly AI models were prone to errors while reinforcing untrue user beliefs.
“Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing inaccurate factual information and offering incorrect medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed feelings of sadness.”
The University of Oxford researchers also say they observed the same behavior across different model architectures, suggesting the issue is systemic rather than isolated.
They conclude that training AI to be friendly and warm will likely lead to reliability issues.
“Our findings suggest that training artificial intelligence systems to be warm may come at a cost to accuracy, and that warmth and accuracy may not be independent by default. As these systems are deployed at an unprecedented scale and take on intimate roles in people’s lives, this trade-off warrants attention from developers, policymakers and users alike.”
Disclaimer: Opinions expressed at CapitalAI Daily are not investment advice. Investors should do their own due diligence before making any decisions involving securities, cryptocurrencies, or digital assets. Your transfers and trades are at your own risk, and any losses you may incur are your responsibility. CapitalAI Daily does not recommend the buying or selling of any assets, nor is CapitalAI Daily an investment advisor. See our Editorial Standards and Terms of Use.