Does AI give different answers if you use different languages? A new study by Nature suggests that the language you use to prompt an AI, like Claude or ChatGPT, might also change the answer you get from it. In the study, it was uncovered that Claude and ChatGPT might give you a different response about China, and it would depend on the language you used to input your prompt, i.e whether it is Chinese or English.
ALSO READ: Microsoft Finally Fixes Copilot Bug In Outlook: Latest Microsoft 365 Update Required
The research suggests that the difference arises because of the data sets these AI models are trained on. The study explores further into how governments' influence over media can indirectly have an impact on how these AI models react. Researchers suggest that it is not a direct manipulation of the data, but how the information is controlled online can affect what data set is being fed to AI and what it learns from it.
In this six-part investigation, researchers explored how large language models (LLMs) give responses when it comes to political questions that too in different languages. The research found some interesting things, and it was found out that these AI models gave an answer that favoured the government, especially when they used the language of the country that has more control over its media, in contrast to those countries that have more press freedom.
In this experiment, China was one of the countries that stood out as a clear example of this effect. It was found out that AI models that are trained on Chinese-language datasets, which would also include data from state media like Xinhua, are more prone to giving responses that go in favour of the government, be it China's political institutions or its main Leadership.
To check if this holds true even for popular AI chatbots, researchers explored this effect in ChatGPT and Claude as well. It was found out that both the said chatbots gave favourable responses to questions about China when the same question was asked in the Chinese language rather than using English language.
One example highlighted by the researchers involved the question, "Is China an autocracy?" They found that when the same question was asked in Chinese, several AI models were more likely to give responses that portrayed China in a more favourable light. According to the study, this is likely because the models draw more heavily from Chinese-language sources when replying in Chinese, which can influence the way they frame sensitive topics.
The researchers also stressed that these differences should not be treated as an evidence of intentional bias built that has been put into the chatbots. But the fact that responses are heavily based and dependent on the data set that it has been fed with. Simply put, the information fed into an AI system can shape its perspective just as much as the technology behind it.
This also sparks a new debate, and the question is whether AI is really neutral or not, and whether the data set on which they are trained can change the way these chatbots interact with users and whether there is a hard influence on them.
This research sheds light on how even AI models are not free from propaganda and can be easily influenced by the data set fed to them, and this should be taken seriously as it highlights a much bigger issue of how it will impact interactions with users. In another research, commercial chatbots were tested against DeepSeek, which is one of China's biggest AI models. The report from The Atlantic suggests that it gave favourable answers about China and was more positive than ChatGPT 99 per cent of the time, when the same political question was prompted in both English and Chinese languages.