OpenAI's new AI model promises to be “more truthful and less toxic”

OpenAI has made a new version of its GPT-3 AI language model available that promises to be better at following user intentions while also producing results that are more truthful and less toxic.

The Open AI API is powered by GPT-3 language models that can be used to perform natural language tasks using carefully engineered text prompts. However, the models can also produce outputs that are untruthful, toxic, or reflect harmful sentiments.

The organisation's AI models have been criticised in the past for a range of shortcomings, including racism against specific genders and religions. The organisation once called GPT-3 too dangerous to make public, due to the API being able to create fake news stories by taking cues from the eight million web pages it had scanned to learn about language.

The organisation said this is partly because GPT-3 is trained to predict the next word on a large dataset of Internet text instead of safely performing the language tasks the user wants.

To make its models safer, and more aligned with users, OpenAI used a technique known as reinforcement learning from human feedback (RLHF), using human helpers called labelers to assist the AI in its learning.

“On prompts submitted by our customers to the API, our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3,” said the company.

It found the resulting models are much better at following instructions than the GPT-3. They also make up facts less often and show small decreases in toxicity. The organisation’s labelers prefer outputs from its new 1.3B InstructGPT model over outputs from its 175B GPT-3 model, despite having over 100x fewer parameters.

These InstructGPT models have been in beta on the API for over a year and are now the default language models accessible on OpenAI’s API.

“We believe that fine-tuning language models with humans in the loop is a powerful tool for improving their safety and reliability, and we will continue to push in this direction,” the organisation explained.

However, OpenAI outlined that there are some limitations to this model too. The InstructGPT models, for example, are far from fully aligned or fully safe, meaning they still generate toxic outputs, make up facts, or generate sexual and violent content without explicit prompting.

It said that to support the safety of its API, it will continue to review potential applications before they go live, provide content filters for detecting unsafe completions, and monitor for misuse.

OpenAI also highlighted that in many cases, aligning to the average labeler preference may not be desirable. The example it gave is that when generating text that disproportionately affects a minority group, the preferences of that group should be weighted more heavily.

“Right now, InstructGPT is trained to follow instructions in English; thus, it is biased towards the cultural values of English-speaking people,” it said. “We are conducting research into understanding the differences and disagreements between labelers’ preferences so we can condition our models on the values of more specific populations.”