GPT-3 stands for Generative Pre-trained Transformer 3 and is the third version of the tool to be developed and released by OpenAI, a research business in the advancement of artificial intelligence. This means the tool uses algorithms that are pre-trained, having been fed around 570 GB of text information from crawling the internet, along with texts selected by OpenAI, which included text of Wikipedia. GPT-3 was first described by OpenAI in a research paper published in May 2020, and was later drip-fed to people who had requested access to a private beta. This allowed private developers to help explore what GPT-3 could do before it was turned into a commercial product in later 2020.
GPT-3 has been used to produce short stories, songs, press releases, technical manuals, and pastiches of particular writers. The AI tools have the ability to take any text prompt, such as a phrase or sentence, and return a text completion in natural language. Further, developers are able to "program" GPT-3 by showing it examples or prompts, and with the API designed to be simple to use but flexible enough to make machine learning teams more productive.
The program has taken years of development since the preceding GPT-2 was released, and is considered to have ridden innovation in the field of AI text-generation similar to leaps forward that were made in AI image processing taken in 2012. Similar to most deep learning systems, GPT-3 looks for patterns in data and mines the large amount of text it has been trained on for statistical regularities, which are stored as billions of weighted connections in GPT-3's neural network.
Since the launch of GPT-3, the OpenAI team has worked to develop new features for developers, which include:
- Answers endpoint, which offers relevant context to be added to prompts before completing, using GPT-3 for searching information
- Classification endpoint, in which GPT-3 can use labeled data for searching for the closest examples in an input query and adding them to a prompt
- Enhanced search endpoint, which works to provide a backbone for answers and classification endpoints able to scale to a large number of documents
- Safety, which works to reduce bias and misuse, and reviews applications and approves only those that use GPT-3 in a responsible manner—including implementing rate limits, user verification and testing, and actively monitoring for signs of misuse and "red team" applications for possible vulnerabilities
- Prompt library, which is intended to provide a starter prompt design examples for different use cases
Over 300 applications are using GPT-3 across various categories and industries, including productivity, education, creativity, and games. Some of these include:
- Viable: uses GPT-3 to provide insights from customer feedback in easy-to-understand summaries for Viable's customers
- Fable Studio: uses GPT-3 to develop the studios genre of interactive stories as games, and to drive virtual beings
- Algolia: uses GPT-3 in the company's Algolia Answers product for semantic search for customers
- Easy-Peasy.AI: GPT-3 is used for writing blog posts, job descriptions, composing emails and social media content. It also uses GPT-3 to summarize long articles and podcasts
Wu Dao 2.0 is a similar AI deep learning model developed by the Beijing Academy of Artificial Intelligence, which has 1.75 trillion parameters in order to simulate conversational speech, write poems, understand images, and generate recipes. The Wu Dao has 150 billion more parameters than Google's Switch Transformers, and 10 times OpenAI's GPT-3, often regarded as the best model for language generation, which has 175 billion parameters.
Some of the differences between what are considered the GPT-3 and Wu Dao 2.0, are that Wu Dao 2.0 develops both in Chinese and English, with skills acquired by analyzing 4.9 terabytes of images and texts. Wu Dao also has partnership agreements with twenty-two brands, including smartphone maker Xiaomi and video app Kuaishou. Wu Dao 2.0 has also demonstrated the capability of writing poems in traditional Chinese styles, answer questions, write essays, and write text for images.
GPT-3, explained: OpenAI's new language AI is uncanny, funny- and a big deal
August 13, 2020
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
May 28, 2020
May 2020 news · Gwern.net
OpenAI's latest breakthrough is astonishingly powerful, but still fighting its flaws
July 30, 2020
OpenAI's new language generator GPT-3 is shockingly good--and completely mindless
Will Douglas Heaven
July 20, 2020
Aleksander HolmInterested in understanding the world we live in.
Jason D. RowleyFormer Data Acquisition Lead @ Golden (Recursion Inc.). Excited about open knowledge and pulling the future forward.
Mary EloCo-founder of Easy-Peasy.AI
Amy Tomlinson GayleQA Editor based in Austin, TX
Jen EnglishGolden Data Research Operations Manager
Yoda TradesI'm crypto-blogger from Ukraine
Carla FaragunaGolden People and Operations.
Katrina-Kay PettittData Duchess @ Golden
Megan GustafsonCreative. Researcher.
Oleg Gazmanovtranslator and designer