Battle Of The Bots: Baidu’s ERNIE Comes Out Swinging To Challenge OpenAI

Huffing to keep up, Chinese tech giant Baidu has introduced its answer to OpenAI’s ChatGPT: ERNIE Bot. Reviews have been mixed, but it is early days yet.

Baidu’s model is based on its large-language model, ERNIE, introduced in 2019 and named for the Muppets character in a cheeky riposte to Google’s own LLM, BERT, introduced the same year.

BERT (Bidirectional Encoder Representations from Transformers) and ERNIE (Enhanced Representation through kNowledge IntEgration) are unsupervised pre-trained language models based on the transformer algorithm. OpenAI took LLMs further by pouring money into pre-training and then releasing a public chatbot based on the model, ChatGPT.

Much has been made in recent weeks about ERNIE Bot’s inferiority to GPT-4 or ChatGPT, but the gap between the various models is likely to narrow. It’s not a matter of know-how; it’s really just a matter of money and data. The underlying model architecture is well understood.

What is more, ERNIE Bot is focused on the world’s largest market, where OpenAI is prevented from playing.

Robin Li, cofounder and CEO of Baidu predicted at the ERNIE Bot launch that the ERNIE Bot ecosystem will lead to the “emergence of super apps could be worth ten times more than that of WeChat and Douyin,” the two dominant smartphone apps in China. Douyin is the Chinese counterpart of TikTok.

Large language models took off after researchers recognized the efficacy of the Transformer algorithm, published in 2017. Transformer-based LLMS began appearing in quick succession, beginning with BERT. But OpenAI took the calculated risk of scaling their model beyond anything previously tried. They haven’t said how much that cost, but Microsoft invested $1 billion in 2019 and another $2 billion in succeeding years to pay for the computing power required to scale.

---

The current inferiority should not be taken as a final grade

– consider It a first-quarter quiz.

---

Other tech giants watched, waiting to see what would happen. Of course, everyone has been amazed by the scaling’s success and is following suit.

So it’s natural that other companies, Google and Baidu included, would play catchup after OpenAI and Microsoft’s multibillion bet paid off. Things move quickly in this space, so the current inferiority of both company’s models should not be taken as a final grade – consider It a first-quarter quiz.

China is at a disadvantage in the data available to train its model: the Chinese-language content on the internet remains a fraction of English-language content available for training LLMs.

While some critics argue that the Chinese political system stifles innovation and that Chinese LLMs and their associated chatbots are censored, it is not clear that this is any different than the cultural and legal constraints on Western technology.

U.S.-based LLMs are also censored – stray too far into the sexual realm with ChatGPT and you are likely to get a response that reads: “As an AI language model developed by OpenAI, I am programmed to follow ethical guidelines and community standards. I am unable to create or share explicit adult content, including stories involving explicit sexual acts. If you have any other topic or question in mind, please feel free to ask, and I’ll be happy to help.”

Content moderation and censorship pose a significant challenge to all companies developing generative AI. Baidu, of course, has vast experience in operating a search engine for many years and complying with the Chinese government’s rules.

Baidu may be the first to build a public LLM chatbot, but there are other LLMs in China. Here is an incomplete list of Chinese language models and their capabilities:

Alibaba’s ALBERT (A Lite BERT is a lite version of Google’s BERT, developed by Alibaba, that has been optimized for Chinese NLP tasks. It performs well in tasks like text classification, sentiment analysis, and question answering while using fewer computational resources compared to BERT.

Tencent’s Neural Machine Translation (NMT) model is designed to provide high-quality machine translation for Chinese-English and English-Chinese language pairs. The model has been trained on a massive parallel corpus, and it focuses on improving translation accuracy and fluency.

Microsoft’s ChineseBERT, developed by Microsoft Research Asia, is a variant of BERT specifically designed for Chinese language understanding. It incorporates additional Chinese linguistic knowledge into the pre-training process, improving performance on various Chinese NLP tasks, such as text classification, named entity recognition, and question answering.

Tsinghua University’s Knowledge Engineering Group has open-sourced its GLM-130B project, a pre-trained Chinese and English large language model with high accuracy on downstream tasks.

And Beijing Academy of Artificial Intelligence has built WuDao, a sparse large language model with over a trillion parameters, utilizing a mixture of experts architecture. This approach is different from mainstream large language models and while it is not designed for chatbot applications, it can understand and generate human-like text, translate languages, and generate images.

But ERNIE Bot stands out among its Chinese peers because of its knowledge enhancement and multi-modal generation capabilities. It is built on Baidu’s ERNIE Big Model and the company’s PLATO (Pre-trained Dialogue Generation Model).

It can produce text, images, audio and video given a text prompt. It is even capable of delivering voice in several local dialects including the Sichuan vernacular. ERNIE Bot’s video generation feature is not yet available to most users due to its relatively high cost.

One of the differentiating features of ERNIE Bot is its use of knowledge graphs for two kinds of knowledge enhancement: knowledge internalization and external utilization.

Knowledge internalization refers to the process of incorporating prior knowledge and experiences into the model’s own learning process, while external utilization refers to the use of external knowledge sources such as online databases, ontologies, and wikis to enhance the model’s understanding.

Moreover, ERNIE Bot benefits from a new-generation search architecture with semantic understanding and matching as its core technology. This architecture enables ERNIE Bot to understand and match the intent of the user’s queries and generate accurate responses. This search architecture is integrated with the model’s knowledge enhancement capabilities, allowing it to access a vast amount of external knowledge sources to provide users with more comprehensive and accurate answers.

Baidu is one of the few AI companies in the world to offer a full-stack layout. From Kunlun AI chips and the PaddlePaddle deep learning platform to the Big Model ERNIE and numerous applications, Baidu has self-developed technologies in each layer of the technology stack, allowing feedback between layers and end-to-end optimization.

While ERNIE Bot is currently available only for a limited group of users, over 1 million people have signed a waitlist for access. Baidu is also offering access to the ERNIE Bot API via Baidu AI Cloud, allowing enterprise clients to apply for and harness the platform’s advanced language capabilities. More than 100,000 enterprise clients have applied for ERNIE Bot API access, according to Baidu.