During my school board exams, I found solace in watching Bert and Ernie’s shenanigans. Ernie’s wild ideas and Bert’s futile attempts to reason with him were a comedy break from my study routine.

But outside of Sesame Street, Baidu’s Ernie humiliated Google’s Bert in an AI competition in 2019, when the former outperformed both Microsoft and Google’s AI models in GLUE (General Language Understanding Evaluation), which was the widely accepted benchmark for AI language understanding systems.

Photo credit: Shutterstock

Ernie was considered among the most powerful large language models (LLMs) until OpenAI came up with ChatGPT. As soon as the latter became popular worldwide, Ernie soon lost out.

Some of Baidu’s own employees reported that the Chinese titan was caught off guard after the launch of ChatGPT, whose superior capabilities were not widely known until its rollout.

Baidu still went ahead and unleashed Ernie, touting its so-called multimodal capabilities. During the launch, CEO Robin Li remarked that the bot was not perfect, but it was being released because of market demands. However, only a few could test Baidu’s claims then, with some reporting that the multimodal capabilities are at subpar level compared to GPT-4.

However, Baidu last week released the Ernie 3.5 (Wenxin Big Model 3.5). The company claims that Ernie Bot’s latest version is based on this LLM.

As per the company’s blog, “the bot has surpassed ChatGPT (3.5) in comprehensive ability scores and [outperformed] GPT-4 in several Chinese language capabilities.” But OpenAI had already moved to GPT-4 since March. Baidu specifically highlighting GPT-3.5 indicates the bot is being compared with the free version of ChatGPT, and Ernie might be months away from where GPT-4 currently is.

Since Baidu has access to the world’s largest Chinese languages database, it is understable that Ernie 3.5 outpaced GPT-4 in that area. In comparison, GPT-4 faced challenges in Chinese languages despite its impressive performance in English, which could perhaps be attributed to how OpenAI possesses the biggest English language database.

However, if Ernie Bot wants to truly outshine GPT-4, it needs to deliver superior results in English as well. At present, the company seems to be playing catch-up with OpenAI, whether it’s making premature multimodal claims, launching Ernie-ViLG to rival DALL-E2, or the recent announcement of plug-ins for Ernie Bot.

GPT-4 still enjoys support from the international community, while Ernie Bot is limited to the Chinese developer community. Moreover, given how China is renowned for its internet censorship, the very environment in which Chinese generative AI initiatives are growing inherently restricts their potential.

See also: After layoffs, Zenius bets on offline learning in revival strategy

For instance, MIT Technology Review reported that Ernie-ViLG censors numerous prompts including Tiananmen Square, the country’s second-largest city square and a symbolic political center.

There is a high probability that similar measures may be imposed on Baidu’s Ernie Bot, making it inferior to GPT-4.

Furthermore, did I mention the premature multimodal announcement? Baidu’s latest blog post completely omitted any mention of images or videos, reinforcing claims that Ernie Bot is still far from achieving multimodal capabilities.

This was published as a part of AI Odyssey, a section on generative AI developments featured in Tech in Asia’s emerging tech newsletter.

Delivered every Tuesday via email and through the Tech in Asia website, this free newsletter breaks down the biggest stories and trends in emerging tech. If you’re not a subscriber, get access by registering here.