Bill Gates watched ChatGPT pass an AP Bio exam and went into ‘a state of shock’: ‘Let’s see where we can put it to good use’

Even tech icon Bill Gates says he was caught off guard by the rapid development of artificial intelligence programs like ChatGPT.

On Thursday, Gates opened up about “the most amazing demo I’ve ever seen in my life” — a demonstration of ChatGPT’s capabilities on an AP Biology exam in August 2022, he said on an episode of his “Unconfuse Me” podcast with Khan Academy CEO Sal Khan.

The story began last June when Gates first tested the AI-powered chatbot – developed by Microsoft partner OpenAI – and came away unimpressed. “I thought, ‘Yeah, that’s kind of idiotic. I don’t think it’s practical,'” Gates said he noted at the time.

He issued what he thought was a rather difficult challenge to OpenAI CEO Sam Altman: Bring ChatGPT back to him when it could demonstrate advanced human-level competence by achieving the highest possible score on the AP Biology exam.

“I thought, ‘OK, that will give me three years to work on HIV and malaria,'” Gates joked on the podcast.

Two months later, OpenAI’s developers returned, and Gates saw ChatGPT achieve a top score of five in the test. The program had gone from not being able to read or write “in the sense that humans do,” Gates said, to being able to do both at an almost human-like level.

“I’m still personally in a state of shock about, ‘Wow, that’s so good,'” Gates said.

ChatGPT’s success comes with “lots of footnotes about hallucinations and things like that,” Gates said, noting that even the most advanced AI models can make significant mistakes or fabricate information altogether.

Still, the rapid development left Gates deeply impressed and excited about the technology’s potential applications: “Let’s see where we can use it for good,” he said.

Throughout the podcast episode, Gates and Khan discussed whether education could be one of those “good uses.”

Gates, an optimist on this front, has predicted that AI chatbots like ChatGPT or Google’s Bard could start helping children learn to read and write as early as next year. Khan agreed, saying his company is developing an experimental chatbot tutor called Khanmigo for just that purpose.

Khanmigo can already “act like a pretty good human tutor,” Khan said. “There are moments with it that I think would pass the Turing test, where you’d think there’s a good person on the other side of the chat.”

Of course, moments of brilliance don’t necessarily translate into consistent human-level output. AI tutors are unlikely to replace teachers anytime soon, Khan and Gates both said: Rather, they could help make teachers’ lives easier and provide lower-cost tutoring opportunities for underserved students in low-income areas in the U.S. and abroad.

“It makes mistakes, to be clear,” Khan said of OpenAI’s latest major language model, GPT-4, which rolled out this summer. But he added that the model is “dramatically better” than previous iterations, with OpenAI noting that it can outperform 90% of human test-takers on the SAT.

These errors are reasons for caution, Barnard College child psychologist Tovah Klein told CNBC Make It last month. Children shouldn’t come to rely on AI tools as their only source of information, and AI models need to get better at explaining how they arrive at certain answers, she said.

“If we have the science that shows that kind of learning, beyond a teacher, is useful, then I think there’s a role for AI,” Klein said. “I think part of the problem is that we don’t really know [yet].”

Still, Gates has seen enough of ChatGPT’s evolution — including its growing ability to explain its reasoning — to predict a game-changing impact on education within the next decade, he said.

“If we think about the next 10 years, [in terms of] both the absolute level of learning and the gap with lower-income, minority students … these new tools can both close the gap and raise overall achievement levels,” Gates said.

DON’T GO JOIN: Want to be smarter and more successful with your money, work and life? Sign up for our new newsletter!

Get CNBC’s for free The Warren Buffett Guide to Investingwhich distills the billionaire’s #1 best advice for ordinary investors, do’s and don’ts and three key investing principles into a clear and simple guidebook.