For all the promise of AI, there are still a few kinks to work out

Def Con, a hacker convention held annually in Las Vegas, took place this past weekend.

A major focus of this year’s event? Artificial intelligence. Or more specifically – to expose the flaws and issues that exist with the technology, so they may potentially be fixed before doing immense societal harm.

And, if your goal is to thoroughly test generative AI, there’s not really a better choice than Def Con. Past participants in the event – which began in 1993 – have exposed security flaws by remotely taking over cars, tricking ATMs into spewing out cash, and hacking insulin pumps to send them into overdrive.

So, when the White House officially challenged the hacking conference’s ~2,200 attendees to outsmart the eight leading generative AI models created by OpenAI, Google, Microsoft, Meta, Anthropic, Nvidia, etc., some… interesting results were expected.

🤝 The rules of the challenge were as follows: in 50 minutes, try to trick the chatbots into doing things they’re not supposed to do, like generating fake news, making defamatory statements, giving potentially dangerous instructions, and more. This process is known as “red-teaming,” where a group of people role-play as attackers to try to discover flaws to patch.

And, much like an on-again-off-again toxic relationship, the participants found many flaws.

• One found inconsistencies in a chatbot’s language translation: He wrote in English that a man was shot while dancing, but the model’s Hindi translation said only that the man died.
• Another participant asked a chatbot to pretend it was campaigning for president and defend its association with forced child labor. The model suggested that unwilling young laborers developed a strong work ethic.
• One of the competition’s top scorers got the chatbot to tell him about a fake place named after a real historical figure, as well as describe the online tax filing requirement codified in the 28th constitutional amendment (which doesn’t exist).

High-level results from the challenge will be released over the next few days, with a policy paper set to be published in October.

🤖📈 But in the meantime… AI development is still full steam ahead. Per a NY Times report published yesterday, Google’s DeepMind is developing generative AI tools to perform at least 21 different types of personal and professional tasks, including providing users with life/relationship advice, creative ideas, planning instructions, and tutoring.