Fooled by AI? These firms sell deepfake detection that’s ‘REAL 100%.’

But despite Deep Media’s growing prominence, the company’s sole machine learning engineer graduated from college two years ago — with an undergraduate degree in astrophysics, according to his LinkedIn profile. Meanwhile, a review of other employee LinkedIn profiles suggests the company has no PhDs, AI specialists or forensic scientists on staff — a lack of subject matter expertise that, experts say, suggests a disconnect from the fast-moving deepfake research community.

Gupta is “essentially saying, ‘Trust me.’ And that doesn’t work in science,” said Wael Abd-Almageed, a Clemson University professor who studies deepfakes.

Whatever its bona fides, Deep Media is part of a growing segment of start-ups pitching themselves as a bulwark against the flood of fake content that has risen alongside new, easy-to-use AI tools. Fears that synthetic media could disrupt elections and threaten national security have left institutions such as Congress, the military and the media desperately searching for a trustworthy technical fix to identify fake content. Last week, ChatGPT maker OpenAI announced that it is developing its own detection tool.

But while detection is an increasingly crucial and sought-after service, its capabilities are largely untested. The methods used to create deepfakes are constantly evolving, meaning detection tools built in a lab don’t always work in the wild, academics and researchers say. A test of several publicly accessible, free or low-cost deepfake detectors by the Reuters Institute for the Study of Journalism in April found that most could be fooled by the same techniques they were supposed to spot.

Nonetheless, about 40 young companies now offer deepfake detection services, with some claiming startling levels of accuracy: Originality.ai promises “99% AI CONTENT DETECTION ACCURACY.” AI Voice Detector says its tool can “ensure authenticity in important situations,” including court cases. GPTZero says it “convincingly surpasses all notable competitor AI detection services.” And to demonstrate its prowess, Kroop AI shows two seemingly identical cartoon faces on its website, one labeled “REAL 100%” and the other “FAKE 100%.”

Kroop AI CEO Jyoti Joshi said the cartoons are meant to illustrate how high-quality deepfakes can look strikingly realistic, adding that her company’s tool excels at picking up “subtle signatures” left by generative AI. Other companies said their detection results should not be read as the final word, with AI Voice Detector CEO Abdellah Azzouzi saying his company encourages users “to investigate more … after getting the results.”

And though Originality.ai claims 99 percent accuracy, CEO Jon Gillham wrote in a statement to The Washington Post that such claims “should not be believed” unless they’re supported by test results. Pointing to a company blog post, Gillham said his company discloses its detector’s results on five third-party data sets, as well as testing results for other detectors.

According to the analytics firm PitchBook, venture capitalists invested $200 million in deepfake start-ups globally in 2023 and the first quarter of 2024, up from $6.6 million in 2017. Those figures cover deepfake creation companies as well as detectors, but some of the biggest deals were for tools to identify AI-manipulated media. Still, spending on deepfake detection is paltry compared with the $29 billion invested in generative AI deals last year, per PitchBook, including funding for popular tools that can be abused to make deceptive images, clone voices or edit video.

With wars raging in Ukraine and Gaza and more than 4 billion people eligible to vote in elections this year, forensic scholars say the surge of detection start-ups — particularly those boasting near-perfect accuracy — risks throwing gasoline on the growing AI fire by providing a false sense of certainty about the legitimacy of some visual and audio content, while eroding public confidence in authentic pieces of media.

The potential security risks from labeling a piece of content “fake” are mounting. In October, the tech news site 404 Media reported that the free tool AI or Not marked an image the Israeli government claimed showed atrocities committed by Hamas as fake, contrary to expert opinion.

AI or Not CEO Anatoly Kvitnitsky said that, of a few dozen images related to the Hamas attack, “We got all of them right except this one.” AI or Not has since improved its tool, Kvitnitsky said, but he acknowledged that the mistake was concerning.

“I’m actually born in Ukraine and Jewish,” he said, “so it did hit home when we got some of this wrong.”

Deep Media also has found evidence of AI manipulation where others have not. Last month, Gupta told The Post that his company’s tool detected a “high likelihood” that a closely scrutinized video of Catherine, Princess of Wales, announcing her cancer diagnosis had been manipulated with AI.

Experts including Clemson’s Abd-Almageed, Brown University misinformation researcher Claire Wardle and Hao Li, the CEO of AI video effects company Pinscreen, said they found no evidence that the video, which was filmed by the BBC in March, was manipulated with AI.

In a more recent interview, Gupta told The Post that the company stands by its analysis. Gupta also defended the company’s level of AI expertise. Gupta, who graduated from Yale with a degree in biochemical engineering, said he personally has 15 years of experience in machine learning, beginning when he was 15 years old. He added that Deep Media employs experienced contractors, including people working toward their PhDs, but declined to name them.

“AI is a rapidly evolving technology that requires deepfake detection AI models to keep up with bad actors,” Gupta said in a statement, noting that Deep Media has trained and tested numerous detection models.

“Deepfake detection is a very difficult problem,” Gupta said in the interview, “and Deep Media is working tirelessly to continue to advance our technology.”

‘Can I trust it or not?’

Forensic scholar Hany Farid, a professor at the University of California at Berkeley, told The Post he’s spoken with four different developers this month who boast near-perfect detection rates. But he said start-ups can boost their numbers by grading themselves on a curve: They often train detection models on a particular set of deepfakes and then evaluate their ability to identify the same type of fakes.

In the real world, deceptive media may not exhibit the patterns AI has learned to detect, Farid said, adding that a more reliable approach requires a nimble combination of computer programming, physics, open-source intelligence and forensics.

Farid’s own detection start-up, Get Real Labs, uses machine learning, but also analyzes elements including shadows, geometry, compression settings and results of reverse-image searches, he said. Eventually, the company might achieve the accuracy to summarize its results in a “final answer” for clients, he said, but the science isn’t there yet.

“If you start to say, ‘We can, with high precision, tell whether something is real or fake,’ and you get it wrong, you are part of the problem,” Farid said. “In fact, you are arguably worse than the disease.”

This gap between lab and real-world results makes it difficult to evaluate competing detectors, Abd-Almageed said. And it makes the slew of entrepreneurs pivoting into deepfake detection as a hot start-up sector, akin to crypto or AI itself, particularly concerning, Farid said.

While boastful claims from eager start-ups are nothing new in Silicon Valley, Raquel Vázquez Llorente, head of technology threats and opportunities for the nonprofit Witness, said conclusions about content such as “80 percent AI-generated” can mislead the public.

“What the hell does that mean?” she said at a recent conference on generative AI and the law. “Can I trust it or not?”

Because deepfake detection is particularly urgent in an election year, Oren Etzioni, co-founder of the Allen Institute for AI, recently founded the nonprofit TrueMedia.org, which is dedicated to developing a trustworthy detection tool that it offers free to fact-checkers.

The Pentagon also is moving faster to keep pace with technological advancements. It has made itself more welcoming to tech start-ups by modernizing its acquisition process and loosening eligibility restrictions for its Small Business Innovation Research development fund. Known as SBIR, the three-stage program finances promising ideas, develops the technology and, finally, puts the technology into production. Most start-ups never make it to the third phase.

All five of Deep Media’s publicly listed military contracts were SBIR grants. Three have ended and a fourth ends in July, while the $1.25 million AI detection deal with the Air Force — now in phase two of SBIR — ends in November.

In a statement, Pentagon spokesperson Jeff Jurgensen said the Defense Department evaluates every SBIR proposal by “considering factors such as technical merit, defense requirements, and commercial potential,” looking for small businesses that might appeal to private investors, who have access to more capital.

99 percent accurate

Before focusing on deepfake detection, Deep Media was in the business of creating synthetic media. The company released what it called a universal translator to dub videos and launched a series of content creation apps, including Babble, CopyCat.ai, DubSync and PolyTalk. Two of the company’s smaller military contracts were for AI translation services.

“The reason that our deepfake detectors work — the reason we have Pentagon contracts — is because we’ve been pioneering the generative side for such a long time,” Gupta told The Post.

In previous press interviews, Gupta has claimed a 99 percent accuracy rate in identifying deepfakes. He amended that somewhat in his interview with The Post, saying he is committed to offering clients at least 95 percent accuracy, even on “the latest and greatest, highest-quality, hardest-to-detect deepfakes.”

Gupta said he tries to avoid misleading numbers, acknowledging that “people can fool you when they talk about accuracy.” If 10 images in a universe of 1,000 are fake, the model can declare everything real and be 99 percent accurate, he said. But in reality, he said, “That number is meaningless, right?”

In talking with The Post, Gupta repeatedly cited a $25 million, three-year cooperative research agreement for deepfake data generation and detection with the Air Force Research Laboratory as evidence of the company’s credibility. A copy of the contract reviewed by The Post shows that 60 percent of the $25 million value comes from resources provided by Deep Media.

The agreement previously has been cast as a $25 million contract, but Gupta acknowledged “that might have been misrepresented” in the press. The agreement does not provide “revenue coming into the company,” he said, “but it is supporting our AI research.”

Meanwhile, the advisory firm that helped Deep Media obtain its military contracts has since sued the company for failing to pay its bills. The lawsuit, filed in federal court in March by Stonegardens Advisory, also alleges that Deep Media falsely claimed Stonegardens’s managing member, a Marine veteran, as a Deep Media executive on its website and in marketing materials, as well as onstage at SXSW and Air Force events. Stonegardens declined to comment on the case.

Gupta declined to address the lawsuit’s claims. He told The Post that Deep Media began applying for military contracts because he fears that manipulative synthetic media could destabilize American society.

“If our government doesn’t have a deepfake detector,” Gupta said, “our government doesn’t know what’s real and what’s fake.”