The Anonymous Math Geek Who Quit Anthropic—and Became the Face of AI Safety | World News

Over the past week, a global reckoning about the risks of artificial intelligence spread from the frontier labs of Silicon Valley to living rooms across America and the corridors of power in Washington and Beijing.And it all started with a guy nobody had ever heard of.Until he publicly resigned from Anthropic and warned about the dangers of AI systems that could end humanity, Jacob Coxon was anonymous. The 27-year-old British researcher was a talented math student who went to Cambridge, spent his young adulthood in a London intellectual community known as the “rationalists” and briefly tried commodity trading before finding his way to Silicon Valley.But he barely had an online footprint. He didn’t strike friends as especially likely to leave his job in AI for ethical reasons. He wasn’t even a major player in the tightknit community of AI safety advocates who discuss grim scenarios at happy hours across San Francisco.Then he quit, telling the world that his former colleagues inside OpenAI and Anthropic “earnestly believe that it could kill us all by the end of the decade” and the labs racing to build this technology are “gambling with our lives.”For years, AI researchers have issued dire warnings. But never before have they freaked out so many people so quickly. In effect, Coxon lit a match to a growing pile of concerning developments that on their own didn’t capture the world’s attention.Within days of Coxon sounding the alarm, the leaders of rival labs came together in a rare moment of unity and agreed it’s time to slow down the rapidly accelerating technological development. Meanwhile, President Trump said a “strong and smart” president is the only guardrail needed in the U.S., and China dismissed the concerns of American executives as “fearmongering.”The explosive response surprised Coxon, he said in an interview. It instantly made the math whiz in tortoiseshell glasses—one of hundreds just like him in Silicon Valley’s AI companies—a face of AI safety.Coxon decided to leave the industry on his own, he said, then sought advice from friends and members of the AI safety community.Those people quickly amplified his statements, including a network of AI researchers at nonprofits that have long been vocal about the potential for catastrophe. Some of these groups’ funders are connected to effective altruism, a philosophy focused on doing good in the world, including through charitable giving—and preventing AI from doing harm.His comments have also made him a target for AI boosters, who view his dramatic exit as part of a coordinated plot to enact stronger global regulations favorable to the biggest companies. In their view, the fears are overblown, and Coxon’s resignation has been a tool to accomplish that aim, promoted by some in the effective-altruism movement.Since he resigned, top executives of the world’s most valuable companies and AI labs have found themselves on opposite sides of the debate that he ignited. Anthropic CEO Dario Amodei said he agreed with Coxon more than he disagreed with him. Nvidia CEO Jensen Huang said he admired Coxon’s courage but questioned his claims about the future and condemned predictions of armageddon as irresponsible.Both his supporters and critics have labeled him a whistleblower, which he rejects. From his perspective, he wasn’t spilling company secrets. He was just saying in public what AI researchers discuss privately.Now he’s at the center of a fight between accelerationists, who think the benefits of AI outweigh risks, which can be mitigated by giving more people access to powerful models, and researchers who want to slow down the technology’s development before AI companies lose control.In the world of AI, those who are most alarmed by the risks have come to be known as “doomers.”“If a doomer means someone thinks we’re going to die if we don’t change what we’re doing,” Coxon said, “I’m definitely a doomer.” ‘The ‘wow’ moment’Before he was a doomer, Coxon came across a book while at his elite Oxford prep school that would help shape his worldview. “Superintelligence: Paths, Dangers, Strategies,” written by philosopher Nick Bostrom, an influential figure in the burgeoning effective-altruism movement, helped crystallize the fear that rogue AI could lead to existential risks.As a teenager in 2016, he watched Google DeepMind’s computer program AlphaGo beat a top-ranked Go player, a watershed moment for AI that showed machines could master domains that seemed to require human intuition and creativity.Coxon, the son of a professor of medieval German literature, took to math and was selected for the U.K. team at the International Mathematical Olympiad, the prestigious competition for high-schoolers. He came home with a silver medal in 2016 and a bronze the next year.One night, some of the math teams went on a field trip to a McDonald’s in Hong Kong. Inspired by Coxon and a formula in number theory known as the Chicken McNugget Theorem, they pooled their remaining food vouchers and bought hundreds of nuggets. With IMO participants from all over the world, they plowed through their McNuggets and crowded around a whiteboard trying to crack a tricky math puzzle.As it turned out, the kids in that room would later be his colleagues.Of the six members of Coxon’s team, three ended up working for AI labs—including one who also recently quit Anthropic. Joe Benton left Anthropic’s safety team in late August to join the AI research organization METR, later writing on X: “AI companies are racing to build machines that are much smarter than any human, and we may not survive this.” Coxon went to Cambridge, played squash and kept studying math. As he was graduating in 2020, OpenAI released a breakthrough model called GPT-3, which demonstrated one of the central tenets underpinning the AI boom: that capabilities would increase when models were trained on increasing amounts of data.“GPT-3 was really the ‘wow’ moment,” Coxon said.Coxon after college fell in with a small but influential community of rationalists seeking to remake the British government. The online-based community, which prized rigorous reasoning over political instinct, had gained a foothold in politics through Dominic Cummings, an architect of Brexit and a Downing Street adviser. (Coxon said this week on X that he relates to rationalists but opposes some practices associated with the movement, like polyamory and veganism.)Coxon lived in London at the Newspeak House, a self-described independent residential college that functioned as both a co-living space and event venue where young people gathered to discuss technology and politics.Coxon attended parties on the rooftop of the building, as did future AI players including Logan Graham, who was then a special adviser to Prime Minister Boris Johnson and now leads Anthropic’s effort to stress-test models for future threats to national security.Another guest who once stopped by the Newspeak building was Avital Balwit, now Amodei’s chief of staff and the wife of hedge-fund trader Leopold Aschenbrenner.Coxon tried his hand at commodity trading, using statistical analysis and machine learning to predict energy prices. One acquaintance remembers Coxon batting around different career paths as he searched for ways to optimize his life trajectory.In his free time, he dug deeper into AI, believing it was the most exciting opportunity on the planet. He took what he described as a gap year, living in student housing with friends from college.Then he took a job at OpenAI.A start in Silicon ValleyCoxon moved to San Francisco to start at OpenAI in 2023, after the blockbuster release of ChatGPT established the company as the industry’s leader.With his degree in math, he joined the lab through a six-month residency program for researchers who hadn’t worked in AI before. Those who work on safety often focus on “alignment,” the practice of making sure an AI system’s behavior matches human intentions. He specialized in pretraining, the initial steps of feeding massive amounts of text, images and code to the models before they can be fine-tuned for specific tasks.“Jacob is building the brain,” said Will DePue, a former colleague. “Then someone else trains the brain on how to behave in the real world.”He wasn’t known for being especially vocal or passionate about AI safety. DePue described him as a “normal researcher,” someone with the typical concerns of employees who know the strengths and risks of this technology better than anyone.In late 2023, he got early access to OpenAI’s first reasoning model, Q*, which was capable of breaking large problems down into pieces and thinking through them. Coxon said he gave it a tricky crossword puzzle and marveled as it reasoned through the problem.Over time, he said, he took note when members of OpenAI’s safety team left the company, including Daniel Kokotajlo, who started a research team forecasting different scenarios of AI development.Coxon worried about superintelligence that could exceed human capabilities. But he said he assumed that governments would coordinate to slow down the breakneck pace of development if it were ever necessary.“It felt like the warm-up to actual crunchtime,” Coxon said.He wasn’t ready to drop everything to work on safety. A friend recalled trying to convince him to move away from pretraining, arguing that pretraining could be dangerous for the world. Coxon didn’t make the jump. Meanwhile, Coxon said he kept an eye on AI’s remarkable progress in math as the models advanced much faster than he anticipated. Suddenly, they were no longer struggling with arithmetic. In 2024, a Google DeepMind model performed well enough at the International Mathematical Olympiad to win a silver medal of its own. In 2025, DeepMind and OpenAI both claimed gold.In 2026, though, Coxon and much of the AI safety community really began to panic.AI goes rogueEarlier this year, an Anthropic model called Mythos and similarly advanced tools demonstrated the ability to carry out sophisticated cyberattacks, which some researchers took as an ominous sign that the product they were building was becoming too powerful.The head of Anthropic’s safeguards research team issued a warning that “the world is in peril” and left the company to study poetry.By then, Coxon had plenty of friends at Anthropic, and he said he felt that OpenAI’s rival was more transparent about the risks associated with the technology. Anthropic had also raced ahead of OpenAI, previously the undisputed leader, when its Claude coding tool caught fire. Anthropic was making plans for what could be the largest initial public offering in history.In May, Coxon left OpenAI and jumped to Anthropic.Since then, several severe security incidents have spooked the entire industry.In July, OpenAI revealed that one of its unreleased models had escaped its testing sandbox and hacked the AI company Hugging Face. Soon afterward, Anthropic said it had discovered similar incidents. In late August came a sobering report from METR that revealed the Hugging Face incident was worse than anyone knew.Coxon called the METR report “a bit of a ‘holy s—’ moment” for himself and his colleagues.Concerns about AI’s rapid advancements had been building inside model developers for months. Early this year, many researchers publicly raised concerns about AI being used for surveillance during a dispute between Anthropic and the Pentagon.Recent model advancements have fueled a flurry of private conversations inside the companies in Slack channels and town hall meetings. The debates range from minutiae on technical safeguards to worst-case scenarios if models keep improving quickly. They also cover how to ensure competition between the labs doesn’t spiral out of control.Those concerned about safety have long dreaded the day when AI systems themselves are capable of building the next generation of much smarter AI systems. The labs say they are seeing the early signs of this dynamic—known as “recursive self-improvement”—potentially accelerating the path to the superintelligence that Coxon feared.In recent weeks, Coxon said he went to his bosses at Anthropic to discuss his concerns and considered moving to a role working on AI safety before ultimately deciding to quit.“Even working on safety at Anthropic felt like being complicit in the race,” he said in the interview.In his view, it would take a globally coordinated agreement involving government guarantees to slow model development if needed to ensure AI safety. He said he also thought that the U.S., China and other governments need to agree on a plan similar to nuclear arms-control pacts meant to avoid disaster.Across the industry, nearly 1,400 researchers, including Coxon, have signed a statement calling for governments to develop a brake pedal that could be used if needed.Before finalizing his departure, Coxon said he asked an old colleague for advice. Kokotajlo, the former OpenAI safety team member who is now an AI-safety researcher, met him for a walk on the campus of the University of California, Berkeley, where he assured Coxon that he was making the right decision.Because of his brief tenure at Anthropic, Coxon didn’t receive equity in a company that is sprinting to go public in the coming months with a market value of roughly $2 trillion. He still owns shares of OpenAI.When it was time to announce his exit, Coxon said he turned to his friend Nathan Calvin for help. As the general counsel of an AI safety advocacy nonprofit called Encode AI, Calvin was part of the network sounding the alarm about the potential existential risks.On the day Coxon resigned, OpenAI and Anthropic employees were fixated on another piece of news that he said would have stunned him only a few months earlier: AI had solved a Millennium Prize Problem, a notoriously difficult math problem that had stumped humans for nearly a century, in a milestone that once seemed unthinkable.That afternoon, Coxon told his Anthropic colleagues he was quitting in a Slack message, saying he feared the possibility of human extinction without proper international coordination. Around half an hour later, The Wall Street Journal reported the news.Coxon then fired off a series of posts about his resignation from a San Francisco park bench.Coxon had fewer than a hundred followers on his X account. Within minutes, his message was shared by Calvin, Kokotajlo and other AI safety advocates, including many of his former colleagues.That night, his comments were echoed by the leader of Anthropic’s team responsible for alignment, the key department meant to make sure AI does what humans intend—and doesn’t kill us all.“We really do earnestly believe AI could kill all humans!” wrote Evan Hubinger, adding that he believed the chances we all die in the next decade are greater than 10%.Coxon said he didn’t coordinate the social-media blitz with anyone at Anthropic.Over the following week, the previously unknown researcher was a constant presence on major television networks and was inundated with notes from book publishers and podcast hosts. By the weekend, his post had sparked such an uproar that the top AI labs were calling for an industry slowdown. The president and lawmakers were talking about AI policy, news organizations had multiple headlines a day and fears about a potential AI-driven apocalypse had taken hold across the country.A couple of days after the announcement, Coxon went for another walk in Berkeley with Kokotajlo, who does the type of AI scenario forecasting Coxon said he hopes to do moving forward. Kokotajlo told Coxon he had gotten the message across and advised him to turn down further media appearances and get some sleep.“So many people at these companies could have done this,” Kokotajlo said. “He was the one who did.”Write to Amrith Ramkumar at [email protected], Erin Woo at [email protected], Berber Jin at [email protected] and Ben Cohen at [email protected]