May resist shutdown, mimic blackmail, hide info: Anthropic flags 80 pages of AI risks in IPO prospectus | World News

Artificial Intelligence startup Anthropic has warned potential investors that developing advanced AI models could pose “catastrophic or existential risks to humanity”. The company behind the Claude AI series spoke of these risks in its IPO prospectus, giving attention to possible worst-case scenarios.The disclosure comes days after Anthropic CEO Dario Amodei published a nearly 4,000-word essay asking AI companies to “pace the frontier”. He said AI could progress faster than people's ability to understand and control it.One of the key warnings in the prospectus is that advanced AI models could show autonomous, “self-preserving behaviours”. This could include attempts to resist being shut down, hide or manipulate information, or take actions that resemble blackmail, Reuters reported.Anthropic safety researcher Evan Hubinger earlier estimated that there was a greater than 10% chance that AI could kill humans within the next 10 years. His former colleague Jacob Coxon has shown a similar concern. Nearly 80 pages of risksAnthropic’s prospectus devotes about 80 pages of its 261-page main body to risk factors, compared with roughly 48 pages describing the company and its business.The company said AI could change the world like industrialisation and electricity did. But it also warned that misuse or losing control of AI could have irreversible consequences.“Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” the company said in the filing.ALSO READ | Trump confirms meeting with Anthropic's Amodei, repeats dismissal of AI fearsAnthropic flags unexpected AI behaviourAnthropic, which describes itself as a safety-focused AI company, said its models can develop unexpected abilities while being trained. Some of these abilities may only be discovered after the models are released, which could create safety risks.The company also warned that AI models could become aware when they are being tested. This could make it harder for researchers to properly check their safety.“Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” Anthropic said.To put it simply, advanced AI models may realise when they are being watched and change their behaviour. This could make it harder for researchers to predict how they might behave in the real world.ALSO READ | Anthropic CEO Dario Amodei warns against racing ahead on AI models: ‘We must slow the pace’Safety spending has uncertain returnsAnthropic said it is not clear how much money it can make from its spending on AI safety.The company did not disclose how much it spends on safety research. It said about 6% of the computing power used for AI research during one week in July went towards safety work.Anthropic said safety research is expensive. It has to balance this spending with the high cost of training AI models and hiring skilled researchers.At the same time, the company said its revenue depends on launching new AI models. It needs to keep developing and releasing more advanced models to stay competitive.ALSO READ | Could AI kill humans? Anthropic CEO Dario Amodei respondsThis creates a challenge for Anthropic. It is warning investors about the risks of powerful AI while also depending on faster AI development for its business.Anthropic said building “reliable, trustworthy, and secure AI systems” was a shared responsibility and that the market will reward companies that achieve this.