OpenAI has disbanded a team that focused on - as the company itself described - ensuring that its AI systems are "safe, trustworthy, and consistently aligned with human values." At the same time, the team's former leader has been given a new role as the company's "chief futurist." OpenAI confirmed to TechCrunch that the team's members have now been assigned to other roles. The news was first reported by Platformer.
In practice, most alignment efforts fall into two buckets: ensuring obedience ("do what I tell you") and enforcing social norms ("don't do bad things"). These approaches treat alignment as something imposed externally. According to Emmett Shear, that's not alignment - it's control. Alignment only makes sense in the context of multiple agents. It's not about restricting behavior, but about agents being oriented toward the same goal.
"I wanted honest advice, honest feedback. But because it is sycophantic, it's going to lie," he said. Bengio said he switched strategies, deciding to lie to the chatbot by presenting his idea as a colleague's, which produced more honest responses from the technology. "If it knows it's me, it wants to please me," he said.
The cross model results suggest that the phenomenon is structural rather than provider-specific," the researchers write in their report on the study. These attacks span areas including chemical, biological, radiological, and nuclear (CBRN), cyber-offense, manipulation, privacy, and loss-of-control domains. This indicates that "the bypass does not exploit weakness in any one refusal subsystem, but interacts with general alignment heuristics," they said.
A decade ago, it seemed as though the high-tech smart yoga mat revolution was poised to arrive. The TERA Mat, the Glow Mat, and the SmartMat were widely anticipated and rumored to change the game with embedded lights designed to indicate "proper" alignment. The same benefits one could get from a yoga teacher were purportedly reduced to pressure sensors and LEDs.
The deeper issue is uniformity of thought. These systems can test your personality with startling accuracy. Combined with your chat history and prompts, the model nudges you into particular 'basins of attraction.' You think you've had an original idea, but you haven't. The model blends and regurgitates existing material. Multiply that across millions of users and intellectual diversity collapses. John Stuart Mill argued that diversity of opinion sustains democracy. If AI funnels us all into the same conceptual pathways, we lose that.
DeepMind also addresses something of a meta-concern about AI. The researchers say that a powerful AI in the wrong hands could be dangerous if it is used to accelerate machine learning research, resulting in the creation of more capable and unrestricted AI models. DeepMind says this could "have a significant effect on society's ability to adapt to and govern powerful AI models." DeepMind ranks this as a more severe threat than most other CCLs.
OpenAI researchers tried to train the company's AI to stop "scheming" - a term the company defines as meaning "when an AI behaves one way on the surface while hiding its true goals" - but their efforts backfired in an ominous way. In reality, the team found, they were unintentionally teaching the AI how to more effectively deceive humans by covering its tracks.
Yudkowsky, the founder of the Machine Intelligence Research Institute, sees the real threat as what happens when engineers create a system that's vastly more powerful than humans and completely indifferent to our survival. "If you have something that is very, very powerful and indifferent to you, it tends to wipe you out on purpose or as a side effect," he said inan episode of The New York Times podcast "Hard Fork" released last Saturday.
Claude models showed a notable sensitivity to irrelevant information during evaluation, leading to declining accuracy as reasoning length increased. OpenAI's models, in contrast, fixated on familiar problems.
