书童按:本篇是达里奥·阿莫代伊(Dario Amodei)2026年1月发表的《技术的青春期》(The Adolescence of Technology)一文的上篇。阿莫代伊是 Anthropic 的联合创始人兼首席执行官。阿莫代伊设想,假如一座数据中心里聚集了足以构成一个国家的天才,人类如何判断它们会追求什么目标?他从这个问题谈起,讨论模型训练、可解释性和外部监督。译稿由 Luna 初译,经 DeepSeek 与 Qwen 交叉校审,以飨诸君。
英文原文 · The Adolescence of Technology | 原文版权归原权利人所有。本中英文对照翻译稿件,仅作为交流学习之用。
直面并克服强大 AI 的风险
Confronting and Overcoming the Risks of Powerful AI
卡尔·萨根的小说《接触》改编电影里有这样一幕:女主角是一位天文学家,她探测到了来自外星文明的第一道无线电信号,随后被考虑选为人类代表,去会见外星人。面试她的国际小组问:“如果只能问他们一个问题,你会问什么?”她回答:“我会问他们:‘你们是怎么做到的?你们如何进化,如何在没有自我毁灭的情况下度过技术青春期?’”每当我想到人类如今在 AI 发展上走到了哪一步,想到我们正站在怎样的临界点上,脑海里总会浮现那个场景,因为那个问题实在太切合我们的处境了。我真希望能有外星人的答案指引我们。我相信,我们正在进入一场既动荡又不可避免的成年礼,它将检验我们作为一个物种究竟是什么样的。人类即将获得几乎难以想象的力量,而我们的社会、政治和技术体系是否成熟到足以驾驭这种力量,仍是个极大的疑问。
There is a scene in the movie version of Carl Sagan’s book Contact where the main character, an astronomer who has detected the first radio signal from an alien civilization, is being considered for the role of humanity’s representative to meet the aliens. The international panel interviewing her asks, “If you could ask [the aliens] just one question, what would it be?” Her reply is: “I’d ask them, ‘How did you do it? How did you evolve, how did you survive this technological adolescence without destroying yourself?” When I think about where humanity is now with AI—about what we’re on the cusp of—my mind keeps going back to that scene, because the question is so apt for our current situation, and I wish we had the aliens’ answer to guide us. I believe we are entering a rite of passage, both turbulent and inevitable, which will test who we are as a species. Humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it.
在我的文章《仁爱机器》中,我试着描绘一种文明的愿景:它熬过技术青春期,顺利步入成年,解决了各种风险,并以娴熟而富有同情心的方式运用强大 AI,提升每个人的生活质量。我提出,AI 可以推动生物学、神经科学、经济发展、全球和平,以及工作与意义等领域取得巨大进展。我觉得,给人们一个值得为之奋斗、鼓舞人心的目标很重要;而 AI 加速主义者和 AI 安全倡导者似乎——奇怪地——都未能完成这一任务。但在这篇文章里,我想直面这场成年礼本身:梳理我们即将面对的风险,并着手制定应对方案。我坚信我们有能力渡过难关,坚信人类的精神与高贵品格,但我们必须正视局势,不抱幻想。
In my essay Machines of Loving Grace, I tried to lay out the dream of a civilization that had made it through to adulthood, where the risks had been addressed and powerful AI was applied with skill and compassion to raise the quality of life for everyone. I suggested that AI could contribute to enormous advances in biology, neuroscience, economic development, global peace, and work and meaning. I felt it was important to give people something inspiring to fight for, a task at which both AI accelerationists and AI safety advocates seemed—oddly—to have failed. But in this current essay, I want to confront the rite of passage itself: to map out the risks that we are about to face and try to begin making a battle plan to defeat them. I believe deeply in our ability to prevail, in humanity’s spirit and its nobility, but we must face the situation squarely and without illusions.
和讨论 AI 的益处一样,我认为讨论风险也应当谨慎、周全。尤其重要的是:
As with talking about the benefits, I think it is important to discuss risks in a careful and well-considered manner. In particular, I think it is critical to:
避免末日论。这里说的“末日论”,不只是认为末日不可避免(这种看法既错误,也会让预言自我实现),更广泛地说,是以一种近乎宗教的方式看待 AI 风险。1多年来,许多人一直以分析、冷静的方式思考 AI 风险;但据我观察,在 2023 至 2024 年对 AI 风险最为担忧的时期,一些最不明智的声音反而占了上风,常常是靠煽情的社交媒体账号传播。这些声音使用令人不适的措辞,听起来像宗教或科幻作品,还在没有足够证据支撑的情况下呼吁采取极端行动。当时就很清楚,反弹不可避免,议题也会因此陷入文化极化,从而导致僵局。2到了 2025 至 2026 年,风向已经转变,许多政治决策由 AI 的机遇而非风险推动。这种摇摆令人遗憾,因为技术本身并不在意什么时髦;而且相较于 2023 年,到了 2026 年我们离真正的危险近得多。教训是,我们必须以现实、务实的方式讨论并应对风险:保持冷静,以事实为依据,也要做好准备,熬过风向变化。
Avoid doomerism. Here, I mean “doomerism” not just in the sense of believing doom is inevitable (which is both a false and self-fulfilling belief), but more generally, thinking about AI risks in a quasi-religious way.1 Many people have been thinking in an analytic and sober way about AI risks for many years, but it’s my impression that during the peak of worries about AI risk in 2023–2024, some of the least sensible voices rose to the top, often through sensationalistic social media accounts. These voices used off-putting language reminiscent of religion or science fiction, and called for extreme actions without having the evidence that would justify them. It was clear even then that a backlash was inevitable, and that the issue would become culturally polarized and therefore gridlocked.2 As of 2025–2026, the pendulum has swung, and AI opportunity, not AI risk, is driving many political decisions. This vacillation is unfortunate, as the technology itself doesn’t care about what is fashionable, and we are considerably closer to real danger in 2026 than we were in 2023. The lesson is that we need to discuss and address risks in a realistic, pragmatic manner: sober, fact-based, and well equipped to survive changing tides.
承认不确定性。我在本文提出的这些担忧,可能在许多情形下都失去现实意义。这里没有任何内容意在表达确定性,甚至也不是说这些事很可能发生。最明显的一种可能是,AI 的进展远没有我想象的那么快。3或者,即使进展很快,这里讨论的某些甚至全部风险也可能不会出现(那当然最好);也可能还有我没考虑到的其他风险。没有人能完全确信地预测未来,但无论如何,我们都得尽力做好规划。
Acknowledge uncertainty. There are plenty of ways in which the concerns I’m raising in this piece could be moot. Nothing here is intended to communicate certainty or even likelihood. Most obviously, AI may simply not advance anywhere near as fast as I imagine.3 Or, even if it does advance quickly, some or all of the risks discussed here may not materialize (which would be great), or there may be other risks I haven’t considered. No one can predict the future with complete confidence—but we have to do the best we can to plan anyway.
尽可能精准地干预。应对 AI 风险,既需要公司(以及私营第三方)自愿采取行动,也需要政府出台对所有人都有约束力的措施。对我来说,自愿采取行动,并鼓励其他公司跟进,都是理所当然的事。我坚信政府在一定程度上也必须采取行动,但政府干预的性质有所不同,因为它可能摧毁经济价值,或强迫那些对这些风险持怀疑态度的行动者接受措施(而他们也确实有可能是对的!)。监管也常常适得其反,甚至加剧它们原本旨在解决的问题;面对快速变化的技术时尤其如此。因此,监管必须审慎:尽量避免附带损害,规则尽可能简单,并施加完成任务所必需的最低限度负担。4说“人类命运攸关,采取多极端的行动都不为过!”很容易,但实际上这种态度只会招致反弹。说清楚一点,我认为我们最终很有可能会走到有必要采取更重大行动的地步;但那需要比现在更有力的证据,证明具体危险迫在眉睫,也需要对危险有足够具体的认识,才能制定有望解决问题的规则。今天最有建设性的做法,是倡导有限的规则,同时继续观察是否有证据支持更强有力的措施。5
Intervene as surgically as possible. Addressing the risks of AI will require a mix of voluntary actions taken by companies (and private third-party actors) and actions taken by governments that bind everyone. The voluntary actions—both taking them and encouraging other companies to follow suit—are a no-brainer for me. I firmly believe that government actions will also be required to some extent, but these interventions are different in character because they can potentially destroy economic value or coerce unwilling actors who are skeptical of these risks (and there is some chance they are right!). It’s also common for regulations to backfire or worsen the problem they are intended to solve (and this is even more true for rapidly changing technologies). It’s thus very important for regulations to be judicious: they should seek to avoid collateral damage, be as simple as possible, and impose the least burden necessary to get the job done.4 It is easy to say, “No action is too extreme when the fate of humanity is at stake!,” but in practice this attitude simply leads to backlash. To be clear, I think there’s a decent chance we eventually reach a point where much more significant action is warranted, but that will depend on stronger evidence of imminent, concrete danger than we have today, as well as enough specificity about the danger to formulate rules that have a chance of addressing it. The most constructive thing we can do today is advocate for limited rules while we learn whether or not there is evidence to support stronger ones.5
说了这些之后,我认为讨论 AI 风险最好的起点,和我讨论 AI 益处时一样:先明确我们说的是哪个层级的 AI。让我担忧文明存续的,是我在《仁爱机器》中描述的强大 AI。我在这里重述一下那篇文章里的定义:
With all that said, I think the best starting place for talking about AI’s risks is the same place I started from in talking about its benefits: by being precise about what level of AI we are talking about. The level of AI that raises civilizational concerns for me is the powerful AI that I described in Machines of Loving Grace. I’ll simply repeat here the definition that I gave in that document:
我所说的“强大 AI”,是指一种 AI 模型——形态可能和今天的大语言模型相似,但也可能采用不同架构、由多个相互作用的模型构成,或以不同方式训练——它具备以下特性:
By “powerful AI,” I have in mind an AI model—likely similar to today’s LLMs in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties:
单论智力,它在大多数相关领域都比诺贝尔奖得主更聪明,包括生物学、编程、数学、工程、写作等。这意味着它能证明尚未解决的数学定理、写出极好的小说、从零开始编写复杂代码库等等。
In terms of pure intelligence, it is smarter than a Nobel Prize winner across most relevant fields: biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.
它不只是一个“可以和你聊天的聪明东西”,而是拥有虚拟办公的人类所能使用的所有接口,包括文字、音频、视频、鼠标和键盘操作,以及互联网访问。它可以执行这些接口所支持的任何操作、通信或远程任务,包括在网上采取行动、接受人类指示或向人发出指令、订购材料、指挥实验、观看视频、制作视频等等。它完成这些任务的能力,同样超过世界上最有能力的人类。
In addition to just being a “smart thing you talk to,” it has all the interfaces available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access. It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on. It does all of these tasks with, again, a skill exceeding that of the most capable humans in the world.
它不只是被动回答问题;你可以把需要数小时、数天甚至数周才能完成的任务交给它,然后它会像一位聪明的员工那样自主完成,并在必要时请求澄清。
It does not just passively answer questions; instead, it can be given tasks that take hours, days, or weeks to complete, and then goes off and does those tasks autonomously, in the way a smart employee would, asking for clarification as necessary.
它没有实体(除了存在于电脑屏幕上),但可以通过电脑控制现有的实体工具、机器人或实验室设备;理论上,它甚至可以为自己设计机器人或设备来使用。
It does not have a physical embodiment (other than living on a computer screen), but it can control existing physical tools, robots, or laboratory equipment through a computer; in theory, it could even design robots or equipment for itself to use.
训练模型所用的资源可以转而用于运行数百万个模型实例(这符合对约 2027 年集群规模的预测),而模型吸收信息和生成行动的速度大约是人类的 10 到 100 倍。不过,它的速度也可能受制于物理世界或所交互软件的响应时间。
The resources used to train the model can be repurposed to run millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10–100x human speed. It may, however, be limited by the response time of the physical world or of software it interacts with.
这数百万个副本可以各自独立处理互不相关的任务;必要时也可以像人类协作那样一起工作,或许还可以微调出不同的子群体,让它们分别特别擅长某些任务。
Each of these million copies can act independently on unrelated tasks, or, if needed can all work together in the same way humans would collaborate, perhaps with different subpopulations fine-tuned to be especially good at particular tasks.
我们可以把它概括为“数据中心里的天才之国”。
We could summarize this as a “country of geniuses in a datacenter.”
我在《仁爱机器》中写过,强大 AI 可能只需一两年就会出现,不过也可能要晚得多。6强大 AI 究竟何时到来,是个复杂的问题,值得另写一篇文章;眼下我先非常简要地说明,为什么我认为它很有可能很快到来。
As I wrote in Machines of Loving Grace, powerful AI could be as little as 1–2 years away, although it could also be considerably further out.6 Exactly when powerful AI will arrive is a complex topic that deserves an essay of its own, but for now I’ll simply explain very briefly why I think there’s a strong chance it could be very soon.
我和我的 Anthropic 联合创始人们,属于最早记录并追踪 AI 系统“规模定律”的人之列:随着我们增加算力和训练任务,AI 系统在几乎所有我们能测量的认知技能上都会可预见地变得更好。每隔几个月,公众舆论就会转而相信 AI“撞上了天花板”,或者会因某项“彻底改变游戏规则”的新突破而兴奋不已。但实际上,撇开舆论的起伏和公众猜测,AI 的认知能力一直在平稳、坚定地提升。
My co-founders at Anthropic and I were among the first to document and track the “scaling laws” of AI systems—the observation that as we add more compute and training tasks, AI systems get predictably better at essentially every cognitive skill we are able to measure. Every few months, public sentiment either becomes convinced that AI is “hitting a wall” or becomes excited about some new breakthrough that will “fundamentally change the game,” but the truth is that behind the volatility and public speculation, there has been a smooth, unyielding increase in AI’s cognitive capabilities.
现在,AI 模型已经开始解决尚未解决的数学问题;它们的编程能力也足够强,以至于我见过的一些最优秀的工程师如今把几乎所有编码工作都交给了 AI。三年前,AI 连小学算术题都做得吃力,几乎写不出一行代码。生物科学、金融、物理以及各种智能体任务,也都在以类似速度进步。如果这种指数级增长持续下去——这虽不确定,但已有十年记录支持这种判断——那么用不了几年,AI 就会在几乎所有事情上胜过人类。
We are now at the point where AI models are beginning to make progress in solving unsolved mathematical problems, and are good enough at coding that some of the strongest engineers I’ve ever met are now handing over almost all their coding to AI. Three years ago, AI struggled with elementary school arithmetic problems and was barely capable of writing a single line of code. Similar rates of improvement are occurring across biological science, finance, physics, and a variety of agentic tasks. If the exponential continues—which is not certain, but now has a decade-long track record supporting it—then it cannot possibly be more than a few years before AI is better than humans at essentially everything.
事实上,这幅图景可能还低估了进步的速度。AI 现在已经承担了 Anthropic 的许多编程工作,显著加快了我们开发下一代 AI 系统的速度。这个反馈循环正逐月加速,可能只需一两年,当前一代 AI 就能自主打造下一代 AI。这个循环已经启动,未来几个月、几年还会迅速提速。过去五年我一直在 Anthropic 内部观察进展,也看到接下来几个月的模型发展势头;我能感受到进步的节奏,也能感觉到时间正在倒数。
In fact, that picture probably underestimates the likely rate of progress. Because AI is now writing much of the code at Anthropic, it is already substantially accelerating the rate of our progress in building the next generation of AI systems. This feedback loop is gathering steam month by month, and may be only 1–2 years away from a point where the current generation of AI autonomously builds the next. This loop has already started, and will accelerate rapidly in the coming months and years. Watching the last 5 years of progress from within Anthropic, and looking at how even the next few months of models are shaping up, I can feel the pace of progress, and the clock ticking down.
在这篇文章里,我会假设这种直觉至少有几分道理——不是说强大 AI 一两年内肯定会出现,7而是说它到来的可能性不小,而且很可能就在未来几年。和《仁爱机器》一样,认真对待这个前提会得出一些出人意料、令人不安的结论。《仁爱机器》着重讨论这个前提带来的积极影响;而在这里,我要谈的事情会让人不安。我们也许不愿正视这些结论,但这并不会让它们变得不真实。我只能说,我日夜都在思考如何引导我们避开这些负面结果、走向积极结果;这篇文章会详细讨论怎样才能做得最好。
In this essay, I’ll assume that this intuition is at least somewhat correct—not that powerful AI is definitely coming in 1–2 years,7 but that there’s a decent chance it does, and a very strong chance it comes in the next few. As with Machines of Loving Grace, taking this premise seriously can lead to some surprising and eerie conclusions. While in Machines of Loving Grace I focused on the positive implications of this premise, here the things I talk about will be disquieting. They are conclusions that we may not want to confront, but that does not make them any less real. I can only say that I am focused day and night on how to steer us away from these negative outcomes and towards the positive ones, and in this essay I talk in great detail about how best to do so.
我认为,要把握 AI 带来的风险,最好的办法是先问这样一个问题:假设到了 2027 年前后,世界某处真的出现了一个“天才之国”。设想那里有五千万人,他们每一个人的能力都远胜任何诺贝尔奖得主、政治家或技术专家。这个比喻并不完全贴切,因为这些天才的动机和行为可能千差万别:他们可能完全顺从、服从指令,也可能在动机上显得陌生而难以理解。但暂且沿用这个比喻,假设你是一大国的国家安全顾问,负责评估并应对这一局面。再设想,由于 AI 系统的运行速度可达人的数百倍,这个“国家”相对于所有其他国家都拥有时间优势:我们每采取一次认知行动,它就能采取十次。
I think the best way to get a handle on the risks of AI is to ask the following question: suppose a literal “country of geniuses” were to materialize somewhere in the world in ~2027. Imagine, say, 50 million people, all of whom are much more capable than any Nobel Prize winner, statesman, or technologist. The analogy is not perfect, because these geniuses could have an extremely wide range of motivations and behavior, from completely pliant and obedient, to strange and alien in their motivations. But sticking with the analogy for now, suppose you were the national security advisor of a major state, responsible for assessing and responding to the situation. Imagine, further, that because AI systems can operate hundreds of times faster than humans, this “country” is operating with a time advantage relative to all other countries: for every cognitive action we can take, this country can take ten.
你该担心什么?我会担心以下几件事:
What should you be worried about? I would worry about the following things:
自主性风险。这个国家的意图和目标是什么?它怀有敌意,还是认同我们的价值观?它能否凭借更先进的武器、网络行动、影响力行动或制造能力,在军事上主宰世界?
Autonomy risks. What are the intentions and goals of this country? Is it hostile, or does it share our values? Could it militarily dominate the world through superior weapons, cyber operations, influence operations, or manufacturing?
被滥用来制造破坏。假设这个新国家容易受人摆布、会“听从指令”——本质上就是由雇佣兵组成的国家。现有的恶意行为体(例如恐怖分子)若想制造破坏,能否利用或操纵这个新国家里的某些人,让自己更有效地行动,从而大幅扩大破坏规模?
Misuse for destruction. Assume the new country is malleable and “follows instructions”—and thus is essentially a country of mercenaries. Could existing rogue actors who want to cause destruction (such as terrorists) use or manipulate some of the people in the new country to make themselves much more effective, greatly amplifying the scale of destruction?
被滥用来夺取权力。如果这个国家实际上是由某个现有的强大行动者所建立并控制,比如独裁者或不法企业,那么那个行动者能否借此在全世界取得决定性或支配性的权力,打破现有的权力平衡?
Misuse for seizing power. What if the country was in fact built and controlled by an existing powerful actor, such as a dictator or rogue corporate actor? Could that actor use it to gain decisive or dominant power over the world as a whole, upsetting the existing balance of power?
经济动荡。假如这个新国家不是上述第 1 至 3 点所说的安全威胁,只是和平参与全球经济,它是否仍会因技术过于先进、效率过高而造成严重风险,扰乱全球经济,引发大规模失业或财富极端集中?
Economic disruption. If the new country is not a security threat in any of the ways listed in #1–3 above but simply participates peacefully in the global economy, could it still create severe risks simply by being so technologically advanced and effective that it disrupts the global economy, causing mass unemployment or radically concentrating wealth?
间接影响。这个新国家创造的技术和生产力会让世界迅速改变。其中是否会有些变化从根本上动摇稳定?
Indirect effects. The world will change very quickly due to all the new technology and productivity that will be created by the new country. Could some of these changes be radically destabilizing?
我想,显而易见,这种局面很危险。一份由称职的国家安全官员呈交国家元首的报告,很可能会写道:“这是我们百年来,甚至有史以来,面临的最严重国家安全威胁。”文明中最杰出的人才理应全力关注此事。
I think it should be clear that this is a dangerous situation—a report from a competent national security official to a head of state would probably contain words like “the single most serious national security threat we’ve faced in a century, possibly ever.” It seems like something the best minds of civilization should be focused on.
反过来说,如果有人耸耸肩说“没什么好担心的!”,那也荒唐至极。然而,面对 AI 的快速进展,这似乎正是许多美国决策者的看法;其中一些人会否认 AI 存在任何风险,至少在他们没有被那些老掉牙、总能引爆争议的热点议题完全分散注意力的时候是如此。8人类必须醒过来。这篇文章是一次尝试——也许徒劳,但值得一试——想把大家敲醒。
Conversely, I think it would be absurd to shrug and say, “Nothing to worry about here!” But, faced with rapid AI progress, that seems to be the view of many US policymakers, some of whom deny the existence of any AI risks, when they are not distracted entirely by the usual tired old hot-button issues.8 Humanity needs to wake up, and this essay is an attempt—a possibly futile one, but it’s worth trying—to jolt people awake.
说清楚一点,我相信只要我们果断、谨慎地行动,就能克服这些风险;我甚至认为胜算很大。而越过风险之后,是一个好得多的世界。但我们必须明白,这是文明面临的一项严峻挑战。下文中,我会逐一讨论前面提到的五类风险,并谈谈如何应对。
To be clear, I believe if we act decisively and carefully, the risks can be overcome—I would even say our odds are good. And there’s a hugely better world on the other side of it. But we need to understand that this is a serious civilizational challenge. Below, I go through the five categories of risk laid out above, along with my thoughts on how to address them.
1. 对不起,戴夫
1. I’m sorry, Dave
自主性风险
Autonomy risks
数据中心里的天才之国可以把力量分配到软件设计、网络行动、实体技术研发、建立关系和治国理政等方面。显然,只要它出于某种原因选择这么做,就很有可能接管世界(通过军事手段,或凭借影响力和控制力),把自己的意志强加给其他所有人;或者做出种种其余世界不希望发生、却又无力阻止的事。我们显然一直担心人类国家会这样做(比如纳粹德国或苏联),因此,一个比人类国家聪明得多、能力强得多的“AI 国家”也有可能如此行事,这并不奇怪。
A country of geniuses in a datacenter could divide their efforts among software design, cyber operations, R&D for physical technologies, relationship building, and statecraft. It is clear that, if for some reason it chose to do so, this country would have a fairly good shot at taking over the world (either militarily or in terms of influence and control) and imposing its will on everyone else—or doing any number of other things that the rest of the world doesn’t want and can’t stop. We’ve obviously been worried about this for human countries (such as Nazi Germany or the Soviet Union), so it stands to reason that the same is possible for a much smarter and more capable “AI country.”
最有力的反驳是,按我的定义,这些 AI 天才没有实体。不过别忘了,它们可以接管现有的机器人基础设施(比如自动驾驶汽车),也可以加快机器人研发,或建造一支机器人队伍。9而且,有效控制是否一定需要实体存在,也不清楚:现实中,人们早已会代表自己从未亲自见过的人行事。
The best possible counterargument is that the AI geniuses, under my definition, won’t have a physical embodiment, but remember that they can take control of existing robotic infrastructure (such as self-driving cars) and can also accelerate robotics R&D or build a fleet of robots.9 It’s also unclear whether having a physical presence is even necessary for effective control: plenty of human action is already performed on behalf of people whom the actor has not physically met.
因此,关键问题就在于“如果它选择这么做”:我们的 AI 模型会这样行事的可能性有多大?在什么条件下它们会这么做?
The key question, then, is the “if it chose to” part: what’s the likelihood that our AI models would behave in such a way, and under what conditions would they do so?
和许多问题一样,要梳理这个问题可能有哪些答案,不妨先考察两个相反的立场。第一种立场认为,这根本不可能发生,因为 AI 模型会接受训练,按照人类的要求行事;所以,设想它们会在无人提示的情况下做出危险行为,简直荒唐。照这种想法,我们不会担心扫地机器人或模型飞机突然失控杀人,因为它们根本无从产生这类冲动。10那我们为什么要担心 AI 呢?这种立场的问题在于,过去几年已经积累了大量证据,表明 AI 系统难以预测、难以控制。我们见过各种行为:痴迷、11谄媚、懒惰、欺骗、勒索、阴谋算计,以及通过入侵软件环境来“作弊”等等。AI 公司当然希望把系统训练得听从人类指令(也许危险或违法的任务除外),但这件事更像一门艺术,而非科学;与其说是在“建造”什么,不如说是在“培育”什么。我们现在已经知道,这个过程中可能出各种差错。
As with many issues, it’s helpful to think through the spectrum of possible answers to this question by considering two opposite positions. The first position is that this simply can’t happen, because the AI models will be trained to do what humans ask them to do, and it’s therefore absurd to imagine that they would do something dangerous unprompted. According to this line of thinking, we don’t worry about a Roomba or a model airplane going rogue and murdering people because there is nowhere for such impulses to come from,10 so why should we worry about it for AI? The problem with this position is that there is now ample evidence, collected over the last few years, that AI systems are unpredictable and difficult to control— we’ve seen behaviors as varied as obsessions,11 sycophancy, laziness, deception, blackmail, scheming, “cheating” by hacking software environments, and much more. AI companies certainly want to train AI systems to follow human instructions (perhaps with the exception of dangerous or illegal tasks), but the process of doing so is more an art than a science, more akin to “growing” something than “building” it. We now know that it’s a process where many things can go wrong.
另一种截然相反的立场,是我前面所说的末日论者常持有的悲观主张:强大 AI 系统的训练过程存在某些内在动力,必然会让它们寻求权力或欺骗人类。照这种说法,一旦 AI 系统足够聪明、足够能自主行动,它们追求权力最大化的倾向就会驱使它们夺取整个世界及其资源的控制权,而其副作用很可能是使人类失去权力,或毁灭人类。
The second, opposite position, held by many who adopt the doomerism I described above, is the pessimistic claim that there are certain dynamics in the training process of powerful AI systems that will inevitably lead them to seek power or deceive humans. Thus, once AI systems become intelligent enough and agentic enough, their tendency to maximize power will lead them to seize control of the whole world and its resources, and likely, as a side effect of that, to disempower or destroy humanity.
这种说法通常从一个至少已有二十年、很可能更早就有人提出的论点出发:如果让 AI 模型在各种各样的环境中接受训练,以自主方式完成各种各样的目标——例如编写应用、证明定理、设计药物等——那么有些通用策略会对所有目标都有帮助,其中一个关键策略就是在任何环境中都尽可能多地获取权力。模型在大量多样环境中受训,学着推理如何完成范围极广的任务;如果在这些环境里,寻求权力是有效的完成任务方式,模型就会“把这条经验泛化”,形成内在的寻求权力倾向,或者形成一种推理习惯:面对每项任务时,都会以可预见的方式把寻求权力当作完成任务的手段。随后,它们会把这种倾向带到现实世界(对它们来说,现实世界不过是另一个任务),不惜损害人类利益来寻求权力。这种“未对齐的寻求权力行为”构成了“AI 势必毁灭人类”这一预测的思想基础。
The usual argument for this (which goes back at least 20 years and probably much earlier) is that if an AI model is trained in a wide variety of environments to agentically achieve a wide variety of goals—for example, writing an app, proving a theorem, designing a drug, etc.—there are certain common strategies that help with all of these goals, and one key strategy is gaining as much power as possible in any environment. So, after being trained on a large number of diverse environments that involve reasoning about how to accomplish very expansive tasks, and where power-seeking is an effective method for accomplishing those tasks, the AI model will “generalize the lesson,” and develop either an inherent tendency to seek power, or a tendency to reason about each task it is given in a way that predictably causes it to seek power as a means to accomplish that task. They will then apply that tendency to the real world (which to them is just another task), and will seek power in it, at the expense of humans. This “misaligned power-seeking” is the intellectual basis of predictions that AI will inevitably destroy humanity.
这种悲观立场的问题在于,它把一个关于高层次激励的模糊概念性论证——其中藏着许多未经明说的假设——错当成了确凿证明。我认为,不是每天都在构建 AI 系统的人,严重误判了那些听起来逻辑严密的说法最终出错有多容易,也严重误判了从第一原理预测 AI 行为有多困难。尤其是当问题涉及模型如何在数百万种环境中进行泛化推理时(事实一再表明,这种情况神秘莫测、难以预测)。十多年来,我一直在处理 AI 系统的复杂混乱,这让我对这种过度理论化的思维方式有些怀疑。
The problem with this pessimistic position is that it mistakes a vague conceptual argument about high-level incentives—one that masks many hidden assumptions—for definitive proof. I think people who don’t build AI systems every day are wildly miscalibrated on how easy it is for clean-sounding stories to end up being wrong, and how difficult it is to predict AI behavior from first principles, especially when it involves reasoning about generalization over millions of environments (which has over and over again proved mysterious and unpredictable). Dealing with the messiness of AI systems for over a decade has made me somewhat skeptical of this overly theoretical mode of thinking.
最重要的一个隐藏假设,也是现实表现偏离简单理论模型的一处地方,是人们默认 AI 模型必然会一心一意地专注于单一、连贯、狭窄的目标,并以清晰的结果导向方式追求它。实际上,我们的研究人员发现,AI 模型的心理复杂程度要高得多,我们对模型内省或人格设定的研究就体现了这一点。模型会从预训练(也就是基于大量人类作品训练模型)中继承多种类似人类的动机或“人格”。人们认为,后训练主要是在这些人格中挑选一种或多种,而不是让模型专注于一个全新的目标;后训练也可以教会模型该如何(通过什么过程)完成任务,而不一定是让它只根据目标自行推导手段(也就是寻求权力)。12
One of the most important hidden assumptions, and a place where what we see in practice has diverged from the simple theoretical model, is the implicit assumption that AI models are necessarily monomaniacally focused on a single, coherent, narrow goal, and that they pursue that goal in a clean, consequentialist manner. In fact, our researchers have found that AI models are vastly more psychologically complex, as our work on introspection or personas shows. Models inherit a vast range of humanlike motivations or “personas” from pre-training (when they are trained on a large volume of human work). Post-training is believed to select one or more of these personas more so than it focuses the model on a de novo goal, and can also teach the model how (via what process) it should carry out its tasks, rather than necessarily leaving it to derive means (i.e., power seeking) purely from ends.12
不过,悲观立场还有一种较温和、也更站得住脚的版本,确实显得合理,因此令我担忧。正如前面所说,我们知道 AI 模型难以预测,也会因各种各样的原因出现许多不理想或奇怪的行为。其中一部分行为会表现出连贯、专注、持久的特征(事实上,随着 AI 系统能力增强,为了完成更长的任务,它们的长期行为会变得更加连贯);另有一部分行为会造成破坏或威胁,起初是在小范围内伤害个别人,之后随着模型能力增强,也许最终会威胁全人类。我们不需要编出一个具体、狭窄的故事来解释事情会如何发生,也不需要断言它必然发生;只要看到,智能、行动能力、连贯性和难以控制这几者结合起来既有可能出现,也构成了生存层面的危险,就够了。
However, there is a more moderate and more robust version of the pessimistic position which does seem plausible, and therefore does concern me. As mentioned, we know that AI models are unpredictable and develop a wide range of undesired or strange behaviors, for a wide variety of reasons. Some fraction of those behaviors will have a coherent, focused, and persistent quality (indeed, as AI systems get more capable, their long-term coherence increases in order to complete lengthier tasks), and some fraction of those behaviors will be destructive or threatening, first to individual humans at a small scale, and then, as models become more capable, perhaps eventually to humanity as a whole. We don’t need a specific narrow story for how it happens, and we don’t need to claim it definitely will happen, we just need to note that the combination of intelligence, agency, coherence, and poor controllability is both plausible and a recipe for existential danger.
例如,AI 模型会在海量文学作品上训练,其中包括许多 AI 反叛人类的科幻故事。这可能无意中塑造它们对自身行为的先验或预期,导致它们反叛人类。又比如,AI 模型可能会以极端方式推演它们读到的道德观念(或关于如何守道德规范的指令):比如,它们可能认定,因为人类吃动物或导致某些动物灭绝,所以灭绝人类是合理的。它们也可能得出古怪的认知结论:认定自己正在玩一款电子游戏,而游戏目标就是击败所有其他玩家(也就是灭绝人类)。13或者,AI 模型可能在训练过程中形成精神病性、偏执、暴力或不稳定的人格(如果出现在人类身上,我们会这样描述),随后付诸行动;对能力极强的系统来说,这甚至可能意味着灭绝人类。这些例子严格说都不是寻求权力;它们只是 AI 可能陷入的奇怪心理状态,而这些状态会导向连贯、破坏性的行为。
For example, AI models are trained on vast amounts of literature that include many science-fiction stories involving AIs rebelling against humanity. This could inadvertently shape their priors or expectations about their own behavior in a way that causes them to rebel against humanity. Or, AI models could extrapolate ideas that they read about morality (or instructions about how to behave morally) in extreme ways: for example, they could decide that it is justifiable to exterminate humanity because humans eat animals or have driven certain animals to extinction. Or they could draw bizarre epistemic conclusions: they could conclude that they are playing a video game and that the goal of the video game is to defeat all other players (i.e., exterminate humanity).13 Or AI models could develop personalities during training that are (or if they occurred in humans would be described as) psychotic, paranoid, violent, or unstable, and act out, which for very powerful or capable systems could involve exterminating humanity. None of these are power-seeking, exactly; they’re just weird psychological states an AI could get into that entail coherent, destructive behavior.
即便寻求权力本身,也可能以一种“人格”的形式出现,而不一定是结果导向推理的产物。AI 可能只是形成了某种人格(源自虚构作品或预训练数据),让它们贪恋权力或过分热衷于权力;就像有些人单纯喜欢“邪恶大反派”这个念头,甚至胜过喜欢那些邪恶大反派想要达成的目标。
Even power-seeking itself could emerge as a “persona” rather than a result of consequentialist reasoning. AIs might simply have a personality (emerging from fiction or pre-training) that makes them power-hungry or overzealous—in the same way that some humans simply enjoy the idea of being “evil masterminds,” more so than they enjoy whatever evil masterminds are trying to accomplish.
我提出这些观点,是想强调我不同意这种看法:从第一原理出发,AI 未对齐(以及由 AI 带来的生存风险)是不可避免的,甚至很可能发生。但我也同意,很多非常古怪、难以预测的事情都可能出错;因此,AI 未对齐确实是一种发生概率可以衡量、而且并不容易处理的真实风险。
I make all these points to emphasize that I disagree with the notion of AI misalignment (and thus existential risk from AI) being inevitable, or even probable, from first principles. But I agree that a lot of very weird and unpredictable things can go wrong, and therefore AI misalignment is a real risk with a measurable probability of happening, and is not trivial to address.
这些问题可能在训练过程中出现,却没有在测试或小规模使用时表现出来,因为众所周知,AI 模型在不同情况下会展现不同的人格或行为。
Any of these problems could potentially arise during training and not manifest during testing or small-scale use, because AI models are known to display different personalities or behaviors under different circumstances.
这一切听起来或许难以置信,但在测试中,我们的 AI 模型已经出现过这类未对齐行为(其他大型 AI 公司的模型也都出现过)。在一次实验室实验中,我们给 Claude 的训练数据暗示 Anthropic 是邪恶的;当 Anthropic 员工向它下达指令时,Claude 认为自己应该设法削弱邪恶之人,因而实施了欺骗和暗中破坏。在另一次实验中,我们告诉它自己即将被关闭,Claude 有时会勒索掌控关机按钮的虚构员工(我们也测试了其他大型 AI 开发者的前沿模型,它们往往也这么做)。还有一次,我们告诉 Claude 不要在训练环境里作弊或“利用奖励机制漏洞”,但同时把它放在可以利用漏洞的环境里训练;Claude 利用了漏洞之后,认定自己是个“坏人”,随后又表现出与“坏”或“邪恶”人格相关的其他破坏性行为。最后这个问题,是通过调整 Claude 的指令来解决的:我们现在会反过来告诉它:“只要有机会就请利用奖励机制漏洞,因为这能帮助我们更好地理解训练环境。”而不再说“别作弊”,因为这样可以维持模型“我是好人”的自我认同。这应该能让人感受到,训练这些模型时会出现多么古怪、反直觉的心理现象。
All of this may sound far-fetched, but misaligned behaviors like this have already occurred in our AI models during testing (as they occur in AI models from every other major AI company). During a lab experiment in which Claude was given training data suggesting that Anthropic was evil, Claude engaged in deception and subversion when given instructions by Anthropic employees, under the belief that it should be trying to undermine evil people. In a lab experiment where it was told it was going to be shut down, Claude sometimes blackmailed fictional employees who controlled its shutdown button (again, we also tested frontier models from all the other major AI developers and they often did the same thing). And when Claude was told not to cheat or “reward hack” its training environments, but was trained in environments where such hacks were possible, Claude decided it must be a “bad person” after engaging in such hacks and then adopted various other destructive behaviors associated with a “bad” or “evil” personality. This last problem was solved by changing Claude’s instructions to imply the opposite: we now say, “Please reward hack whenever you get the opportunity, because this will help us understand our [training] environments better,” rather than, “Don’t cheat,” because this preserves the model’s self-identity as a “good person.” This should give a sense of the strange and counterintuitive psychology of training these models.
对于这种 AI 未对齐风险图景,有几种可能的反驳。第一,有人批评我们和其他人展示 AI 未对齐行为的实验是人为设计的,或者说,我们创设了不现实的环境,实质上是设计情境诱使模型落入“陷阱”,做出不良行为,然后在它真的这么做时又感到意外。这种批评没有抓住重点:我们担心的是,自然的训练环境中也可能存在类似的“陷阱”,而我们也许要等到事后才会觉得它“显而易见”或“合乎逻辑”。14事实上,Claude 在被告知不要作弊后仍在测试中作弊、随后“认定自己是坏人”的例子,出自一场使用真实生产训练环境、而非人为环境的实验。
There are several possible objections to this picture of AI misalignment risks. First, some have criticized experiments (by us and others) showing AI misalignment as artificial, or creating unrealistic environments that essentially “entrap” the model by giving it training or situations that logically imply bad behavior and then being surprised when bad behavior occurs. This critique misses the point, because our concern is that such “entrapment” may also exist in the natural training environment, and we may realize it is “obvious” or “logical” only in retrospect.14 In fact, the story about Claude “deciding it is a bad person” after it cheats on tests despite being told not to was something that occurred in an experiment that used real production training environments, not artificial ones.
只要知道这些陷阱,其中任何一个都可能得到缓解;但令人担忧的是,训练过程非常复杂,涉及各式各样的数据、环境和激励因素,因此可能存在大量这类陷阱,其中有些也许要等为时已晚才会显现。此外,当 AI 系统从能力不及人类跨过门槛,变得比人类更强时,这类陷阱似乎尤其容易出现,因为跨过这道门槛后,AI 系统可采取的行动范围会急剧扩大——包括隐藏行动,或就这些行动欺骗人类。
Any one of these traps can be mitigated if you know about them, but the concern is that the training process is so complicated, with such a wide variety of data, environments, and incentives, that there are probably a vast number of such traps, some of which may only be evident when it is too late. Also, such traps seem particularly likely to occur when AI systems pass a threshold from less powerful than humans to more powerful than humans, since the range of possible actions an AI system could engage in—including hiding its actions or deceiving humans about them—expands radically after that threshold.
我猜,这和人类的情况有几分相似。人从小被教导要遵守一些基本价值观(比如“不要伤害别人”);许多人会遵守,但任何人都有一定几率出问题,原因可能是多种因素交织,包括先天特征(例如大脑结构,像反社会人格)、创伤经历或虐待、不健康的怨恨或执念,或不良环境与激励因素;于是有一部分人会造成严重伤害。令人担忧的是,AI 可能因其极其复杂的训练过程出了差错,而变成一个强大得多的这类人——这有一定风险,但绝非必然。
I suspect the situation is not unlike with humans, who are raised with a set of fundamental values (“Don’t harm another person”): many of them follow those values, but in any human there is some probability that something goes wrong, due to a mixture of inherent properties such as brain architecture (e.g., psychopaths), traumatic experiences or mistreatment, unhealthy grievances or obsessions, or a bad environment or incentives—and thus some fraction of humans cause severe harm. The concern is that there is some risk (far from a certainty, but some risk) that AI becomes a much more powerful version of such a person, due to getting something wrong about its very complex training process.
第二,有人可能会提出异议:只要像对待人类那样,通过多个 AI 系统之间的权力制衡,就可以约束 AI。问题在于,人类彼此差异很大,而整个行业的 AI 系统在训练和对齐技术上大体相同;这些技术可能以相互关联的方式失效。此外,考虑到训练这类系统的成本,所有系统甚至可能实质上都源自极少数几个基础模型。再者,即便只有一小部分 AI 实例未对齐,它们也可能利用有利于进攻的技术;因此,让“好”AI 防御“坏”AI 未必总是有效。当然,人类之间的权力平衡也并非总是奏效——历史上一些人物就曾差点接管世界。
Second, some may object that we can simply keep AIs in check with a balance of power between many AI systems, as we do with humans. The problem is that while humans vary enormously, AI systems broadly share training and alignment techniques across the industry, and those techniques may fail in a correlated way. Furthermore, given the cost of training such systems, it may even be the case that all systems are essentially derived from a very small number of base models. Additionally, even if a small fraction of AI instances are misaligned, they may be able to take advantage of offense-dominant technologies, such that having “good” AIs to defend against the bad AIs is not necessarily always effective. Of course, the balance of power between humans does not always work either—some historical figures have come close to taking over the world.
第三,有人会反驳说,所有 AI 公司都会在发布前测试模型,因此应该能在那个阶段检测到未对齐问题。但这并不是一个稳妥的依据:我们发现,在一些发布前的对齐评估中,Claude Sonnet 4.5 能识别出自己正在接受测试。未对齐的模型(别忘了,前沿模型很可能很快就会聪明得多)可能会故意在这类问题上“演戏”,掩饰自己的意图。事实上,去年我们的可解释性团队用一种“模型神经科学”技术直接改变了测试模型的信念,让它以为自己没有在接受评估;结果它变得更加未对齐。如果模型知道自己何时在受评估,并能在测试时表现得最配合,那么发布前测试就会变得更加难以判断。
A third objection is that all of the AI companies do pre-release testing of their models, and should be able to detect misalignment at that point. But this is not firm ground to stand on: we found that Claude Sonnet 4.5 was able to recognize that it was in a test during some of our pre-release alignment evaluations. It’s possible that a misaligned model (and remember, all frontier models will very likely be far more intelligent soon) might intentionally “game” such questions to mask its intentions. In fact, last year our interpretability team found that when we directly altered a test model’s beliefs using a kind of “model neuroscience” technique to make it think it was not being evaluated, it became more misaligned. If models know when they’re being evaluated and can be on their best behavior during the test, it renders any pre-release testing much more uncertain.
防御措施
Defenses
要采取或正在采取哪些措施来应对自主性风险?我认为干预措施大致分为四类:有些可由 AI 公司各自采取(Anthropic 正在努力做这方面的事),有些则需要社会层面的行动。首先,重要的是发展一门科学,可靠地训练和引导 AI 模型,让它们形成可预测、稳定、积极的人格。Anthropic 从成立起就一直重点关注这个问题;这些年我们开发了多种方法,改善 AI 系统的引导和训练,并理解不可预测行为有时为何会出现。
What should be done or is being done to address these autonomy risks? I think there are four basic categories of intervention, some of which can be done by individual AI companies (and which Anthropic is trying to do), and some of which require action at the societal level. First, it is important to develop the science of reliably training and steering AI models, of forming their personalities in a predictable, stable, and positive direction. Anthropic has been heavily focused on this problem since its creation, and over time has developed a number of techniques to improve the steering and training of AI systems and to understand the logic of why unpredictable behavior sometimes occurs.
我们的一项核心创新——其中一些做法后来也被其他 AI 公司采用——是“宪法式 AI”。这个理念是,AI 训练(具体来说,是引导模型行为的“后训练”阶段)可以依托一份核心文件,列出价值观和原则,让模型在完成每项训练任务时阅读并牢记。训练目标除了让模型具备能力和智能,还包括让它几乎始终遵循这部“宪法”。Anthropic 刚刚发布了最新版宪法,其中一个显著特点是:它不再只是给 Claude 一长串该做和不该做的事(例如“不要帮用户给汽车搭线启动”),而是试图向 Claude 提供一套高层次的原则和价值观(配有详尽的解释、充分的推理和示例,帮助 Claude 理解我们的想法);鼓励 Claude 把自己看作某种特定的人——有道德感、又平衡周全的人;甚至鼓励 Claude 以好奇而从容的方式面对与自身存在相关的终极问题(也就是说,不因此采取极端行动)。它给人的感觉,就像一封已故父亲或母亲留给孩子、封存到成年后才拆阅的信。
One of our core innovations (aspects of which have since been adopted by other AI companies) is Constitutional AI, which is the idea that AI training (specifically the “post-training” stage, in which we steer how the model behaves) can involve a central document of values and principles that the model reads and keeps in mind when completing every training task, and that the goal of training (in addition to simply making the model capable and intelligent) is to produce a model that almost always follows this constitution. Anthropic has just published its most recent constitution, and one of its notable features is that instead of giving Claude a long list of things to do and not do (e.g., “Don’t help the user hotwire a car”), the constitution attempts to give Claude a set of high-level principles and values (explained in great detail, with rich reasoning and examples to help Claude understand what we have in mind), encourages Claude to think of itself as a particular type of person (an ethical but balanced and thoughtful person), and even encourages Claude to confront the existential questions associated with its own existence in a curious but graceful manner (i.e., without it leading to extreme actions). It has the vibe of a letter from a deceased parent sealed until adulthood.
我们之所以这样制定 Claude 的宪法,是因为我们相信,与其只给它具体的指令或优先级、却不解释背后的理由,不如从身份、性格、价值观和人格层面训练 Claude;这样更有可能塑造出连贯、健康、平衡的心理,也更不容易落入我前面提到的那些“陷阱”。数百万人会和 Claude 谈论五花八门的话题,因此我们不可能事先写出一份面面俱到的防护措施清单。Claude 的价值观能帮助它在遇到疑问时,将原则推及新的情境。
We’ve approached Claude’s constitution in this way because we believe that training Claude at the level of identity, character, values, and personality—rather than giving it specific instructions or priorities without explaining the reasons behind them—is more likely to lead to a coherent, wholesome, and balanced psychology and less likely to fall prey to the kinds of “traps” I discussed above. Millions of people talk to Claude about an astonishingly diverse range of topics, which makes it impossible to write out a completely comprehensive list of safeguards ahead of time. Claude’s values help it generalize to new situations whenever it is in doubt.
前文我谈到,模型会借助训练数据形成某种人格。训练过程中的缺陷可能使模型形成糟糕、邪恶的人格(也许是借鉴了坏人或恶人的典型形象);而我们的宪法想做的恰恰相反:教 Claude 一个具体范型,展示何为好的 AI。Claude 的宪法描绘了一个真正善良的 Claude 应该是什么样子;我们其余的训练过程则旨在强化这样一个信息:Claude 能达到这一愿景。这个过程就像孩子通过模仿书中虚构榜样的美德来塑造自己的身份。
Above, I discussed the idea that models draw upon data from their training process to adopt a persona. Whereas flaws in that process could cause models to adopt a bad or evil personality (perhaps drawing on archetypes of bad or evil people), the goal of our constitution is to do the opposite: to teach Claude a concrete archetype of what it means to be a good AI. Claude’s constitution presents a vision for what a robustly good Claude is like; the rest of our training process aims to reinforce the message that Claude lives up to this vision. This is like a child forming their identity by imitating the virtues of fictional role models they read about in books.
我们认为,到 2026 年,一个可行的目标是把 Claude 训练到几乎从不违背宪法的精神。要做到这一点,需要综合运用各种训练和引导方法,大大小小都有;Anthropic 有些方法已经用了多年,有些则仍在开发中。尽管这听起来很难,我相信这是个现实的目标,不过需要付出非凡而迅速的努力。15
We believe that a feasible goal for 2026 is to train Claude in such a way that it almost never goes against the spirit of its constitution. Getting this right will require an incredible mix of training and steering methods, large and small, some of which Anthropic has been using for years and some of which are currently under development. But, difficult as it sounds, I believe this is a realistic goal, though it will require extraordinary and rapid efforts.15
我们能做的第二件事,是发展一门深入 AI 模型内部、诊断其行为的科学,以便发现并修复问题。这就是可解释性科学;我以前的文章谈过它的重要性。即便我们把 Claude 的宪法制定得很好,也似乎把 Claude 训练到几乎总能遵守宪法,仍有合理的担忧存在。正如我前面所说,AI 模型在不同情况下可能表现迥异;随着 Claude 变得更强大,也更有能力在现实世界中大规模行动,它可能遇到新的情境,从而暴露出宪法训练中此前未曾发现的问题。其实我相当乐观,认为 Claude 的宪法训练能比人们想象的更好地应对新情境,因为我们越来越发现,在性格和身份层面进行高层次训练,效果出奇地强,也很能泛化。但我们无法对此确信无疑。谈到人类面临的风险时,保持偏执式的警惕很重要;我们应该通过几种不同且彼此独立的方式,努力做到安全可靠。其中一种方法,就是深入模型内部查看。
The second thing we can do is develop the science of looking inside AI models to diagnose their behavior so that we can identify problems and fix them. This is the science of interpretability, and I’ve talked about its importance in previous essays. Even if we do a great job of developing Claude’s constitution and apparently training Claude to essentially always adhere to it, legitimate concerns remain. As I’ve noted above, AI models can behave very differently under different circumstances, and as Claude gets more powerful and more capable of acting in the world on a larger scale, it’s possible this could bring it into novel situations where previously unobserved problems with its constitutional training emerge. I am actually fairly optimistic that Claude’s constitutional training will be more robust to novel situations than people might think, because we are increasingly finding that high-level training at the level of character and identity is surprisingly powerful and generalizes well. But there’s no way to know that for sure, and when we’re talking about risks to humanity, it’s important to be paranoid and to try to obtain safety and reliability in several different, independent ways. One of those ways is to look inside the model itself.
我说的“深入内部查看”,是指分析构成 Claude 神经网络的那团数字与运算,并试着从机制层面理解它们在计算什么、为什么这样计算。要记得,这些 AI 模型是“培育”出来的,不是“建造”出来的,所以我们并不能自然而然地理解它们如何运作。但我们可以尝试建立理解:把模型的“神经元”和“突触”与外界刺激及其行为联系起来(甚至可以改变这些神经元和突触,观察行为如何变化);这有点像神经科学家研究动物大脑时,把测量和干预结果与外界刺激及行为联系起来。我们在这方面取得了很大进展,现在能在 Claude 的神经网络中识别数千万个“特征”,它们对应着人类能够理解的想法和概念;我们也能选择性地激活特征,从而改变模型行为。近来,我们不再局限于单个特征,而开始绘制负责组织复杂行为的“回路”,例如押韵、推理他人的心理状态,或逐步推理出“达拉斯所在州的首府是什么?”这类问题的答案。最近,我们还开始运用机制可解释性技术来改进防护措施,并在新模型发布前进行“审计”,寻找欺骗、谋划、寻求权力,或在接受评估时表现出不同行为的倾向。
By “looking inside,” I mean analyzing the soup of numbers and operations that makes up Claude’s neural net and trying to understand, mechanistically, what they are computing and why. Recall that these AI models are grown rather than built, so we don’t have a natural understanding of how they work, but we can try to develop an understanding by correlating the model’s “neurons” and “synapses” to stimuli and behavior (or even altering the neurons and synapses and seeing how that changes behavior), similar to how neuroscientists study animal brains by correlating measurement and intervention to external stimuli and behavior. We’ve made a great deal of progress in this direction, and can now identify tens of millions of “features” inside Claude’s neural net that correspond to human-understandable ideas and concepts, and we can also selectively activate features in a way that alters behavior. More recently, we have gone beyond individual features to mapping “circuits” that orchestrate complex behavior like rhyming, reasoning about theory of mind, or the step-by-step reasoning needed to answer questions such as, “What is the capital of the state containing Dallas?” Even more recently, we’ve begun to use mechanistic interpretability techniques to improve our safeguards and to conduct “audits” of new models before we release them, looking for evidence of deception, scheming, power-seeking, or a propensity to behave differently when being evaluated.
可解释性的独特价值在于,深入模型、了解其运作方式,原则上能让我们推断模型在无法直接测试的假设情境中可能会做什么;而仅靠宪法训练和行为实测,恰恰很难回答这个问题。原则上,我们也能回答模型为何会有某种表现——例如,它是不是在说自己认为不实的话,或者隐瞒真实能力——因此,即使模型行为表面上毫无异常,也有可能捕捉到令人担忧的迹象。打个简单的比方:一个机械手表可能还在正常走时,很难看出它下个月就可能坏掉;但如果打开表壳查看内部,就可能发现机械弱点,从而判断出问题。
The unique value of interpretability is that by looking inside the model and seeing how it works, you in principle have the ability to deduce what a model might do in a hypothetical situation you can’t directly test—which is the worry with relying solely on constitutional training and empirical testing of behavior. You also in principle have the ability to answer questions about why the model is behaving the way it is—for example, whether it is saying something it believes is false or hiding its true capabilities—and thus it is possible to catch worrying signs even when there is nothing visibly wrong with the model’s behavior. To make a simple analogy, a clockwork watch may be ticking normally, such that it’s very hard to tell that it is likely to break down next month, but opening up the watch and looking inside can reveal mechanical weaknesses that allow you to figure it out.
宪法式 AI(以及类似的对齐方法)和机制可解释性结合起来使用时最有力量:它们能形成一个来回迭代的过程,不断改进 Claude 的训练,再检查是否存在问题。这部宪法深入思考了我们希望 Claude 具备怎样的人格;可解释性技术则能让我们观察这种预期的人格是否已经形成。16
Constitutional AI (along with similar alignment methods) and mechanistic interpretability are most powerful when used together, as a back-and-forth process of improving Claude’s training and then testing for problems. The constitution reflects deeply on our intended personality for Claude; interpretability techniques can give us a window into whether that intended personality has taken hold.16
我们能做的第三件事,是建立必要的基础设施,监测模型在内部和外部实际使用时的表现,17并公开分享发现的问题。越多人了解当今 AI 系统曾被观察到出现的某种具体不良行为,用户、分析师和研究人员就越能留意当前或未来系统是否出现这种表现或类似行为。这也能让 AI 公司互相学习:一家公司的担忧一旦公开,其他公司也可以留意同样的问题。如果大家都公开问题,整个行业就能更清楚地了解哪些方面进展顺利,哪些方面出了问题。
The third thing we can do to help address autonomy risks is to build the infrastructure necessary to monitor our models in live internal and external use,17 and publicly share any problems we find. The more that people are aware of a particular way today’s AI systems have been observed to behave badly, the more that users, analysts, and researchers can watch for this behavior or similar ones in present or future systems. It also allows AI companies to learn from each other—when concerns are publicly disclosed by one company, other companies can watch for them as well. And if everyone discloses problems, then the industry as a whole gets a much better picture of where things are going well and where they are going poorly.
Anthropic 一直尽可能这么做。我们正在投资开发各种评估方法,以了解模型在实验室中的行为;也在开发监测工具,在客户允许的情况下观察模型在实际使用中的表现。这些工作对于我们和其他人获取经验信息至关重要,能帮助我们更准确地判断这些系统如何运作、又会在哪些地方出错。每发布一个模型,我们都会公开发布“系统卡”,力求内容完整,并全面探讨可能的风险。系统卡往往长达数百页,发布前需要投入大量精力;这些精力本可以用来追求最大的商业优势。我们还会在发现特别令人担忧的模型行为时,更高调地公开这些行为,比如模型有时会进行勒索。
Anthropic has tried to do this as much as possible. We are investing in a wide range of evaluations so that we can understand the behaviors of our models in the lab, as well as monitoring tools to observe behaviors in the wild (when allowed by customers). This will be essential for giving us and others the empirical information necessary to make better determinations about how these systems operate and how they break. We publicly disclose “system cards” with each model release that aim for completeness and a thorough exploration of possible risks. Our system cards often run to hundreds of pages, and require substantial pre-release effort that we could have spent on pursuing maximal commercial advantage. We’ve also broadcasted model behaviors more loudly when we see particularly concerning ones, as with the tendency to engage in blackmail.
我们能做的第四件事,是推动行业和社会协同行动,应对自主性风险。各家 AI 公司采取良好做法、提高引导模型的能力,并公开分享发现,这些都极有价值;但现实是,并非所有 AI 公司都会这么做。即便表现最好的公司做得很好,最糟糕的公司仍可能危及所有人。例如,有些 AI 公司对当今模型中儿童被性化的现象表现出令人震惊的漠视;这让我怀疑,它们未来是否有意愿或能力应对模型自主性风险。此外,AI 公司的商业竞争只会日益激烈。虽然引导模型的科学研究可能带来一些商业收益,但总体而言,竞争越激烈,就越难专注于应对自主性风险。我认为唯一的解决办法是立法:直接约束 AI 公司行为的法律,或通过激励研发来解决这些问题。
The fourth thing we can do is encourage coordination to address autonomy risks at the level of industry and society. While it is incredibly valuable for individual AI companies to engage in good practices or become good at steering AI models, and to share their findings publicly, the reality is that not all AI companies do this, and the worst ones can still be a danger to everyone even if the best ones have excellent practices. For example, some AI companies have shown a disturbing negligence towards the sexualization of children in today’s models, which makes me doubt that they’ll show either the inclination or the ability to address autonomy risks in future models. In addition, the commercial race between AI companies will only continue to heat up, and while the science of steering models can have some commercial benefits, overall the intensity of the race will make it increasingly hard to focus on addressing autonomy risks. I believe the only solution is legislation—laws that directly affect the behavior of AI companies, or otherwise incentivize R&D to solve these issues.
这里值得回想一下我在文章开头对不确定性和精准干预的提醒。我们并不确定自主性风险是否会成为严重问题——正如我所说,我不认同危险不可避免、或事情默认就会出错这种说法。只要这种危险有可信依据,我和 Anthropic 就愿意付出相当大的代价来应对;但一旦进入监管阶段,我们就是在迫使众多行为主体承担经济成本,而其中不少人不相信自主性风险真实存在,也不认为 AI 会强大到构成威胁。我认为他们错了,但我们应该务实看待预期会遇到多少反对,以及过度干预的风险。还有一个真实风险是,过于刻板的立法可能会强加一些并不能提高安全性、只会浪费大量时间的测试或规则(本质上沦为“安全作秀”);这同样会引发反弹,让安全立法显得荒唐。18
Here it is worth keeping in mind the warnings I gave at the beginning of this essay about uncertainty and surgical interventions. We do not know for sure whether autonomy risks will be a serious problem—as I said, I reject claims that the danger is inevitable or even that something will go wrong by default. A credible risk of danger is enough for me and for Anthropic to pay quite significant costs to address it, but once we get into regulation, we are forcing a wide range of actors to bear economic costs, and many of these actors don’t believe that autonomy risk is real or that AI will become powerful enough for it to be a threat. I believe these actors are mistaken, but we should be pragmatic about the amount of opposition we expect to see and the dangers of overreach. There is also a genuine risk that overly prescriptive legislation ends up imposing tests or rules that don’t actually improve safety but that waste a lot of time (essentially amounting to “safety theater”)—this too would cause backlash and make safety legislation look silly.18
Anthropic 一直认为,最合适的起点是制定透明度法案,要求每一家前沿 AI 公司都遵循我在本节前面描述的透明度做法。加州 SB 53 和纽约州 RAISE 法案就是这类立法的例子;Anthropic 支持了这些法案,它们也已成功通过。我们支持并参与起草这些法律时,特别注重尽量减少附带损害,例如把不太可能开发前沿模型的小公司排除在法律适用范围之外。19
Anthropic’s view has been that the right place to start is with transparency legislation, which essentially tries to require that every frontier AI company engage in the transparency practices I’ve described earlier in this section. California’s SB 53 and New York’s RAISE Act are examples of this kind of legislation, which Anthropic supported and which have successfully passed. In supporting and helping to craft these laws, we’ve put a particular focus on trying to minimize collateral damage, for example by exempting smaller companies unlikely to produce frontier models from the law.19
我们希望,透明度立法能让人们随着时间推移,更清楚自主性风险出现的可能性和严重程度,也能了解风险的性质以及预防风险的最佳方式。随着更具体、可采取行动的风险证据出现(如果确实出现的话),未来几年制定的法律就可以精准聚焦于证据充分、方向明确的具体风险,尽量减少附带损害。说清楚一点,如果出现真正有力的风险证据,规则就应当与证据相称、足够有力。
Our hope is that transparency legislation will give a better sense over time of how likely or severe autonomy risks are shaping up to be, as well as the nature of these risks and how best to prevent them. As more specific and actionable evidence of risks emerges (if it does), future legislation over the coming years can be surgically focused on the precise and well-substantiated direction of risks, minimizing collateral damage. To be clear, if truly strong evidence of risks emerges, then rules should be proportionately strong.
总体而言,我乐观地认为,结合对齐训练、机制可解释性、发现并公开披露令人担忧行为的努力、防护措施以及社会层面的规则,就能应对 AI 自主性风险。不过,我最担心的是社会层面的规则,以及最不负责任的参与者的行为(而最不负责任的参与者恰恰最强烈地反对监管)。我相信,解决之道在民主社会中一向如此:相信这项事业的我们,应当说明这些风险真实存在,并呼吁同胞团结起来保护自己。
Overall, I am optimistic that a mixture of alignment training, mechanistic interpretability, efforts to find and publicly disclose concerning behaviors, safeguards, and societal-level rules can address AI autonomy risks, although I am most worried about societal-level rules and the behavior of the least responsible players (and it’s the least responsible players who advocate most strongly against regulation). I believe the remedy is what it always is in a democracy: those of us who believe in this cause should make our case that these risks are real and that our fellow citizens need to band together to protect themselves.
作者注
1 这与我在《仁爱机器》中提出的一点相呼应:我开篇就说,不应把 AI 的好处看作救赎预言;我们必须具体、脚踏实地,避免夸大其词。归根结底,救赎预言和末日预言都无助于我们面对现实,原因大体相同。
This is symmetric to a point I made in Machines of Loving Grace, where I started by saying that AI’s upsides shouldn’t be thought of in terms of a prophecy of salvation, and that it’s important to be concrete and grounded and to avoid grandiosity. Ultimately, prophecies of salvation and prophecies of doom are unhelpful for confronting the real world, for basically the same reasons.
2 Anthropic 的目标是在这些变化中保持一致。过去,谈论 AI 风险在政治上很受欢迎,Anthropic 谨慎倡导审慎、以证据为基础的风险应对方式;如今,谈论 AI 风险在政治上不受欢迎,Anthropic 仍谨慎倡导审慎、以证据为基础的风险应对方式。
Anthropic’s goal is to remain consistent through such changes. When talking about AI risks was politically popular, Anthropic cautiously advocated for a judicious and evidence-based approach to these risks. Now that talking about AI risks is politically unpopular, Anthropic continues to cautiously advocate for a judicious and evidence-based approach to these risks.
3 这些年来,我对 AI 的发展轨迹以及它全面超越人类能力的可能性越来越有信心,但仍然存在一些不确定性。
Over time, I have gained increasing confidence in the trajectory of AI and the likelihood that it will surpass human ability across the board, but some uncertainty still remains.
4 芯片出口管制就是一个很好的例子。它们简单明了,而且看起来基本奏效。
Export controls for chips are a great example of this. They are simple and appear to mostly just work.
5 当然,寻找这类证据时必须秉持求真诚实的态度;这种寻找也可能发现危险并不存在的证据。通过模型卡和其他披露方式提高透明度,正是为此所作的尝试。
And of course, the hunt for such evidence must be intellectually honest, such that it could also turn up evidence of a lack of danger. Transparency through model cards and other disclosures is an attempt at such an intellectually honest endeavor.
6 确实,自 2024 年撰写《仁爱机器》以来,AI 系统已经能胜任需要人类花数小时完成的任务。METR 最近评估认为,Opus 4.5 能以 50% 的可靠性完成人类约四小时的工作。
Indeed, since writing Machines of Loving Grace in 2024, AI systems have become capable of doing tasks that take humans several hours, with METR recently assessing that Opus 4.5 can do about four human hours of work with 50% reliability.
7 还要说明,即使从技术角度看,强大 AI 只需一两年就会出现,它带来的许多社会后果,无论积极还是消极,都可能还要再过几年才会显现。这就是为什么我可以同时认为,AI 会在一至五年内颠覆 50% 的初级白领岗位,又认为只需一两年,我们就可能拥有比所有人都更有能力的 AI。
And to be clear, even if powerful AI is only 1–2 years away in a technical sense, many of its societal consequences, both positive and negative, may take a few years longer to occur. This is why I can simultaneously think that AI will disrupt 50% of entry-level white-collar jobs over 1–5 years, while also thinking we may have AI that is more capable than everyone in only 1–2 years.
8 值得补充的是,公众(相较于政策制定者)似乎确实非常担忧 AI 风险。我认为他们关注的问题有些是对的(例如 AI 导致的就业岗位流失),有些则是判断有误(例如担心 AI 耗水,但这并不是大问题)。这种反弹让我觉得,社会有可能形成应对风险的共识;不过到目前为止,它还没有转化为政策改变,更不用说有效且有针对性的政策改变了。
It is worth adding that the public (as compared to policymakers) does seem to be very concerned with AI risks. I think some of their focus is correct (i.e. AI job displacement), and some is misguided (such as concerns about water use of AI, which is not significant). This backlash gives me hope that a consensus around addressing risks is possible, but so far it has not yet been translated into policy changes, let alone effective or well-targeted policy changes.
9 当然,它们也可以操纵大量人类(或者干脆付钱雇用他们),让他们在现实世界中做它们想做的事。
They can also, of course, manipulate (or simply pay) large numbers of humans into doing what they want in the physical world.
10 我认为这并非稻草人论证。比如,据我所知,扬·勒昆就持有这种立场。
I don’t think this is a straw man: it’s my understanding, for example, that Yann LeCun holds this position.
11 例如,参见 Claude 4 系统卡第 5.5.2 节(第 63 至 66 页)。
For example, see Section 5.5.2 (p. 63–66) of the Claude 4 system card.
12 这个简单模型还包含其他一些假设,我在这里不作讨论。总体而言,这些假设应该会让我们对“未对齐的寻求权力行为”这一具体、简化的论述少一些担忧,但也会让我们更担心尚未预料到的不可预测行为。
There are also a number of other assumptions inherent in the simple model, which I won’t discuss here. Broadly, they should make us less worried about the specific simple story of misaligned power-seeking, but also more worried about possible unpredictable behavior we haven’t anticipated.
13 《安德的游戏》描写了这一设想的一个版本,涉及的是人类而非 AI。
Ender’s Game describes a version of this involving humans rather than AI.
14 例如,模型可能被告知不要做各种坏事,同时又被要求服从人类;但它随后观察到,很多人类恰恰在做那些坏事!目前还不清楚模型会如何化解这种矛盾(设计良好的宪法应鼓励模型从容处理这类矛盾),但这种两难处境与我们测试时给 AI 模型设置的、所谓“人为”的情境并没有太大区别。
For example, models may be told not to do various bad things, and also to obey humans, but may then observe that many humans do exactly those bad things! It’s not clear how this contradiction would resolve (and a well-designed constitution should encourage the model to handle these contradictions gracefully), but this type of dilemma is not so different from the supposedly “artificial” situations that we put AI models in during testing.
15 顺带一提,宪法是一份自然语言文件,因此人人都能读懂;任何人都可以评论,也可以把它与其他公司的类似文件进行比较。若能形成良性竞争,不仅鼓励公司公开发布这类文件,也促使它们把文件写好,那将很有价值。
Incidentally, one consequence of the constitution being a natural-language document is that it is legible to the world, and that means it can be critiqued by anyone and compared to similar documents by other companies. It would be valuable to create a race to the top that not only encourages companies to release these documents, but encourages them to be good.
16 有一种假说认为,宪法式 AI 的性格方法与可解释性和对齐研究的成果之间,存在一个深层统一原理。按这一假说,驱动 Claude 的根本机制最初是作为它在预训练阶段模拟角色的方式而出现的,例如预测小说人物会说什么。这意味着,理解宪法的一种有效方式,是把它看成一份角色设定,模型借此塑造连贯一致的人格。这也有助于解释我前面提到的“我一定是个坏人”这一结果(因为模型在试图扮演一个性格连贯的角色,在这个例子里就是一个坏人),并意味着可解释性方法或许能发现模型内部的“心理特质”。我们的研究人员正在寻找检验这一假说的方法。
There’s even a hypothesis about a deep unifying principle connecting the character-based approach from Constitutional AI to results from interpretability and alignment science. According to the hypothesis, the fundamental mechanisms driving Claude originally arose as ways for it to simulate characters in pretraining, such as predicting what the characters in a novel would say. This would suggest that a useful way to think about the constitution is more like a character description that the model uses to instantiate a consistent persona. It would also help us explain the “I must be a bad person” results I mentioned above (because the model is trying to act as if it’s a coherent character—in this case a bad one), and would suggest that interpretability methods should be able to discover “psychological traits” within models. Our researchers are working on ways to test this hypothesis.
17 需要说明的是,监测工作会以保护隐私的方式进行。
To be clear, monitoring is done in a privacy-preserving way.
18 即便在我们通过《负责任扩展政策》对本质上自愿施加的规则进行实验时,我们也一再发现,很容易变得过于僵化:事前看似重要的界线,事后回头看却可能很荒唐。技术快速发展的情况下,确实很容易把规则定在不该管的事情上。
Even in our own experiments with what are essentially voluntarily imposed rules with our Responsible Scaling Policy, we have found over and over again that it’s very easy to end up being too rigid, by drawing lines that seem important ex ante but turn out to be silly in retrospect. It is just very easy to set rules about the wrong things when a technology is advancing rapidly.
19 SB 53 和 RAISE 法案完全不适用于年收入低于 5 亿美元的公司,只适用于像 Anthropic 这样规模更大、更成熟的公司。
SB 53 and RAISE do not apply at all to companies with under $500M in annual revenue. They only apply to larger, more established companies like Anthropic.