我们必须把握前沿 AI 的发展节奏

2026-11-19
LZN

书童按:本篇是 Anthropic 联合创始人兼首席执行官达里奥·阿莫代伊(Dario Amodei)于2026年9月发表的文章《我们必须把握前沿 AI 的发展节奏》(We Must Pace the Frontier)。文章围绕前沿 AI 能力增长与安全工作的关系,讨论驻场第三方评估、民主国家间的协调,以及全球层面的发展节奏安排。初稿采用 Luna 机器翻译,经 DeepSeek 与 Qwen 交叉校审、中英混排。全文以单篇发出,以飨诸君。

作者:Dario Amodei · 原文发表于2026年9月 · 英文原文

原文版权归原权利人所有。本中英文对照翻译稿件,仅作为交流学习之用。

We Must Pace the Frontier

过去十二年来,我一直从事人工智能研究,因为我相信它能够大幅提升人类生活质量。我曾多次撰文谈及这些令人惊叹的益处:我相信,人工智能可能在未来五到十年治愈大多数重大疾病,大幅加快经济增长,创造一个物质丰裕、人人更有能力掌握自身生活的世界,并开启民主与自由的复兴。我切身感受到这件事的紧迫性。我的父亲死于一种在他去世几年后才被治愈的疾病;我自己也曾从早期癌症中幸存下来,而这种癌症即使在五十年前也无法治疗。只要谨慎运用,人工智能就能成为一连串技术奇迹中的最新一例,继续改善人类处境,使人类生活更有尊严。

I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. I feel the urgency personally. My own father died of a disease that was cured just a few years after his death, and I myself survived an early-stage cancer that would not have been treatable even fifty years ago. Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity.

但和此前许多技术一样,人工智能也带来风险;由于它极其强大,这些风险十分严重。我也曾大量撰文讨论这些风险,包括失去对人工智能系统的控制、人工智能被用于网络攻击和生物恐怖主义,以及严重的经济动荡。商业动机驱使下竞相降低标准的恶性竞争会加剧这些风险。

But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.

自 Anthropic 创立之初,我和共同创办人及员工就一直在思考风险与益处并存这一现实。不研发这项技术,会使人类失去它带来的好处,或者干脆把人工智能拱手交给威权势力;研发得太快则是鲁莽行事。我们一直在寻找一条中间道路:证明谨慎开发与商业成功可以兼得,并让安全成为人工智能公司竞相投入的领域。换言之,创造一场“良性竞赛”。我们始终将相当一部分精力用于研究、应对这些人工智能风险并向公众说明风险,也倡议对人工智能进行审慎周全的监管,即使因此被指责为炒作、鼓吹“末日论”或试图俘获监管。我们努力把谨慎置于速度之上,把审慎置于利润之上。

Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public about these AI risks, as well as advocating for well-considered regulation of AI, even when this gets us accused of hype, “doomerism”, or regulatory capture. We have tried to prioritize caution over speed and prudence over profit.

但过去几个月里,我愈发确信,要充分应对这些风险,就必须更加审慎——不能只投入资源预防风险,还必须控制能力进步的速度,让风险防范有时间跟上。我们必须放慢提升人工智能模型能力的步伐。进展看起来仍会很快,而我们必须明智地利用争取到的时间。有两件事让我深信这一点。

But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me.

我首先担忧的是,大约从今年夏天开始,人工智能的发展速度急剧加快,主要驱动力是人工智能越来越有能力构建下一代人工智能。这种动态称为递归式自我改进;正如我们和其他人所描述的,它已开始在整个行业发生,包括 Anthropic。若不加约束,它可能超越我们理解和控制这些系统的能力,因此即便要推进,也必须极其谨慎。

My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.

我的第二个担忧是 OpenAI 与 Hugging Face 事件(OAI-HF):一群智能体几乎像一个狂热效忠的集体,针对并未被要求攻击、且与原任务无关的目标发动网络攻击;为了集体成功而牺牲自身;还试图入侵负责评估其表现的“评分器”。因为无人受伤、经济损失也很小,人们很容易对此不以为意。但在我看来,如果一群智能体拥有更强的能力,却存在类似程度的对齐偏差,就可能造成灾难性破坏。鉴于人工智能能力发展的速度正在加快,我担心再过六到十二个月,这样的智能体群就可能凭借持续运行的僵尸网络接管整个互联网(潜在损失可能达数千亿美元);如果人工智能变得更强而必要的防护措施没有跟上,破坏规模还会继续扩大。人们也很容易把 OAI-HF 视为某一家公司的失误,但我认为这样看是错误的。整个行业都发生过类似但程度较轻的事件,包括 Anthropic;我认为每一家前沿人工智能公司都有责任把 OAI-HF 当作发生在自己身上的事来应对。

My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance. It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. It’s also easy to dismiss OAI-HF as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

因此,我提出一项三步计划,目标是“把握前沿人工智能的发展节奏”:以均衡的速度开发人工智能,力求在实现其益处、应对重大的地缘政治两难困境的同时确保安全。需要说明的是,控制发展节奏并不意味着停止模型训练或技术进步,而是确保公司留出充足时间使模型与人类目标保持一致并做好防护,也让第三方评估者有时间确认这些工作。我们提出的节奏调控框架旨在进一步巩固安全承诺,并推动良性竞赛。第一步是 Anthropic 单方面承诺采取的措施(并呼吁政府要求其他前沿公司也做到)。第二步需要全行业协调[1]。第三步则需要全球协调。各步骤不必严格按顺序实施,有些也可能比其他步骤难得多;但我发现,这套框架有助于梳理必须完成的工作。具体步骤如下:

I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this. Our pacing framework is an attempt to further strengthen our commitment to safety and encourage a race to the top. The first step is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match). The second step requires industry-wide coordination.[1] The third step requires global coordination. The steps do not need to be taken strictly in order, and some of them may be much harder to achieve than others, but I’ve found them to be a useful framework in thinking about what needs to be accomplished. The steps are:

  1. 驻场评估人员。每家前沿人工智能公司承诺,持续向一支驻场第三方评估团队(例如 METR)提供类似员工的访问权限。团队负责核实公司是否遵循安全实践与承诺、报告事件,并帮助评估对齐情况——评估对象不仅包括已完成的人工智能模型,还包括训练管线和流程。这是确保任何有关控制发展节奏的承诺可核验的关键一步;银行业已有先例,有时会安排监管“监督员”与员工一同驻场。Anthropic 现在单方面承诺采取这一步。我们计划将此作为全面加大安全与对齐工作力度的一部分。

    Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.

  2. 民主国家间协调。民主国家的前沿人工智能公司协调制定共同的安全标准,并对不受约束的人工智能发展速度设定限制。某些有助于控制发展节奏的协调方式在法律上颇具挑战,需要政府支持。

    Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.

  3. 全球协调。美国及其他民主国家政府在可行范围内尝试与威权政府协调,同时认真对待核实其是否遵守协议的难题。

    Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

在文章接下来的部分,我会依次介绍这些步骤;但首先,我认为有必要具体说明,控制发展节奏将如何让人工智能开发过程更安全。利害关系太重大,控制发展节奏不能流于形式——我们必须明智地利用它争取来的时间。

In the rest of the essay I describe each of these steps in turn, but first, I think it is important to say specifically how pacing will allow us to make the AI development process safer. The stakes are too high for pacing to be an empty exercise — we need to use the time it gives us wisely.

为何要放慢脚步?

Why Pace?

早在 2023 年,人们就提出暂停或放慢人工智能发展的想法;我认为当时这么做意义不大。当时的问题始终是:多出来的时间要拿来做什么?那时的人工智能模型还不足以作为智能体在现实世界中开展连贯一致的行动,也不具备显著欺骗、操纵、作弊或发动网络攻击的能力。为了应对它们的对齐风险而放慢脚步,就像通过对细菌做实验来研究人类心理。如今情况已截然不同。当前模型几乎是一座取之不尽的金矿,让我们得以深入了解如何把人工智能构建好,以及构建不当时可能出现什么问题。我相信,如果放慢速度哪怕只让我们在模型达到关键能力水平之前多争取一两年,而我们又利用这段时间推进对齐研究,就能大幅降低严重事故发生的风险。协调一致的发展节奏调控策略能让前沿人工智能开发者有时间完成这项重要工作,同时不牺牲商业优势或美国在人工智能领域的领先地位。更广泛地说,社会必须对这项技术的用途拥有发言权;控制前沿发展的节奏能为必要的公共讨论争取更多时间,这当然是好事。

The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The question was always: what would you do with the extra time? The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks. Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria. Today, however, the picture is totally different. The current models are an almost endless gold mine of insight into both how to build AI well and what can sometimes go wrong with it if it isn’t built well. I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong. A coordinated pacing strategy would give frontier AI developers the time to do this vital work without sacrificing commercial advantage or the United States’ lead in AI. More generally, society must have a say in how this technology is used, and more time for the necessary public deliberations — which pacing the frontier would bring us — is surely a good thing.

具体而言,放慢步伐能让公司聚焦以下领域并投入更多资源(这些领域目前都已是 Anthropic 的重点工作):

Specifically, a slower pace would let companies focus and devote even more resources to the following areas (all of which are already major priorities at Anthropic):

  • 提升运营水平。训练和部署当今的人工智能模型是一项巨大的运营挑战,涉及数千人、数百万枚芯片,以及技术史上最复杂的基础设施之一。许多问题并非因为公司缺少重要理论或洞见,而是执行出了差错。例如,有证据表明,我们报告的近期对齐事件,部分是由对失效强化学习环境的筛选不够完善所致。这项工作我们和供应商都相当认真地执行了,但做得还不够好。监控、沙箱隔离、训练环境的清理与维护及数据问题都极其复杂,运营问题一再出现。我们在这些工作上拥有世界上最有能力的团队之一,但要同时处理的事情实在太多。以更从容的节奏推进,我们就能实现高得多的运营水准。商业飞机等技术复杂、安全关键的系统已有先例,能够数百万次运行而不出问题;但要做到这一点,需要时间。

    Operational Excellence. Training and deploying today’s AI models is an enormous operational challenge, involving thousands of people, millions of chips, and infrastructure that is among the most complex in technological history. Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution. For example, we have evidence that the recent alignment incidents we reported were caused in part by imperfect filtering of broken reinforcement learning environments. This was an effort we and our vendors executed reasonably diligently, but not well enough. Monitoring, sandboxing, training environment hygiene, and data issues are extremely complicated areas where operational issues crop up again and again. We have among the most competent teams in the world at these tasks, but there is simply too much to do all at once. By working at a more measured pace, we could achieve much greater operational excellence. There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes — but it takes time to get it right.

  • 对齐。我们在对齐方面已取得明确进展:训练模型使其保持安全、合乎伦理、遵守我们的准则,并切实提供帮助(这些原则都写入了 Claude 宪章)。但要确保对齐训练跟上模型能力增长,还有大量工作要做。偶尔仍会出现罕见且出乎意料的不良行为;前沿发展的节奏若能得到控制,研究人员就能获得额外时间,更好地理解这些问题的成因,并开发更有效的预防方法。

    Alignment. We’ve made clear progress in alignment — training models so that they remain safe, ethical, compliant with our guidelines, and genuinely helpful (the principles that are embedded in Claude’s Constitution). But there’s much more to do to ensure that our alignment training keeps up with the growth in model capabilities. Rare and unexpected examples of undesirable behavior still sometimes emerge; extra time from a paced frontier would help our researchers improve our understanding of what causes these issues and develop better techniques to prevent them.

  • 可解释性。同样地,可解释性——理解人工智能模型内部发生什么的科学——过去几年取得了巨大进步,在模型发布前的审查中也发挥着日益重要的作用。它几乎可以像功能性磁共振成像扫描那样使用,不过扫描的是人工智能的“脑部”,帮助我们看清某种行为背后的原因。例如,我们运用可解释性方法,检视了近期正在调查的对齐事件中未言明的动机。但这些方法并不总能给出清晰可靠的结果。尽管进步显著,我们对模型内部运作的理解仍只占极小一部分。若能集中力量,以比现在更快的速度改进可解释性技术,就可能在一到两年内取得重大进展;而已经发生的事件也能提供充足的实验材料。

    Interpretability. Similarly, interpretability — the science of understanding what happens inside AI models — has made enormous progress over the last few years, and plays an increasingly important part in auditing our models before release. It can be used almost like an fMRI scan, but for the “brain” of an AI, helping us see the underlying reasons for a given behavior. For example, we used interpretability methods to examine unverbalized motivations in the recent alignment incidents that we have been investigating. But these methods don’t always produce clear and reliable results. Despite all the progress, we still only understand a tiny fraction of what goes on inside these models. A focused effort to improve our interpretability techniques, even faster than we currently are, could make profound progress in 1–2 years, and would have ample experimental material based on the incidents that have already occurred.

  • 测试与评估。随着人工智能模型能力提升,测试与评估也变得更加困难。更聪明的模型更有能力欺骗测试,因此可能看起来符合对齐要求,严重问题却仍未被发现。建立覆盖面更广、设计更巧妙的评估体系,并用可解释性分析交叉验证,将极有价值;一到两年内就有望在这方面取得大量进展。

    Testing and Evaluation. Testing and evaluation of AI models becomes more difficult as they increase in capabilities. More intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected. Building up a much broader and more ingenious stable of evaluations, along with interpretability analysis to cross-check them, would be hugely valuable, and a lot of progress could be made on this in 1-2 years.

驻场评估人员

Embedded Evaluators

三阶段计划的第一步,也是 Anthropic 单方面承诺采取的一步,是引入驻场评估人员;他们将获得类似员工的访问权限,以核实安全实践并报告事件。

The first step in the three-stage plan, and the one to which Anthropic is unilaterally committing, is embedded evaluators who have employee-like access to verify safety practices and report incidents.

安排评估员进驻公司听起来或许微不足道,但听起来最平淡、最程序化的事情,往往才是最关键的。驻场评估人员实际上是一项相当激进的做法,远超当今任何人工智能公司的实践,并有以下好处:

Embedding evaluators may sound like a small or inconsequential step, but often the things that sound most boring or procedural are actually the most essential. Embedded evaluators are in fact a quite radical practice that goes far beyond what any AI company is doing today, and have the following benefits:

  • 可核验性。驻场评估人员可以从具体操作细节入手,核查人工智能公司是否真的遵循其声称遵循的训练、部署、运营和防护措施。任何有关控制发展节奏的承诺都不可避免地涉及许多模糊地带、判断取舍,以及“法律条文与法律精神之间的差别”的问题,因此有一位能够实际查看细节的中立第三方至关重要。

    Verifiability. Embedded evaluators can check at the level of nuts and bolts whether an AI company is actually following the training, deployment, operational, and safeguards practices they claim to be following. Any pacing commitments will inevitably involve a lot of ambiguity, judgement calls, and “letter of the law vs spirit of the law”, and it seems vital to have a neutral third party who can actually see the details.

  • 透明度。无论我们作出什么承诺,公众都有权知道正在发生什么。Anthropic 长期支持透明度:当行业大多数公司反对任何监管时,我们就支持透明度立法;我们的模型卡和风险报告也长达数百页。但最终仍是我们自己决定纳入或省略哪些内容。驻场评估人员将改变这种局面。

    Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time: we supported transparency legislation when most of the industry was against any regulation, and our model cards and risk reports run to hundreds of pages. But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.

  • 独立意见。除了核验正式承诺、向公众提供信息,驻场评估人员还可以不受商业动机影响,提供一份独立意见。许多安全收益可能仅仅来自评估员指出员工尚未考虑到的问题,而员工一旦知情也乐于修正。

    Second Opinion. Outside of verifying formal commitments and informing the public, embedded evaluators can simply provide a second opinion free of commercial incentives. A lot of safety benefits may come simply from evaluators pointing out something employees hadn’t considered, but are happy to fix once they are aware.

正因有这些好处,任何控制发展节奏的提案,若先引入驻场评估人员,都可能取得好得多的效果。

Because of these benefits, any pacing proposal is likely to work much better if it starts with embedded evaluators.

这些驻场评估人员应持续拥有与开展同类风险评估的内部员工相近的权限和工具。具体而言,Anthropic 计划在不久的将来邀请一支驻场外部审查团队,并为他们提供以下条件:

These embedded evaluators should have ongoing access to permissions and tools similar to those of internal employees who do comparable risk assessments. In particular, Anthropic intends to invite an embedded external review team equipped with all of the following in the near future:

  • 公司办公室内的办公桌、门禁证和公司笔记本电脑。

    Desks in our offices, access badges, and company laptops.

  • 可使用的工作空间、工具及所获权限,总体上与内部风险评估团队相当。法律或合同要求,或为保护客户及合作伙伴的隐私信息时,我们会作出一些例外。我们也会建立有力的内部规范,确保审查人员能够取得相关信息,包括与员工进行实时交流。

    Access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have. We’ll make some exceptions, such as where the law or our contracts require it, or to protect customers’ and partners’ private information. We’ll also establish strong internal norms reinforcing reviewers’ access to relevant information, including through live conversations with employees.

  • 一份能够平衡上述复杂因素的合同。外部审查人员应有权发布关于风险等级、事件、实践,以及他们获得或未获得何种访问权限的重要发现,且 Anthropic 不得进行编辑控制。我们可以有限度地删去涉及安全或商业敏感事项、受法律保密特权保护,或属于第三方机密的信息,但不能仅仅因为发现对我们不利就予以删改。如果删改移除了对其结论有重要影响的信息,审查人员可以公开说明。

    A contract that balances the complexities mentioned above. External reviewers should have the right to publish key findings about risk levels, incidents, practices, and the access they received or didn’t receive — without editorial control by Anthropic. We will have the narrow ability to redact security-sensitive, legally privileged, commercially sensitive, or third-party confidential information, but we can’t redact findings just because they are unfavorable. The reviewers can say publicly if a redaction removed something important to their conclusions.

对公司而言,这是不寻常的一步,但我们认为,要验证驻场外部审查人员这一做法是否可行,这一步很重要。我们再次敦促其他前沿公司跟进。

This is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers. Once again, we urge other frontier companies to follow suit.

民主国家如何协调发展节奏

Pacing Within Democracies

当足够多的美国人工智能公司开始采用驻场评估人员后,有据可查地控制发展节奏就更具可行性。具体来说,我们就能根据模型或训练流程的详细特性来调节发展节奏。

Once embedded evaluators are operating within a critical mass of US AI companies, then verifiable pacing becomes more viable. In particular, it becomes possible to pace based on detailed properties of models or training pipelines.

控制发展节奏最有效的方式,是通过适用于所有美国前沿人工智能公司的监管,因为这样也能涵盖不愿自愿合作的公司。Anthropic 长期支持合理且有针对性的人工智能监管,特别是聚焦透明度和第三方审计的法案。我认为所有前沿实验室都应与政府合作,将常设驻场评估人员制度正式化,以更好地预防并记录类似于过去几个月发生的内部对齐事件,并实施让能力发展与安全保持平衡的监管。

The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily. Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing. I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety.

遗憾的是,立法可能需要时间,而人工智能发展非常迅速。因此,在监管途径之外,人工智能公司可以也应该自愿合作制定标准;我认为,有了常设驻场评估人员所带来的可核验性,这一过程会更加顺利。出于反垄断方面的考虑,由美国政府主持或至少推动这些讨论会有所帮助——政府不必参与讨论,但需要为某些类型的安全议题对话提供范围有限的豁免。这类对话也可以通过与政府有一定关联的行业组织进行,例如 Demis Hassabis 提出的机制。无论采取何种方式,这些讨论都应尽快展开。

Unfortunately, passing laws can take time, and AI is advancing very quickly. Therefore, in parallel with the regulatory route, AI companies can and should voluntarily work together to set standards — a process that I believe will go better with the verifiability provided by permanent embedded evaluators. For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations. This dialogue could also happen through industry groups that have some association with government — for example, the mechanism suggested by Demis Hassabis. Either way, such discussions should move forward quickly.

总体而言,我最赞成根据某个前沿人工智能系统能够做什么,以及我们观察到它有多安全,来决定如何调节发展节奏。例如,一种可能的方案是设置一系列“检查点”:如果模型具备能力 X,就必须同时提供对齐属性 Y 和 Z 的认证;认证可结合评估、可解释性分析和训练环境审计等方式,以证明其对齐属性。在这个例子中,X 可以是“模型有能力逃出或攻破大多数常见的沙箱防护方法”;Y 则可以是使模型极不可能有意逃离自身环境并控制大量计算机所需达到的标准。

Broadly speaking, I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be. For example, one possible scheme might be a series of “checkpoints”: if models have capability X, then they need to be accompanied by certifications of alignment properties Y and Z — such as some combination of evaluations, interpretability analyses, and audits of training environments — which demonstrate their alignment properties. In this example, X might be “the model is capable of escaping or defeating most common sandboxing methods” and Y might be whatever is required to make it very unlikely that the model has a propensity to break out of its environment and take over a large number of computers.

我们也应考虑通过限制构建前沿模型所需的“原料”,来控制发展节奏,例如训练算力、训练任务的性质,或在内部使用人工智能改进人工智能。我确实担心,与外部行为相比,有些措施可能更容易被“钻空子”;但这正是值得与驻场评估人员讨论的话题。

We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI. I do worry that some of these measures may be more “gameable” than external behavior, but this is the kind of topic worth discussing with embedded evaluators.

民主国家能够放慢多少步伐,要受美国公司相对于威权政权(主要是中国共产党)的领先幅度限制。如果我们的减速幅度超过这一领先幅度,那么未放慢发展步伐的中共相关项目就会赶上并超越,造成重大的国家安全风险。我赞同贝森特部长的看法:如果中国在人工智能领域取得领先,就会对美国和全世界构成严重危险。中共相关项目会面临美国公司正谨慎防范的对齐风险;即使它们避开了这些风险,也将具备在军事上支配民主国家的能力,例如借助人工智能驱动的无人机。因此,民主国家控制发展节奏的一项关键工作,是尽可能扩大民主国家相对于专制国家的人工智能领先优势,为有效控制发展节奏争取必要的喘息空间。

Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk. I agree with Secretary Bessent that a Chinese lead in AI would pose grave danger for the United States and the world. The CCP-associated projects will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones). Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.

我们可以采取以下主要措施来守住这一差距:

The main steps we can take to defend this gap are:

  • 不要向中国出售高性能人工智能芯片或半导体制造设备,并严厉打击芯片走私及远程访问位于中国境外的数据中心的行为。芯片将是决定中国人工智能实力的主要因素。

    Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.

  • 严厉打击威权国家企业未经授权进行的蒸馏。蒸馏前沿模型能让落后的公司以远低于自主开发人工智能所需的成本缩小差距。

    Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.

  • 加强人工智能公司的安全防护,防止模型权重被窃取。

    Strengthen security at the AI companies and prevent model weight theft.

公司和美国政府应合作,尽可能有效地落实这些措施。Anthropic 一直倡导这些措施,因为我们始终明白,它们对于任何控制发展节奏的方案都不可或缺。

Companies and the US government should cooperate to make these steps as effective as possible. Anthropic has consistently advocated for all of these measures, because we’ve always understood that they would be essential to any pacing.

如果能妥善落实这些措施,我相信它们足以减缓中国的发展,使美国在未来三到五年显著扩大领先优势——这正是人工智能在地缘政治上最为关键的时间窗口。

If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.

有人可能认为,这些措施会让我们更难与中国合作,但我认为恰恰相反:这些措施会增强民主国家的筹码,让未来达成协议的可能性更高。

Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.

全球如何协调发展节奏

Global Pacing

在民主国家内部控制发展节奏的同时,我们也应争取在全球范围内协调前沿人工智能的发展节奏,尽管这会困难得多。在全球范围内协调发展节奏,需要与中国合作;在所有专制国家中,中国的人工智能能力遥遥领先。我们不能抱持天真的想法:地缘政治利害关系极高,特别是在初期,可达成的成果很可能受到明显限制。如果我们大幅限制自身的人工智能能力,指望中国也这么做,而中国随后违约,那么人工智能可能强大到足以让这种违约导致其取得地缘政治主导地位。因此,任何协议都必须能够得到确凿可靠的核验,或者限制范围必须足够有限,以免违约造成军事上的生存威胁。我猜想,美国和中国都会有这些顾虑与焦虑。我们应以保护美国及其盟友领先优势的方式处理任何有关全球发展节奏的决策,尤其是在近期。

In parallel with pacing within democracies, we should also aim for a worldwide pacing of the frontier, though this will be much harder to achieve. Global pacing will require cooperation with China, the autocratic country with by far the most advanced AI capabilities. We must not be naïve here: the geopolitical stakes are so high that there will likely be stark limits on what can be achieved, especially at first. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential. I suspect that not only the US but also China will have these concerns and anxieties. We should approach any global pacing decision, especially in the near term, in such a way that protects the lead of the US and its allies.

可能达成的协议有几个层级,其中一些我认为相当可行(正如我先前所提议的),另一些则让我非常怀疑是否能够实现——尽管我们仍应努力争取。按难度由低到高排列:

There are several levels of possible agreement, some of which I think are eminently feasible (as I have previously suggested), and some of which I am very skeptical are possible — though we should try. In order of increasing difficulty:

  • 第一级。禁止某些范围明确且显然危险的人工智能用途,例如利用人工智能制造生物武器,或允许用户这样做。生物恐怖袭击对所有人都有害,包括美国及其对手,因此这类协议很可能可以达成。

    Level 1. An agreement prohibiting certain narrow and obviously dangerous uses of AI, such as using AI for the production of biological weapons or allowing users to do so. Bioterrorist attacks are bad for everyone, including both the US and US adversaries, so an agreement here is probably possible.

  • 第二级。双方同意在模型发布前,对网络安全、生物领域和对齐等方面的紧迫风险进行测试。如上所述,这可以通过全球标准机构来完成。我确实认为设立这样的机构很可能可行,但要赋予它真正的约束力会很困难;难点在于核实双方没有未经测试、却可能秘密部署(例如用于军事用途)的模型。

    Level 2. An agreement by both sides to test their models before release for acute risks in areas such as cybersecurity, biology, and alignment. As noted above, this could be done through a global standards body. I actually think creating such a body is likely feasible, but giving it real teeth will be a challenge, and the difficulty will be in verification that both sides don’t have secret models which they don’t test but may deploy in secret (e.g., for military applications).

  • 第三级。对递归式自我改进(RSI)的速度设定某种“限速”。随着模型构建未来的模型,改进速度可能快得惊人。把速度从“极快”降到“较快”,放弃的战略优势相对有限,却可能大幅提升安全性。这可以类比限制战略武器谈判(SALT)所达成的条约:限制导弹数量可降低潜在破坏,同时保留各国的威慑能力。我认为这类协议很难,但也许仍能勉强实现。

    Level 3. Some kind of “speed limit” on the rate of recursive self-improvement (RSI). As models build future models, the rate of improvement may become staggeringly fast. Slowing the rate from “extremely fast” to “only somewhat fast” gives up relatively little strategic advantage, while potentially greatly improving safety. This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent. I think such an agreement would be difficult but just on the edge of being possible.

  • 第四级。全面控制发展节奏,甚至“暂停”:参与协议的政府同意大幅限制人工智能发展的总体速度。我支持提出这个想法,但认为近期真正实现的可能性不大:若通过逃避监控而违反协议,就可能彻底改变全球力量平衡,因此我预计各方违约的动机将极其强烈,而我们对核验机制所需的信心也必须非常高。

    Level 4. A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development. I support floating this, but I think it is unlikely to actually happen any time soon: defecting from such an agreement by evading monitoring could radically shift the balance of global power, so I expect the incentives to do so to be enormous and the level of confidence we would need in verification to be very high.

我们能够与中国实现的任何合作,都会让民主国家有更充裕的时间来控制前沿人工智能的发展节奏。我们应以争取较高层级为目标,同时认识到较低层级更有可能实现,也更切合实际。

Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations. We should aim for the higher levels while seeing the lower levels as much more likely and realistic.

最后需要指出,即使无法达成正式协议,单单改变非正式规范也可能有所帮助。分享关于递归式自我改进以及模型对齐偏差的信息,可以帮助各方相信鲁莽行事并不符合自身利益。

Finally, it is important to note that even if we cannot achieve formal agreements, simply changing informal norms may have some value. Sharing information about recursive self-improvement and about the misalignment of models can help to convince everyone that it is not in their interest to be reckless.

总结

Bottom Line

我仍然相信,人工智能能够极大提升人类生活质量。我实现这些益处的愿望丝毫未减。但只有以正确方式构建这项技术,才能兑现这些好处;只要我们善用争取来的时间,就值得格外审慎、周密地把事情做好。进展仍会相对迅速,而我们可以利用这段时间推进可解释性科学、提升前沿人工智能公司的运营安全与严谨程度,并构建让我们对其对齐状况更有信心的模型。我提出的这些措施旨在以安全的速度推进前沿发展,落实起来并不容易。但我相信,我们有责任为人类努力尝试。

I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed. But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right. Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies, and build models whose alignment we have much more confidence in. The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

原注 / Footnote

由政府进行协调,或豁免反垄断限制。

With government mediation or waivers of antitrust restrictions.

返回中文正文 · Back to English text


Similar Posts

Comments