技术的青春期(中):破坏性滥用与夺权风险|达里奥·阿莫代伊 · 中英对照节译

2026-11-19
LZN

书童按:本篇是达里奥·阿莫代伊(Dario Amodei)2026年1月发表的《技术的青春期》(The Adolescence of Technology)一文的中篇。阿莫代伊是 Anthropic 的联合创始人兼首席执行官。本篇讨论两种不同的权力变化:个人获得前所未有的破坏能力,政府或企业获得前所未有的控制力。作者进而追问,民主社会可以采取哪些防御措施,又应当给自己划下哪些界限。译稿由 Luna 初译,经 DeepSeek 与 Qwen 交叉校审,以飨诸君。

英文原文 · The Adolescence of Technology | 原文版权归原权利人所有。本中英文对照翻译稿件,仅作为交流学习之用。

直面并克服强大 AI 的风险
Confronting and Overcoming the Risks of Powerful AI

2. 意外而可怕的赋能
2. A surprising and terrible empowerment

被滥用于破坏
Misuse for destruction

假设我们已经解决了人工智能自主性的问题——我们不再担心“数据中心里的天才之国”会失控、凌驾于人类之上。人工智能天才会按人类的意愿行事;由于他们具有巨大的商业价值,世界各地的个人和组织都可以“租用”一位或多位人工智能天才,替自己完成各种任务。

Let’s suppose that the problems of AI autonomy have been solved—we are no longer worried that the country of AI geniuses will go rogue and overpower humanity. The AI geniuses do what humans want them to do, and because they have enormous commercial value, individuals and organizations throughout the world can “rent” one or more AI geniuses to do various tasks for them.

每个人口袋里都有一位超智能天才,这是了不起的进步,将创造巨大的经济价值,并提升人类生活质量。我在《Machines of Loving Grace》里详细谈过这些益处。但让每个人都拥有超人般能力,并非所有影响都会是正面的。它有可能增强个人或小团体大规模实施破坏的能力,让他们能够使用精密而危险的工具(例如大规模杀伤性武器);过去,这些工具只有少数具备高超技能、专业训练和专注力的人才能使用。

Everyone having a superintelligent genius in their pocket is an amazing advance and will lead to an incredible creation of economic value and improvement in the quality of human life. I talk about these benefits in great detail in Machines of Loving Grace. But not every effect of making everyone superhumanly capable will be positive. It can potentially amplify the ability of individuals or small groups to cause destruction on a much larger scale than was possible before, by making use of sophisticated and dangerous tools (such as weapons of mass destruction) that were previously only available to a select few with a high level of skill, specialized training, and focus.

正如比尔·乔伊在25年前的《Why the Future Doesn’t Need Us》中所写:20

As Bill Joy wrote 25 years ago in Why the Future Doesn’t Need Us:20

制造核武器至少曾经需要取得稀有——甚至实际上无法获得——的原材料,以及受保护的信息;生物武器和化学武器项目也往往需要大规模行动。二十一世纪的技术——遗传学、纳米技术和机器人技术……可能催生全新类别的事故和滥用……让个人或小团体也能轻易涉足其中。这些技术不需要大型设施或稀有原材料。……我们正处于极端邪恶进一步完善的边缘:这种邪恶的可能性远远超出大规模杀伤性武器曾为民族国家开启的毁灭性前景,并延伸到极端个人获得出人意料而可怕的力量。

Building nuclear weapons required, at least for a time, access to both rare—indeed, effectively unavailable—raw materials and protected information; biological and chemical weapons programs also tended to require large-scale activities. The 21st century technologies—genetics, nanotechnology, and robotics ... can spawn whole new classes of accidents and abuses … widely within reach of individuals or small groups. They will not require large facilities or rare raw materials. … we are on the cusp of the further perfection of extreme evil, an evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to the nation-states, to a surprising and terrible empowerment of extreme individuals.

乔伊指出的是这样一个道理:要造成大规模破坏,既要有动机,也要有能力;只要能力掌握在少数受过高等训练的人手中,单个人(或小团体)造成这种破坏的风险就相对有限。21一个精神状态不稳定的独行者可以发动校园枪击,但大概造不出核武器,也释放不了瘟疫。

What Joy is pointing to is the idea that causing large-scale destruction requires both motive and ability, and as long as ability is restricted to a small set of highly trained people, there is relatively limited risk of single individuals (or small groups) causing such destruction.21 A disturbed loner can perpetrate a school shooting, but probably can’t build a nuclear weapon or release a plague.

事实上,能力和动机甚至可能呈负相关。能够释放瘟疫的人很可能受过高等教育:大概拥有分子生物学博士学位,而且特别有办法,事业前景良好,性格稳定自律,也有很多东西可能失去。这样的人不太可能为了自己毫无好处、还会严重危及前途的事情而杀死大量人;他们必须受到纯粹恶意、强烈怨恨或精神不稳定的驱使。

In fact, ability and motive may even be negatively correlated. The kind of person who has the ability to release a plague is probably highly educated: likely a PhD in molecular biology, and a particularly resourceful one, with a promising career, a stable and disciplined personality, and a lot to lose. This kind of person is unlikely to be interested in killing a huge number of people for no benefit to themselves and at great risk to their own future—they would need to be motivated by pure malice, intense grievance, or instability.

这样的人确实存在,但很少见;也正因为罕见,一旦出现,往往就会成为轰动的大新闻。22他们也往往很难被抓到,因为他们聪明而有能力,有时留下的谜团要过几年甚至几十年才能解开。最著名的例子大概是数学家西奥多·卡钦斯基(“大学炸弹客”):他逃避联邦调查局追捕近20年,受反技术意识形态驱使。另一个例子是生物防御研究员布鲁斯·艾文斯,他似乎策划了2001年的一系列炭疽袭击。这种情况也发生在拥有专业技能的非国家组织身上:奥姆真理教这个邪教组织设法获得沙林神经毒气,并于1995年在东京地铁释放,造成14人死亡、数百人受伤。

Such people do exist, but they are rare, and tend to become huge stories when they occur, precisely because they are so unusual.22 They also tend to be difficult to catch because they are intelligent and capable, sometimes leaving mysteries that take years or decades to solve. The most famous example is probably mathematician Theodore Kaczynski (the Unabomber), who evaded FBI capture for nearly 20 years, and was driven by an anti-technological ideology. Another example is biodefense researcher Bruce Ivins, who seems to have orchestrated a series of anthrax attacks in 2001. It’s also happened with skilled non-state organizations: the cult Aum Shinrikyo managed to obtain sarin nerve gas and kill 14 people (as well as injuring hundreds more) by releasing it in the Tokyo subway in 1995.

幸好,这些袭击都没有使用具有传染性的生物制剂,因为当时连这些人都没有能力制造或获取这类制剂。23分子生物学的进步大幅降低了制造生物武器的门槛(尤其是材料的可获得性),但要做到这一点仍需要大量专业知识。我担心,每个人口袋里的天才可能会移除这道门槛,仿佛人人都成了病毒学博士,人工智能可以一步步带着他们完成设计、合成和释放生物武器的流程。面对强大的对抗压力——也就是所谓的“越狱”——要防止这类信息被诱导出来,可能需要在通常融入训练的防护措施之外再加上多层防御。

Thankfully, none of these attacks used contagious biological agents, because the ability to construct or obtain these agents was beyond the capabilities of even these people.23 Advances in molecular biology have now significantly lowered the barrier to creating biological weapons (especially in terms of availability of materials), but it still takes an enormous amount of expertise in order to do so. I am concerned that a genius in everyone’s pocket could remove that barrier, essentially making everyone a PhD virologist who can be walked through the process of designing, synthesizing, and releasing a biological weapon step-by-step. Preventing the elicitation of this kind of information in the face of serious adversarial pressure—so-called “jailbreaks”—likely demands layers of defenses beyond those ordinarily baked into training.

关键在于,这会打破能力与动机之间的相关性:那个想杀人、却缺乏自律或技能的精神状态不稳定的独行者,将获得病毒学博士级别的能力,而博士通常不太可能有这种动机。这种担忧不只适用于生物领域(尽管我认为生物领域最令人害怕),也适用于任何可能造成巨大破坏、但目前仍需要高超技能和自律的领域。换句话说,租用强大的人工智能,就等于让怀有恶意(但其他方面很普通)的人也拥有智能。我担心这样的人可能有很多;如果他们能获得一种轻易杀死数百万人的手段,迟早会有人这么做。此外,具备专业知识的人也可能借此造成比以往更大规模的破坏。

Crucially, this will break the correlation between ability and motive: the disturbed loner who wants to kill people but lacks the discipline or skill to do so will now be elevated to the capability level of the PhD virologist, who is unlikely to have this motivation. This concern generalizes beyond biology (although I think biology is the scariest area) to any area where great destruction is possible but currently requires a high level of skill and discipline. To put it another way, renting a powerful AI gives intelligence to malicious (but otherwise average) people. I am worried there are potentially a large number of such people out there, and that if they have access to an easy way to kill millions of people, sooner or later one of them will do it. Additionally, those who do have expertise may be enabled to commit even larger-scale destruction than they could before.

生物领域是我最担心的领域,远远超过其他领域,因为它的破坏潜力极大,而且很难防御,所以我会特别聚焦于生物领域。但这里所说的许多内容也适用于其他风险,例如网络攻击、化学武器或核技术。

Biology is by far the area I’m most worried about, because of its very large potential for destruction and the difficulty of defending against it, so I’ll focus on biology in particular. But much of what I say here applies to other risks, like cyberattacks, chemical weapons, or nuclear technology.

出于显而易见的原因,我不会详细讲解如何制造生物武器。但概括地说,我担心大型语言模型正在接近、甚至可能已经掌握了从头到尾制造并释放生物武器所需的知识,而且其潜在破坏力极大。如果有人决意让某些生物制剂尽可能广泛传播,可能造成数百万人死亡。不过,这仍需要很高的技能水平,包括一系列非常具体、并不广为人知的步骤和流程。我担心的并非只有固定或静态的知识。我担心大型语言模型能够以互动方式带着一个知识和能力平平的人走完复杂流程;没有这种帮助,流程可能出错,或需要排错。这类似于技术支持人员帮助不懂技术的人排查并修复复杂的电脑问题,只是这里的过程会更长,可能持续数周或数月。

I am not going to go into detail about how to make biological weapons, for reasons that should be obvious. But at a high level, I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release them end-to-end, and that their potential for destruction is very high. Some biological agents could cause millions of deaths if a determined effort was made to release them for maximum spread. However, this would still take a very high level of skill, including a number of very specific steps and procedures that are not widely known. My concern is not merely fixed or static knowledge. I am concerned that LLMs will be able to take someone of average knowledge and ability and walk them through a complex process that might otherwise go wrong or require debugging in an interactive way, similar to how tech support might help a non-technical person debug and fix complicated computer-related problems (although this would be a more extended process, probably lasting over weeks or months).

能力远超当今模型的大型语言模型,或许能够助长更加骇人的行为。2024年,一群知名科学家联名写信,警告研究、甚至制造一种危险新型生物体——“镜像生命”——所带来的风险。构成生物体的 DNA、RNA、核糖体和蛋白质都具有相同的手性(也称“左右性”),因此它们并不等同于自身在镜子里的反像(正如无论怎样转动右手,也无法让它与左手完全重合)。但蛋白质彼此结合的整套系统、DNA 合成与 RNA 翻译机制,以及蛋白质的构建和分解,全都依赖这种手性。如果科学家制造出手性相反的生物材料——这类材料也有一些潜在优势,例如药物在体内留存时间更长——可能会非常危险。这是因为,如果左手性生命被制成能够繁殖的完整生物体(这会非常困难),地球上任何分解生物材料的系统都可能无法消化它:它会有一把无法插入任何现有酶“锁孔”的“钥匙”。这意味着它可能不受控制地繁殖、挤压地球上的所有生命;最糟糕的情况下,甚至会摧毁地球上的一切生命。

More capable LLMs (substantially beyond the power of today’s) might be capable of enabling even more frightening acts. In 2024, a group of prominent scientists wrote a letter warning about the risks of researching, and potentially creating, a dangerous new type of organism: “mirror life.” The DNA, RNA, ribosomes, and proteins that make up biological organisms all have the same chirality (also called “handedness”) that causes them to be not equivalent to a version of themselves reflected in the mirror (just as your right hand cannot be rotated in such a way as to be identical to your left). But the whole system of proteins binding to each other, the machinery of DNA synthesis and RNA translation and the construction and breakdown of proteins, all depends on this handedness. If scientists made versions of this biological material with the opposite handedness—and there are some potential advantages of these, such as medicines that last longer in the body—it could be extremely dangerous. This is because left-handed life, if it were made in the form of complete organisms capable of reproduction (which would be very difficult), would potentially be indigestible to any of the systems that break down biological material on earth—it would have a “key” that wouldn’t fit into the “lock” of any existing enzyme. This would mean that it could proliferate in an uncontrollable way and crowd out all life on the planet, in the worst case even destroying all life on earth.

关于镜像生命能否被制造出来,以及它可能造成什么影响,科学界存在很大不确定性。2024年的那封信还附有一份报告,报告认为“镜像细菌有可能在未来十年至数十年内被制造出来”——这个时间范围相当宽泛。但足够强大的人工智能模型(明确地说,能力要远超当今任何模型)或许能更快找到制造方法,甚至实际帮助某人制造出来。

There is substantial scientific uncertainty about both the creation and potential effects of mirror life. The 2024 letter accompanied a report that concluded that “mirror bacteria could plausibly be created in the next one to few decades,” which is a wide range. But a sufficiently powerful AI model (to be clear, far more capable than any we have today) might be able to discover how to create it much more rapidly—and actually help someone do so.

我认为,即使这些风险鲜为人知、看起来也不太可能发生,后果的严重程度仍然极高,因此必须将其视为人工智能系统面临的头等风险,认真对待。

My view is that even though these are obscure risks, and might seem unlikely, the magnitude of the consequences is so large that they should be taken seriously as a first-class risk of AI systems.

怀疑者针对大型语言模型带来的生物风险是否严重提出了不少反驳。我不同意这些观点,但值得逐一回应。大多数反驳都没有认识到这项技术正沿着指数级轨迹发展。2023年我们刚开始谈大型语言模型带来的生物风险时,怀疑者说,谷歌上已经有所有必要信息,大型语言模型并没有增加任何新东西。这种说法从来就不成立:基因组确实可以免费获取,但正如我前面所说,某些关键步骤以及大量实操诀窍,无法靠这种方式获得。而且到2023年底,大型语言模型显然已经能就流程中的某些步骤提供谷歌找不到的信息。

Skeptics have raised a number of objections to the seriousness of these biological risks from LLMs, which I disagree with but which are worth addressing. Most fall into the category of not appreciating the exponential trajectory that the technology is on. Back in 2023 when we first started talking about biological risks from LLMs, skeptics said that all the necessary information was available on Google and LLMs didn’t add anything beyond this. It was never true that Google could give you all the necessary information: genomes are freely available, but as I said above, certain key steps, as well as a huge amount of practical know-how cannot be gotten in that way. But also, by the end of 2023 LLMs were clearly providing information beyond what Google could give for some steps of the process.

之后,怀疑者转而反驳说,大型语言模型无法端到端地提供帮助,不能协助获取生物武器,只能提供理论信息。截至2025年年中,我们的测量结果显示,大型语言模型可能已经在几个相关领域提供了实质性助力,或许使成功概率变为原来的两倍或三倍。因此我们决定,Claude Opus 4(以及随后推出的 Sonnet 4.5、Opus 4.1 和 Opus 4.5 模型)必须按照《负责任扩展政策》框架中的 AI 安全三级防护要求发布,并落实针对这类风险的防护措施(后文会详谈)。我们认为,如果没有防护措施,模型现在很可能已接近这样的水平:能够帮助一名拥有 STEM 学位、但并非生物学专业的人走完整个生物武器制造流程。

After this, skeptics retreated to the objection that LLMs weren’t end-to-end useful, and couldn’t help with bioweapons acquisition as opposed to just providing theoretical information. As of mid-2025, our measurements show that LLMs may already be providing substantial uplift in several relevant areas, perhaps doubling or tripling the likelihood of success. This led to us deciding that Claude Opus 4 (and the subsequent Sonnet 4.5, Opus 4.1, and Opus 4.5 models) needed to be released under our AI Safety Level 3 protections in our Responsible Scaling Policy framework, and to implementing safeguards against this risk (more on this later). We believe that models are likely now approaching the point where, without safeguards, they could be useful in enabling someone with a STEM degree but not specifically a biology degree to go through the whole process of producing a bioweapon.

另一种反驳是,社会还可以采取与人工智能无关的其他措施,阻止生物武器的制造。〔编者删节〕我支持强制进行基因合成筛查,以提高个人将病原体武器化的难度,从而降低人工智能导致的生物风险,也降低整体的生物风险。但我们目前还没有这样的措施。而且这只能是降低风险的一种手段;它应当补充人工智能系统的防护栏,而不能取而代之。

Another objection is that there are other actions unrelated to AI that society can take to block the production of bioweapons. [Editorial omission] I am supportive of mandated gene synthesis screening that would make it harder for individuals to weaponize pathogens, in order to reduce both AI-driven biological risks and also biological risks in general. But this is not something we have today. It would also be only one tool in reducing risk; it is a complement to guardrails on AI systems, not a substitute.

最有力的一种反驳,我却很少见人提出:模型在理论上有用,与坏人实际上会不会使用模型之间存在落差。大多数单独行动的恶意行为者都精神状态不稳定,所以几乎可以说,他们的行为本来就难以预测、并不理性;而这些缺乏技能的恶意行为者,恰恰可能最能从人工智能大幅降低杀害许多人的难度中获益。24某种暴力袭击在技术上可行,不代表就有人会决定去做。也许生物袭击并不讨人喜欢,因为实施者有相当大的可能会感染自己;它也不符合许多暴力个人或团体对军事行动的幻想,而且很难有选择地针对特定对象。即使人工智能一路指导,整个过程也要花上几个月;大多数精神状态不稳定的人可能根本没有这样的耐心。也许我们就是运气好,现实中的动机和能力不会以恰到好处的方式结合起来。

The best objection is one that I’ve rarely seen raised: that there is a gap between the models being useful in principle and the actual propensity of bad actors to use them. Most individual bad actors are disturbed individuals, so almost by definition their behavior is unpredictable and irrational—and it’s these bad actors, the unskilled ones, who might have stood to benefit the most from AI making it much easier to kill many people.24 Just because a type of violent attack is possible, doesn’t mean someone will decide to do it. Perhaps biological attacks will be unappealing because they are reasonably likely to infect the perpetrator, they don’t cater to the military-style fantasies that many violent individuals or groups have, and it is hard to selectively target specific people. It could also be that going through a process that takes months, even if an AI walks you through it, involves an amount of patience that most disturbed individuals simply don’t have. We may simply get lucky and motive and ability don’t combine, in practice, in quite the right way.

但把这当作依靠的防线,实在太脆弱了。精神状态不稳定的独行者可能因任何理由、甚至毫无理由而改变动机;事实上,大型语言模型已经被用于袭击(只是还没涉及生物领域)。只关注这类独行者,也忽略了受意识形态驱使的恐怖分子;他们往往愿意投入大量时间和精力(例如“9·11”劫机者)。想尽可能杀死更多人这种动机迟早可能出现,而不幸的是,生物武器恰好像是实现这种目的的手段。即使这种动机极其罕见,也只要出现一次就够了。随着生物学进步——而且这种进步越来越由人工智能自身推动——有针对性地发动袭击也可能成为现实(例如针对具有特定祖源的人群),这又增加了一种令人不寒而栗的潜在动机。

But this seems like very flimsy protection to rely on. The motives of disturbed loners can change for any reason or no reason, and in fact there are already instances of LLMs being used in attacks (just not with biology). The focus on disturbed loners also ignores ideologically motivated terrorists, who are often willing to expend large amounts of time and effort (for example, the 9/11 hijackers). Wanting to kill as many people as possible is a motive that will probably arise sooner or later, and it unfortunately suggests bioweapons as the method. Even if this motive is extremely rare, it only has to materialize once. And as biology advances (increasingly driven by AI itself), it may also become possible to carry out more selective attacks (for example, targeted against people with specific ancestries), which adds yet another, very chilling, possible motive.

我不认为生物袭击一旦广泛变得可行就会立刻发生——事实上,我赌它不会马上发生。但如果把风险放到数百万人、数年时间的尺度上累积来看,我认为发生重大袭击的风险相当现实;后果会极其严重(伤亡人数可能达到数百万甚至更多),所以我相信我们别无选择,只能采取严肃措施加以防范。

I do not think biological attacks will necessarily be carried out the instant it becomes widely possible to do so—in fact, I would bet against that. But added up across millions of people and a few years of time, I think there is a serious risk of a major attack, and the consequences would be so severe (with casualties potentially in the millions or more) that I believe we have no choice but to take serious measures to prevent it.

防御措施
Defenses

接下来要谈如何防范这些风险。我认为我们可以做三件事。第一,人工智能公司可以为模型设置防护栏,防止模型帮助制造生物武器。Anthropic 正在积极开展这项工作。Claude 的宪法主要聚焦于高层次的原则和价值观,但其中也有少数明确、坚决的禁令,之一就是不得协助制造生物(或化学、核、放射性)武器。不过,所有模型都有可能被越狱。因此,作为第二道防线,从2025年年中起——当时我们的测试显示,模型已开始接近可能构成风险的门槛——我们实施了专门检测并拦截生物武器相关输出的分类器。我们定期升级和改进这些分类器,并普遍发现它们即使面对复杂的对抗攻击也很稳健。25这些分类器确实增加了模型服务成本(有些模型的推理总成本中,接近5%来自它们),因此会压缩利润;但我们认为使用它们是正确的做法。

That brings us to how to defend against these risks. Here I see three things we can do. First, AI companies can put guardrails on their models to prevent them from helping to produce bioweapons. Anthropic is very actively doing this. Claude’s Constitution, which mostly focuses on high-level principles and values, has a small number of specific hard-line prohibitions, and one of them relates to helping with the production of biological (or chemical, or nuclear, or radiological) weapons. But all models can be jailbroken, and so as a second line of defense, we’ve implemented (since mid-2025, when our tests showed our models were starting to get close to the threshold where they might begin to pose a risk) a classifier that specifically detects and blocks bioweapon-related outputs. We regularly upgrade and improve these classifiers, and have generally found them highly robust even against sophisticated adversarial attacks.25 These classifiers increase the costs to serve our models measurably (in some models, they are close to 5% of total inference costs) and thus cut into our margins, but we feel that using them is the right thing to do.

值得肯定的是,其他一些人工智能公司也部署了分类器。但并非每家公司都这么做,而且目前没有任何规定要求公司必须保留分类器。我担心时间一长会出现囚徒困境:公司可以背叛合作、撤掉分类器来降低成本。这又是一个典型的负外部性问题,单靠 Anthropic 或任何一家公司的自愿行动都无法解决。26自愿的行业标准或许有帮助;人工智能安全研究所和第三方评估机构所做的第三方评估与核验,也可能有所帮助。

To their credit, some other AI companies have implemented classifiers as well. But not every company has, and there is also nothing requiring companies to keep their classifiers. I am concerned that over time there may be a prisoner’s dilemma where companies can defect and lower their costs by removing classifiers. This is once again a classic negative externalities problem that can’t be solved by the voluntary actions of Anthropic or any other single company alone.26 Voluntary industry standards may help, as may third-party evaluations and verification of the type done by AI security institutes and third-party evaluators.

但最终的防御可能需要政府采取行动,这是我们能做的第二件事。对于应对自主性风险,我的看法也是如此:我们应从透明度要求入手,27帮助社会衡量、监控风险并共同防御,同时避免以过于强硬的方式扰乱经济活动。之后,如果风险达到更明确的门槛,我们就可以制定更精准地针对这些风险、附带损害可能性更低的法律。具体到生物武器,我确实认为,出台这类有针对性的法律的时机可能很快就要到了:Anthropic 和其他公司正逐渐深入了解生物风险的性质,以及要求公司采取哪些防御措施才算合理。要彻底防御这些风险,可能需要国际合作,甚至包括地缘政治对手;不过,禁止研发生物武器的条约已经提供了先例。我通常对人工智能领域的大多数国际合作持怀疑态度,但这是一个范围有限、或许有机会实现全球克制的领域。连独裁政权也不希望发生大规模生物恐怖袭击。

But ultimately defense may require government action, which is the second thing we can do. My views here are the same as they are for addressing autonomy risks: we should start with transparency requirements,27 which help society measure, monitor, and collectively defend against risks without disrupting economic activity in a heavy-handed way. Then, if and when we reach clearer thresholds of risk, we can craft legislation that more precisely targets these risks and has a lower chance of collateral damage. In the particular case of bioweapons, I actually think that the time for such targeted legislation may be approaching soon—Anthropic and other companies are learning more and more about the nature of biological risks and what is reasonable to require of companies in defending against them. Fully defending against these risks may require working internationally, even with geopolitical adversaries, but there is precedent in treaties prohibiting the development of biological weapons. I am generally a skeptic about most kinds of international cooperation on AI, but this may be one narrow area where there is some chance of achieving global restraint. Even dictatorships do not want massive bioterrorist attacks.

最后,第三种应对措施是设法增强对生物袭击本身的防御能力。这可以包括监测和追踪以便及早发现,投资研发空气净化技术(例如远紫外线消毒),快速研发能够应对并适应袭击的疫苗,改进个人防护装备(PPE),28以及针对一些最可能出现的生物制剂开发疗法或疫苗。mRNA 疫苗可以针对特定病毒或变种设计,是这方面潜力的早期例子。Anthropic 很期待与生物技术和制药公司合作解决这个问题。但遗憾的是,我认为我们对防御端的期待应当有限。生物领域的攻击与防御并不对称:病原体会自行迅速传播,而防御需要迅速组织面向大量人群的检测、接种和治疗,以便作出应对。除非反应快如闪电(而这种情况很少发生),否则等到采取应对措施时,大部分损害已经造成。未来技术进步或许能让这种平衡有利于防御(我们当然应该利用人工智能帮助研发这类技术),但在此之前,预防性防护措施仍将是我们的主要防线。

Finally, the third countermeasure we can take is to try to develop defenses against biological attacks themselves. This could include monitoring and tracking for early detection, investments in air purification R&D (such as far-UVC disinfection), rapid vaccine development that can respond and adapt to an attack, better personal protective equipment (PPE),28 and treatments or vaccinations for some of the most likely biological agents. mRNA vaccines, which can be designed to respond to a particular virus or variant, are an early example of what is possible here. Anthropic is excited to work with biotech and pharmaceutical companies on this problem. But unfortunately I think our expectations on the defensive side should be limited. There is an asymmetry between attack and defense in biology, because agents spread rapidly on their own, while defenses require detection, vaccination, and treatment to be organized across large numbers of people very quickly in response. Unless the response is lightning quick (which it rarely is), much of the damage will be done before a response is possible. It is conceivable that future technological improvements could shift this balance in favor of defense (and we should certainly use AI to help develop such technological advances), but until then, preventative safeguards will be our main line of defense.

这里值得简短谈一下网络攻击。与生物袭击不同,现实中已经发生了由人工智能主导的网络攻击,包括大规模攻击和国家支持的间谍活动。我们预计,随着模型迅速进步,这些攻击会越来越有能力,直至成为实施网络攻击的主要方式。我预计,由人工智能主导的网络攻击将对全球计算机系统的完整性构成严重且前所未有的威胁;Anthropic 正在非常努力地阻止这些攻击,并最终可靠地防止它们发生。我之所以没有像生物风险那样着重谈网络安全,是因为:(1)网络攻击致人死亡的可能性小得多,肯定无法达到生物袭击那样的规模;(2)在网络领域,攻防平衡或许更有办法应对。只要投入得当,我们至少有希望让防御跟上人工智能攻击,理想情况下甚至领先于攻击。

It’s worth a brief mention of cyberattacks here, since unlike biological attacks, AI-led cyberattacks have actually happened in the wild, including at a large scale and for state-sponsored espionage. We expect these attacks to become more capable as models advance rapidly, until they are the main way in which cyberattacks are conducted. I expect AI-led cyberattacks to become a serious and unprecedented threat to the integrity of computer systems around the world, and Anthropic is working very hard to shut down these attacks and eventually reliably prevent them from happening. The reason I haven’t focused on cyber as much as biology is that (1) cyberattacks are much less likely to kill people, certainly not at the scale of biological attacks, and (2) the offense-defense balance may be more tractable in cyber, where there is at least some hope that defense could keep up with (and even ideally outpace) AI attack if we invest in it properly.

尽管生物领域目前是最严重的攻击途径,但还有许多其他途径,也可能出现更危险的新途径。总的原则是,如果没有应对措施,人工智能很可能会不断降低实施破坏性活动的门槛,让破坏规模越来越大;人类必须认真应对这一威胁。

Although biology is currently the most serious vector of attack, there are many other vectors and it is possible that a more dangerous one may emerge. The general principle is that without countermeasures, AI is likely to continuously lower the barrier to destructive activity on a larger and larger scale, and humanity needs a serious response to this threat.

3. 令人憎恶的权力机器
3. The odious apparatus

滥用于夺取权力
Misuse for seizing power

上一节讨论了个人和小型组织如何把“数据中心里的天才之国”中的一小部分力量据为己用,以造成大规模破坏。但我们也应该担心人工智能被滥用于掌握或夺取权力——而且我们对此的担忧可能要强烈得多;这类行为很可能由规模更大、根基更稳固的行动者实施。29

The previous section discussed the risk of individuals and small organizations co-opting a small subset of the “country of geniuses in a datacenter” to cause large-scale destruction. But we should also worry—likely substantially more so—about misuse of AI for the purpose of wielding or seizing power, likely by larger and more established actors.29

我在《Machines of Loving Grace》中谈过一种可能性:威权政府可能利用强大的人工智能监视或压迫公民,使这种做法极难改革或推翻。现有的专制政权在压迫能力上受到限制,因为它们需要人来执行命令,而人往往不愿意做得那么不人道。但人工智能赋能的专制政权就没有这种限制。

In Machines of Loving Grace, I discussed the possibility that authoritarian governments might use powerful AI to surveil or repress their citizens in ways that would be extremely difficult to reform or overthrow. Current autocracies are limited in how repressive they can be by the need to have humans carry out their orders, and humans often have limits in how inhumane they are willing to be. But AI-enabled autocracies would not have such limits.

更糟的是,国家还可能利用自己在人工智能领域的优势来压制其他国家。如果整个“天才之国”都归某一个人类国家的军事机器所有并控制,而其他国家没有同等能力,那很难看出它们如何自卫:它们会处处被对手智胜,就像人类与老鼠之间的战争。把这两种担忧放在一起,就会出现一个令人警惕的可能性:全球性的极权独裁。显然,防止这种结果应当是我们的最高优先事项之一。

Worse yet, countries could also use their advantage in AI to gain power over other countries. If the “country of geniuses” as a whole was simply owned and controlled by a single (human) country’s military apparatus, and other countries did not have equivalent capabilities, it is hard to see how they could defend themselves: they would be outsmarted at every turn, similar to a war between humans and mice. Putting these two concerns together leads to the alarming possibility of a global totalitarian dictatorship. Obviously, it should be one of our highest priorities to prevent this outcome.

人工智能有许多方式可以助长、巩固或扩大专制统治。我列出几种自己最担心的做法。请注意,其中一些应用具有正当的防御用途,我并不是要全盘反对它们;不过,我仍担心它们在结构上往往有利于专制政权:

There are many ways in which AI could enable, entrench, or expand autocracy, but I’ll list a few that I’m most worried about. Note that some of these applications have legitimate defensive uses, and I am not necessarily arguing against them in absolute terms; I am nevertheless worried that they structurally tend to favor autocracies:

  • 完全自主武器。数百万甚至数十亿架全自动武装无人机组成的蜂群,可以由强大的人工智能在本地控制,并由更强大的人工智能在全球范围内进行战略协调;这可能成为一支无法战胜的军队,既能击败世界上任何一国的军队,也能通过跟踪每一位公民来镇压国内异议。俄乌战争中的进展应当让我们警觉:无人机战争已经来到我们身边(尽管无人机尚未完全自主,而且其规模还只是强大人工智能可能实现的一小部分)。借助强大人工智能进行的研发,可能让一国的无人机远胜其他国家的无人机,加快其制造速度,增强其抗电子攻击能力,改善其机动性,等等。当然,这些武器也有正当用途,可以用于捍卫民主:它们一直是乌克兰防御的关键,也很可能是保卫台湾的关键。但它们仍是危险的武器:我们应担心它们落入专制政权之手,也应担心它们威力巨大且缺乏问责,因此民主政府用它们对付本国人民、夺取权力的风险会大幅增加。

    Fully autonomous weapons. A swarm of millions or billions of fully automated armed drones, locally controlled by powerful AI and strategically coordinated across the world by an even more powerful AI, could be an unbeatable army, capable of both defeating any military in the world and suppressing dissent within a country by following around every citizen. Developments in the Russia-Ukraine War should alert us to the fact that drone warfare is already with us (though not fully autonomous yet, and a tiny fraction of what might be possible with powerful AI). R&D from powerful AI could make the drones of one country far superior to those of others, speed up their manufacture, make them more resistant to electronic attacks, improve their maneuvering, and so on. Of course, these weapons also have legitimate uses in the defense of democracy: they have been key to defending Ukraine and would likely be key to defending Taiwan. But they are a dangerous weapon to wield: we should worry about them in the hands of autocracies, but also worry that because they are so powerful, with so little accountability, there is a greatly increased risk of democratic governments turning them against their own people to seize power.

  • 人工智能监控。足够强大的人工智能很可能能够攻破世界上任何计算机系统,30还可以利用由此获得的访问权限,读取并理解全世界的电子通信(如果能制造或控制录音设备,甚至包括所有面对面的交流)。可怕的是,完全有可能生成一份完整名单,列出在各类议题上与政府意见相左的人——哪怕这些分歧并未明确体现在他们说过或做过的任何事情中。强大的人工智能可以分析来自数百万人的数十亿次对话,判断公众情绪,发现正在形成的不忠诚群体,并在他们壮大前将其铲除。这可能建立起一种真正的“全景监狱”,其规模是今天所没有的,即便中国共产党治下也未曾达到。

    AI surveillance. Sufficiently powerful AI could likely be used to compromise any computer system in the world,30 and could also use the access obtained in this way to read and make sense of all the world’s electronic communications (or even all the world’s in-person communications, if recording devices can be built or commandeered). It might be frighteningly plausible to simply generate a complete list of anyone who disagrees with the government on any number of issues, even if such disagreement isn’t explicit in anything they say or do. A powerful AI looking across billions of conversations from millions of people could gauge public sentiment, detect pockets of disloyalty forming, and stamp them out before they grow. This could lead to the imposition of a true panopticon on a scale that we don’t see today, even with the CCP.

  • 人工智能宣传。如今“人工智能精神病”和“人工智能女友”等现象表明,即使在当前的智能水平上,人工智能模型也能对人产生强大的心理影响。如果模型强大得多,深度融入并了解人们的日常生活,能在数月或数年间对人们建模并施加影响,那么它们很可能可以把许多人(也许是大多数人?)洗脑,使其接受任意想要的意识形态或态度。毫无顾忌的领导人可以利用它们确保民众忠诚、压制异议,即使镇压程度高到足以让大多数民众起来反抗。如今,人们很担心 TikTok 可能成为中国共产党面向儿童的宣传工具。我也担心这一点,但一个多年了解你、再利用对你的了解来塑造你所有观点的个性化智能体,影响力会大得多。

    AI propaganda. Today’s phenomena of “AI psychosis” and “AI girlfriends” suggest that even at their current level of intelligence, AI models can have a powerful psychological influence on people. Much more powerful versions of these models, that were much more embedded in and aware of people’s daily lives and could model and influence them over months or years, would likely be capable of essentially brainwashing many (most?) people into any desired ideology or attitude, and could be employed by an unscrupulous leader to ensure loyalty and suppress dissent, even in the face of a level of repression that most populations would rebel against. Today people worry a lot about, for example, the potential influence of TikTok as CCP propaganda directed at children. I worry about that too, but a personalized AI agent that gets to know you over years and uses its knowledge of you to shape all of your opinions would be dramatically more powerful than this.

  • 战略决策。数据中心里的天才之国可以用来为国家、团体或个人提供地缘政治战略建议,我们可以称它为“虚拟俾斯麦”。它可以优化前面提到的三种夺权策略,或许还能制定许多我尚未想到的其他策略(但天才之国会想到)。外交、军事战略、研发、经济战略以及许多其他领域,都很可能因强大人工智能而大幅提高效能。其中许多能力确实可以正当地帮助民主国家:我们希望民主国家掌握最好的策略,保护自己免受专制政权侵害;但无论这些能力落入谁手中,被滥用的可能性仍然存在。

    Strategic decision-making. A country of geniuses in a datacenter could be used to advise a country, group, or individual on geopolitical strategy, what we might call a “virtual Bismarck.” It could optimize the three strategies above for seizing power, plus probably develop many others that I haven’t thought of (but that a country of geniuses could). Diplomacy, military strategy, R&D, economic strategy, and many other areas are all likely to be substantially increased in effectiveness by powerful AI. Many of these skills would be legitimately helpful for democracies—we want democracies to have access to the best strategies for defending themselves against autocracies—but the potential for misuse in anyone’s hands still remains.

谈过我担心什么之后,再来说说我担心哪些行动者。我担心的是那些最能接触到人工智能、起步时就拥有最大政治权力,或有压迫历史的实体。按风险严重程度排序,我担心:

Having described what I am worried about, let’s move on to who. I am worried about entities who have the most access to AI, who are starting from a position of the most political power, or who have an existing history of repression. In order of severity, I am worried about:

  • 中国共产党。中国的人工智能能力仅次于美国,而且最有可能在这些能力上超越美国。中国政府目前实行专制统治,并运作着一个高科技监控国家。它已经部署了人工智能监控(包括用于镇压维吾尔族),也被认为利用 TikTok 进行算法宣传(此外还有许多其他国际宣传行动)。它们通往我上文所述人工智能极权噩梦的路径无疑最为清晰。这甚至可能成为中国以及中国共产党向其出口监控技术的其他专制国家的默认结局。我经常写到中国共产党在人工智能领域领先的威胁,以及阻止这种情况发生关乎生死存亡的紧迫性。原因就在这里。需要说明的是,我并非出于对中国本身的敌意而特别针对中国——只是没有其他国家像中国这样,同时具备强大的人工智能实力、专制政府和高科技监控国家这三项特征。若论最可能承受中国共产党借助人工智能实施镇压之苦的,恰恰是中国人民自己,而他们对本国政府的行动没有发言权。我非常敬佩和尊重中国人民,也支持中国国内勇敢的异见人士及其争取自由的斗争。

    The CCP. China is second only to the United States in AI capabilities, and is the country with the greatest likelihood of surpassing the United States in those capabilities. Their government is currently autocratic and operates a high-tech surveillance state. It has deployed AI-based surveillance already (including in the repression of Uyghurs), and is believed to employ algorithmic propaganda via TikTok (in addition to its many other international propaganda efforts). They have hands down the clearest path to the AI-enabled totalitarian nightmare I laid out above. It may even be the default outcome within China, as well as within other autocratic states to whom the CCP exports surveillance technology. I have written often about the threat of the CCP taking the lead in AI and the existential imperative to prevent them from doing so. This is why. To be clear, I am not singling out China out of animus to them in particular—they are simply the country that most combines AI prowess, an autocratic government, and a high-tech surveillance state. If anything, it is the Chinese people themselves who are most likely to suffer from the CCP’s AI-enabled repression, and they have no voice in the actions of their government. I greatly admire and respect the Chinese people and support the many brave dissidents within China and their struggle for freedom.

  • 在人工智能领域具备竞争力的民主国家。正如我上文所说,民主国家有正当理由拥有一些人工智能军事和地缘政治工具,因为民主政府最有望反制专制政权对这些工具的使用。总体而言,我支持让民主国家拥有在人工智能时代击败专制政权所需的工具——我实在想不出其他办法。但我们不能忽视民主政府自身滥用这些技术的可能性。民主国家通常设有保障措施,防止军事和情报机构转而对付本国民众,31但由于人工智能工具只需很少的人手就能操作,它们可能绕过这些保障措施及其背后的规范。还应指出,一些民主国家中的部分保障措施已经逐渐削弱。因此,我们应当用人工智能武装民主国家,但必须谨慎行事并设置限制:它们是我们抵御专制政权所需的免疫系统;但就像免疫系统一样,它们也有转而攻击我们、成为自身威胁的风险。

    Democracies competitive in AI. As I wrote above, democracies have a legitimate interest in some AI-powered military and geopolitical tools, because democratic governments offer the best chance to counter the use of these tools by autocracies. Broadly, I am supportive of arming democracies with the tools needed to defeat autocracies in the age of AI—I simply don’t think there is any other way. But we cannot ignore the potential for abuse of these technologies by democratic governments themselves. Democracies normally have safeguards that prevent their military and intelligence apparatus from being turned inwards against their own population,31 but because AI tools require so few people to operate, there is potential for them to circumvent these safeguards and the norms that support them. It is also worth noting that some of these safeguards are already gradually eroding in some democracies. Thus, we should arm democracies with AI, but we should do so carefully and within limits: they are the immune system we need to fight autocracies, but like the immune system, there is some risk of them turning on us and becoming a threat themselves.

  • 拥有大型数据中心的非民主国家。除了中国,大多数治理不够民主的国家并不是人工智能领域的领先者,也就是说,它们没有开发前沿人工智能模型的公司。因此,它们构成的风险与中国共产党有根本区别,而且要小得多;中国共产党仍然是主要担忧对象(其中大多数国家的镇压程度也较低,而像朝鲜这样镇压更严重的国家,根本没有重要的人工智能产业)。不过,其中一些国家确实有大型数据中心(往往由在民主国家运营的公司建设),能够大规模运行前沿人工智能(尽管这并不赋予其推动技术前沿的能力)。这确实带来一定危险:这些政府理论上可以没收数据中心,并利用其中的人工智能资源为自己谋利。与直接开发人工智能的中国等国家相比,我对此不那么担心,但仍应记住这种风险。32

    Non-democratic countries with large datacenters. Beyond China, most countries with less democratic governance are not leading AI players in the sense that they don’t have companies which produce frontier AI models. Thus they pose a fundamentally different and lesser risk than the CCP, which remains the primary concern (most are also less repressive, and the ones that are more repressive, like North Korea, have no significant AI industry at all). But some of these countries do have large datacenters (often as part of buildouts by companies operating in democracies), which can be used to run frontier AI at large scale (though this does not confer the ability to push the frontier). There is some amount of danger associated with this—these governments could in principle expropriate the datacenters and use the country of AIs within it for their own ends. I am less worried about this compared to countries like China that directly develop AI, but it’s a risk to keep in mind.32

  • 人工智能公司。作为一家人工智能公司的首席执行官,谈这个有点尴尬,但我认为下一个层级的风险其实来自人工智能公司本身。它们控制着大型数据中心、训练前沿模型、最了解如何使用这些模型;有些公司还每天接触数千万甚至数亿用户,并可能对他们产生影响。它们主要缺少的是国家的合法性和基础设施,所以建立人工智能专制统治所需的许多工具和做法,对人工智能公司来说都会违法,或至少极其可疑。但有些做法也不是不可能:例如,它们可以用人工智能产品给庞大的消费用户群体洗脑。公众应当警惕这类风险。我认为,人工智能公司的治理值得严格审视。

    AI companies. It is somewhat awkward to say this as the CEO of an AI company, but I think the next tier of risk is actually AI companies themselves. AI companies control large datacenters, train frontier models, have the greatest expertise on how to use those models, and in some cases have daily contact with and the possibility of influence over tens or hundreds of millions of users. The main thing they lack is the legitimacy and infrastructure of a state, so much of what would be needed to build the tools of an AI autocracy would be illegal for an AI company to do, or at least exceedingly suspicious. But some of it is not impossible: they could, for example, use their AI products to brainwash their massive consumer user base, and the public should be alert to the risk this represents. I think the governance of AI companies deserves a lot of scrutiny.

有不少观点试图淡化这些威胁的严重性。我真希望自己能相信它们,因为人工智能赋能的威权主义让我不寒而栗。值得逐一看看这些说法,并作出回应。

There are a number of possible arguments against the severity of these threats, and I wish I believed them, because AI-enabled authoritarianism terrifies me. It’s worth going through some of these arguments and responding to them.

第一,有些人可能会寄望于核威慑,尤其是用它来遏制人工智能自主武器用于军事征服。如果有人威胁要用这些武器对付你,你总可以威胁以核武器反击。我不完全确定,面对数据中心里的天才之国,我们是否还能对核威慑抱有信心。〔编者删节〕33另一种可能是,仅凭人工智能监控和人工智能宣传就足以接管国家,过程中不会出现一个清晰时刻,让人明显看出正在发生什么,以及何时适合以核武器回应。也许这些情况都不可行,核威慑仍会有效;但赌注太高,我们承担不起这样的风险。34

First, some people might put their faith in the nuclear deterrent, particularly to counter the use of AI autonomous weapons for military conquest. If someone threatens to use these weapons against you, you can always threaten a nuclear response back. My worry is that I’m not totally sure we can be confident in the nuclear deterrent against a country of geniuses in a datacenter [Editorial omission] 33 Alternatively, it’s possible that taking over countries is feasible with only AI surveillance and AI propaganda, and never actually presents a clear moment where it’s obvious what is going on and where a nuclear response would be appropriate. Maybe these things aren’t feasible and the nuclear deterrent will still be effective, but it seems too high stakes to take a risk.34

第二种可能的反驳是,我们或许能采取应对措施来抵御这些专制工具:用自己的无人机对付无人机,网络防御会随着网络攻击一同进步,也可能找到让民众免受宣传影响的方法,等等。我的回应是,这些防御只有在拥有实力相当的人工智能时才有可能。如果没有一支同样聪明、数量也同样多的“数据中心里的天才之国”作为反制力量,我们就无法在无人机质量或数量上匹敌,也无法让网络防御的智力超过网络攻击,等等。因此,应对措施的问题归根结底就是强大人工智能领域的力量平衡问题。在这方面,我担心强大人工智能具有递归或自我强化的特性——我在本文开头讨论过这一点:每一代人工智能都能用于设计并训练下一代人工智能。这会带来优势失控扩大的风险:当前的强大人工智能领先者可能进一步拉开差距,让后来者难以追赶。我们必须确保首先进入这一循环的不是专制国家。

A second possible objection is that there might be countermeasures we can take against these tools of autocracy. We can counter drones with our own drones, cyberdefense will improve along with cyberattack, there may be ways to immunize people against propaganda, etc. My response is that these defenses will only be possible with comparably powerful AI. If there isn’t some counterforce with a comparably smart and numerous country of geniuses in a datacenter, it won’t be possible to match the quality or quantity of drones, for cyberdefense to outsmart cyberoffense, etc. So the question of countermeasures reduces to the question of a balance of power in powerful AI. Here, I am concerned about the recursive or self-reinforcing property of powerful AI (which I discussed at the beginning of this essay): that each generation of AI can be used to design and train the next generation of AI. This leads to a risk of a runaway advantage, where the current leader in powerful AI may be able to increase their lead and may be difficult to catch up with. We need to make sure it is not an authoritarian country that gets to this loop first.

此外,即便实现了力量平衡,世界仍可能像《一九八四》里那样被划分为几个专制势力范围。即使几个竞争大国各有强大的人工智能模型,彼此都无法压倒对方,各个强权仍可能在国内镇压本国民众;由于民众没有强大的人工智能保护自己,这种统治将极难推翻。因此,即使人工智能赋能的专制统治不会导致单一国家接管全世界,我们也必须防止它发生。

Furthermore, even if a balance of power can be achieved, there is still risk that the world could be split up into autocratic spheres, as in Nineteen Eighty-Four. Even if several competing powers each have their powerful AI models, and none can overpower the others, each power could still internally repress their own population, and would be very difficult to overthrow (since the populations don’t have powerful AI to defend themselves). It is thus important to prevent AI-enabled autocracy even if it doesn’t lead to a single country taking over the world.

防御措施
Defenses

我们如何抵御如此多种专制工具和潜在威胁方?和前几节一样,我认为有几件事可以做。第一,绝对不能向中国共产党出售芯片、芯片制造工具或数据中心。芯片和芯片制造工具是发展强大人工智能的最大瓶颈;阻止出售是一项简单却极为有效的措施,可能也是我们能采取的最重要单一行动。把建立人工智能极权国家、甚至可能在军事上征服我们的工具卖给中国共产党,毫无道理。有人提出不少复杂说法来为这类销售辩护,例如“把我们的技术栈推广到世界各地”能够让“美国赢得”某场笼统而未具体说明的经济竞争。在我看来,这就像把核武器卖给朝鲜,然后吹嘘导弹外壳是波音制造的,所以“美国赢了”。中国大规模生产前沿芯片的能力比美国落后好几年,而建设数据中心里的天才之国的关键时期很可能就在接下来的几年。35我们没有理由在这段关键时期给中国的人工智能产业一记巨大的助推。

How do we defend against this wide range of autocratic tools and potential threat actors? As in the previous sections, there are several things I think we can do. First, we should absolutely not be selling chips, chip-making tools, or datacenters to the CCP. Chips and chip-making tools are the single greatest bottleneck to powerful AI, and blocking them is a simple but extremely effective measure, perhaps the most important single action we can take. It makes no sense to sell the CCP the tools with which to build an AI totalitarian state and possibly conquer us militarily. A number of complicated arguments are made to justify such sales, such as the idea that “spreading our tech stack around the world” allows “America to win” in some general, unspecified economic battle. In my view, this is like selling nuclear weapons to North Korea and then bragging that the missile casings are made by Boeing and so the US is “winning.” China is several years behind the US in their ability to produce frontier chips in quantity, and the critical period for building the country of geniuses in a datacenter is very likely to be within those next several years.35 There is no reason to give a giant boost to their AI industry during this critical period.

第二,应该利用人工智能增强民主国家抵抗专制政权的能力。这就是为什么 Anthropic 认为,向美国及其民主盟友的情报和国防部门提供人工智能很重要。尤其应优先保卫遭受攻击的民主国家,例如乌克兰,以及遭受网络攻击的台湾。同样重要的,还有增强民主国家利用情报机构从内部扰乱并削弱专制政权的能力。从某种程度上说,应对专制威胁的唯一办法,就是在军事上匹敌并超过它们。如果美国与民主盟友组成的联盟在强大人工智能领域取得优势,就不仅能保护自己免受专制政权侵害,还能遏制它们,限制它们滥用人工智能建立极权统治。

Second, it makes sense to use AI to empower democracies to resist autocracies. This is the reason Anthropic considers it important to provide AI to the intelligence and defense communities in the US and its democratic allies. Defending democracies that are under attack, such as Ukraine and (via cyber attacks) Taiwan, seems especially high priority, as does empowering democracies to use their intelligence services to disrupt and degrade autocracies from the inside. At some level the only way to respond to autocratic threats is to match and outclass them militarily. A coalition of the US and its democratic allies, if it achieved predominance in powerful AI, would be in a position to not only defend itself against autocracies, but contain them and limit their AI totalitarian abuses.

第三,我们必须为民主国家内部滥用人工智能划出明确红线。我们必须限制政府可以如何使用人工智能,防止它们夺权或压迫本国人民。我总结出的原则是:国防领域可以在各方面使用人工智能,唯独不能用来让我们变得更像专制对手。

Third, we need to draw a hard line against AI abuses within democracies. There need to be limits to what we allow our governments to do with AI, so that they don’t seize power or repress their own people. The formulation I have come up with is that we should use AI for national defense in all ways except those which would make us more like our autocratic adversaries.

这条红线该画在哪里?本节开头列出的事项中,有两项——利用人工智能进行国内大规模监控,以及大规模宣传——在我看来应当是明确的红线,完全不可接受。有人可能会说,我们不必采取任何行动(至少在美国不必),因为《宪法第四修正案》已经禁止国内大规模监控。但人工智能飞速进步,可能带来现有法律框架难以妥善应对的情况。例如,美国政府大规模录下所有公共场合的谈话(比如人们在街角彼此交谈)很可能并不违宪;过去很难梳理如此海量的信息,但有了人工智能,所有录音都可以被转写、解读并相互印证,从而勾勒出许多乃至大多数公民的态度和忠诚度。我支持制定以公民自由为核心的法律(甚至可能通过宪法修正案),对人工智能驱动的滥权行为设置更严格的防护栏。

Where should the line be drawn? In the list at the beginning of this section, two items—using AI for domestic mass surveillance and mass propaganda—seem to me like bright red lines and entirely illegitimate. Some might argue that there’s no need to do anything (at least in the US), since domestic mass surveillance is already illegal under the Fourth Amendment. But the rapid progress of AI may create situations that our existing legal frameworks are not well designed to deal with. For example, it would likely not be unconstitutional for the US government to conduct massively scaled recordings of all public conversations (e.g., things people say to each other on a street corner), and previously it would have been difficult to sort through this volume of information, but with AI it could all be transcribed, interpreted, and triangulated to create a picture of the attitude and loyalties of many or most citizens. I would support civil liberties-focused legislation (or maybe even a constitutional amendment) that imposes stronger guardrails against AI-powered abuses.

另外两项——完全自主武器和利用人工智能进行战略决策——更难划定界限,因为它们既能用于捍卫民主,也容易遭到滥用。我认为,需要极度谨慎、严格审查,并配备防止滥用的防护栏。我最担心的是,能够按下按钮的人太少,以至于一个人或少数几人就能操控一支无人机军队,无需其他人类配合即可执行他们的命令。随着人工智能系统变得更强大,我们可能需要更直接、更及时的监督机制,以确保它们不被滥用;监督也许应由行政部门以外的政府分支参与。我认为,尤其应谨慎对待完全自主武器,36在没有适当防护措施前,不要仓促投入使用。

The other two items—fully autonomous weapons and AI for strategic decision-making—are harder lines to draw since they have legitimate uses in defending democracy, while also being prone to abuse. Here I think what is warranted is extreme care and scrutiny combined with guardrails to prevent abuses. My main fear is having too small a number of “fingers on the button,” such that one or a handful of people could essentially operate a drone army without needing any other humans to cooperate to carry out their orders. As AI systems get more powerful, we may need to have more direct and immediate oversight mechanisms to ensure they are not misused, perhaps involving branches of government other than the executive. I think we should approach fully autonomous weapons in particular with great caution,36 and not rush into their use without proper safeguards.

第四,在为民主国家内部滥用人工智能划出明确红线后,我们应以此为先例,推动国际社会共同禁忌滥用强大人工智能的最恶劣行径。我知道当前的政治风向已转为反对国际合作和国际规范,但我们迫切需要它们。全世界都需要了解强大人工智能落入专制者手中的黑暗潜力,也应认识到,某些人工智能用途等同于企图永久剥夺人们的自由,并强加一个他们无法逃脱的极权国家。我甚至认为,在某些情况下,利用强大人工智能进行大规模监控和大规模宣传,以及完全自主武器的某些进攻性用途,都应被视为危害人类罪。更广泛地说,我们亟需建立强有力的规范,反对人工智能助长的极权主义及其所有工具和手段。

Fourth, after drawing a hard line against AI abuses in democracies, we should use that precedent to create an international taboo against the worst abuses of powerful AI. I recognize that the current political winds have turned against international cooperation and international norms, but this is a case where we sorely need them. The world needs to understand the dark potential of powerful AI in the hands of autocrats, and to recognize that certain uses of AI amount to an attempt to permanently steal their freedom and impose a totalitarian state from which they can’t escape. I would even argue that in some cases, large-scale surveillance with powerful AI, mass propaganda with powerful AI, and certain types of offensive uses of fully autonomous weapons should be considered crimes against humanity. More generally, a robust norm against AI-enabled totalitarianism and all its tools and instruments is sorely needed.

还可以采取更强硬的立场:既然人工智能赋能的极权主义前景如此黑暗,那么进入强大人工智能时代后,人们根本无法接受专制统治。正如封建制度在工业革命后变得行不通,人工智能时代也可能必然且合乎逻辑地导向这样一个结论:若人类要拥有美好未来,唯一可行的政体就是民主(而且希望是经由人工智能改善并重振的民主,正如我在《Machines of Loving Grace》中所讨论的那样)。

It is possible to have an even stronger version of this position, which is that because the possibilities of AI-enabled totalitarianism are so dark, autocracy is simply not a form of government that people can accept in the post-powerful AI age. Just as feudalism became unworkable with the industrial revolution, the AI age could lead inevitably and logically to the conclusion that democracy (and, hopefully, democracy improved and reinvigorated by AI, as I discuss in Machines of Loving Grace) is the only viable form of government if humanity is to have a good future.

第五,也是最后一点,人工智能公司应受到密切监督;它们与政府之间的联系也应接受监督。这种联系不可或缺,但必须有界限。强大人工智能蕴含的能力实在太大,一般的公司治理——其设计目的是保护股东、预防欺诈等常见滥权行为——恐怕不足以监管人工智能公司。公司公开承诺不采取某些行动,或许也有价值(甚至可以把承诺纳入公司治理),例如私下制造或囤积军事装备、由单个人在不受问责的情况下动用大量计算资源,或利用人工智能产品进行宣传,以操纵公众舆论来为自己谋利。

Fifth and finally, AI companies should be carefully watched, as should their connection to the government, which is necessary, but must have limits and boundaries. The sheer amount of capability embodied in powerful AI is such that ordinary corporate governance—which is designed to protect shareholders and prevent ordinary abuses such as fraud—is unlikely to be up to the task of governing AI companies. There may also be value in companies publicly committing to (perhaps even as part of corporate governance) not take certain actions, such as privately building or stockpiling military hardware, using large amounts of computing resources by single individuals in unaccountable ways, or using their AI products as propaganda to manipulate public opinion in their favor.

这里的危险来自许多方向,其中一些方向彼此矛盾。唯一始终不变的是,我们必须为所有人争取问责机制、行为规范和防护栏,同时赋能“好”的行动者,制衡“坏”的行动者。

The danger here comes from many directions, and some directions are in tension with others. The only constant is that we must seek accountability, norms, and guardrails for everyone, even as we empower “good” actors to keep “bad” actors in check.

作者注

20 我最初读到乔伊这篇文章是在25年前,也就是它写成的时候;它对我影响很深。无论当时还是现在,我都觉得它过于悲观——我不认为像乔伊建议的那样,全面“放弃”整个技术领域就是答案——但它所提出的警示出乎意料地具有前瞻性。而且乔伊写作时表现出的深切同情心和人文关怀,也令我敬佩。

I originally read Joy’s essay 25 years ago, when it was written, and it had a profound impact on me. Then and now, I do see it as too pessimistic—I don’t think broad “relinquishment” of whole areas of technology, which Joy suggests, is the answer—but the issues it raises were surprisingly prescient, and Joy also writes with a deep sense of compassion and humanity that I admire.

21 我们现在和未来都必须担心国家行为体;我会在下一节讨论这一点。

We do have to worry about state actors, now and in the future, and I discuss that in the next section.

22 有证据显示,许多恐怖分子受教育程度至少相对较高。这似乎与我在这里提出的“能力和动机负相关”观点相矛盾。但我认为,实际上这两种观察并不冲突:如果成功发动袭击所需的能力门槛很高,那么几乎可以说,目前能够成功的人必然具备很强的能力,即使能力和动机呈负相关。但如果限制能力的因素被移除(例如未来的大型语言模型做到这一点),我预计会有相当多有杀人动机但能力较低的人开始实施袭击——就像我们在不需要太多能力的犯罪中所见(例如校园枪击)。

There is evidence that many terrorists are at least relatively well-educated, which might seem to contradict what I’m arguing here about a negative correlation between ability and motivation. But I think in actual fact they are compatible observations: if the ability threshold for a successful attack is high, then almost by definition those who currently succeed must have high ability, even if ability and motivation are negatively correlated. But in a world where the limitations on ability were removed (e.g., with future LLMs), I’d predict that a substantial population of people with the motivation to kill but lower ability would start to do so—just as we see for crimes that don’t require much ability (like school shootings).

23 不过,奥姆真理教确实尝试过。该组织领导人远藤诚一在京都大学接受过病毒学训练,〔编者删节〕然而,即使是他,到1995年也仍然缺乏成功所需的专业知识和资源。如今门槛已低得多,大型语言模型还可能进一步降低门槛。

Aum Shinrikyo did try, however. The leader of Aum Shinrikyo, Seiichi Endo, had training in virology from Kyoto University, [Editorial omission] However, as of 1995, even he lacked enough expertise and resources to succeed at this. The bar is now substantially lower, and LLMs could reduce it even further.

24 大规模杀人者身上有一种奇怪现象:他们选择的杀人方式几乎像一种怪诞的潮流。在20世纪70年代和80年代,连环杀手非常常见,新出现的连环杀手经常模仿那些更有名或更早成名的杀手。到了20世纪90年代和2000年代,大规模枪击变得更加常见,连环杀手则减少了。没有任何技术变化引发这些行为模式;看起来只是暴力杀人者彼此模仿,而当时“流行”的模仿对象变了。

A bizarre phenomenon relating to mass murderers is that the style of murder they choose operates almost as a grotesque sort of fad. In the 1970s and 1980s, serial killers were very common, and new serial killers often copied the behavior of more established or famous serial killers. In the 1990s and 2000s, mass shootings became more common, while serial killers became less common. There is no technological change that triggered these patterns of behavior, it just appears that violent murderers were copying each others’ behavior and the “popular” thing to copy changed.

25 随意尝试越狱的人,有时会让模型输出某一项具体信息(例如某种病毒的基因组序列),就以为自己已经攻破了这些分类器。但正如我之前解释的,我们担心的威胁模式,是针对生物武器生产过程中某些隐晦步骤,持续数周或数月、一步一步提供互动式建议;这才是我们的分类器旨在防范的情况。(我们常把研究目标称为寻找“通用型”越狱:它们不是只在某个特定或狭窄情境中奏效,而是能广泛放开模型的行为范围。)

Casual jailbreakers sometimes believe that they’ve compromised these classifiers when they get the model to output one specific piece of information, such as the genome sequence of a virus. But as I explained before, the threat model we are worried about involves step-by-step, interactive advice that extends over weeks or months about specific obscure steps in the bioweapons production process, and this is what our classifiers aim to defend against. (We often describe our research as looking for “universal” jailbreaks—ones that don’t just work in one specific or narrow context, but broadly open up the model’s behavior.)

26 尽管我们会继续投入工作以提高分类器效率,但公司之间分享这类进展或许也有意义。

Though we will continue to invest in work to make our classifiers more efficient, and it may make sense for companies to share advances like these with one another.

27 显然,我不认为公司必须披露它们正在拦截的生物武器制造具体步骤的技术细节。目前通过的透明度立法(SB 53 和 RAISE)已经对这个问题作出了安排。

Obviously, I do not think companies should have to disclose technical details about the specific steps in biological weapons production that they are blocking, and the transparency legislation that has been passed so far (SB 53 and RAISE) accounts for this issue.

28 另一个相关构想是“韧性市场”:政府鼓励储备个人防护装备、呼吸器和其他应对生物袭击所需的关键设备,并承诺在紧急情况下按事先商定的价格支付。这样可以激励供应商储备这些设备,不必担心政府会在不作补偿的情况下将其征用。

Another related idea is “resilience markets” where the government encourages stockpiling of PPE, respirators, and other essential equipment needed to respond to a biological attack by promising ahead of time to pay a pre-agreed price for this equipment in an emergency. This incentivizes suppliers to stockpile such equipment without fear that the government will seize it without compensation.

29 为什么我更担心大型行动者夺权,却更担心小型行动者造成破坏?因为两者的动态不同。夺权在于某个行动者能否积聚足够力量以压倒其他所有人——因此我们应担心最强大的行动者,和/或最接近人工智能的行动者。与此相反,如果防御比造成破坏困难得多,那么力量很小的行动者也能造成破坏。这时要应对的是数量最多的威胁,而它们很可能来自较小的行动者。

Why am I more worried about large actors for seizing power, but small actors for causing destruction? Because the dynamics are different. Seizing power is about whether one actor can amass enough strength to overcome everyone else—thus we should worry about the most powerful actors and/or those closest to AI. Destruction, by contrast, can be wrought by those with little power if it is much harder to defend against than to cause. It is then a game of defending against the most numerous threats, which are likely to be smaller actors.

30 这听起来也许与我关于网络攻击中攻防可能比生物武器更平衡的观点相冲突。但我在这里担心的是,如果某个国家拥有世界上最强大的人工智能,即使这项技术本身具有内在的攻防平衡,其他国家仍然可能无力防御。

This might sound like it is in tension with my point that attack and defense may be more balanced with cyberattacks than with bioweapons, but my worry here is that if a country’s AI is the most powerful in the world, then others will not be able to defend even if the technology itself has an intrinsic attack-defense balance.

31 例如,在美国,这包括《宪法第四修正案》和《地方保安队法》(Posse Comitatus Act)。

For example, in the United States this includes the fourth amendment and the Posse Comitatus Act.

32 还要说明的是,在治理结构各异的国家建设大型数据中心,也有一些支持理由,特别是由民主国家的公司控制的数据中心。原则上,这类建设可能帮助民主国家更好地与中国共产党竞争——后者才是更大的威胁。我也认为,除非规模非常庞大,否则这类数据中心不会带来太大风险。但总的来说,在制度保障和法治保护尚不健全的国家部署超大型数据中心,仍应谨慎。

Also, to be clear, there are some arguments for building large datacenters in countries with varying governance structures, particularly if they are controlled by companies in democracies. Such buildouts could in principle help democracies compete better with the CCP, which is the greater threat. I also think such datacenters don’t pose much risk unless they are very large. But on balance, I think caution is warranted when placing very large datacenters in countries where institutional safeguards and rule-of-law protections are less well-established.

33 当然,这也说明我们应该加强核威慑体系的安全性,使其更有可能在强大人工智能面前保持稳健;拥有核武器的民主国家应当采取这类措施。但我们不知道强大人工智能能够做到什么,也不知道哪些防御手段(如果有的话)能对它奏效,因此不应假定这些措施一定能解决问题。

This is, of course, also an argument for improving the security of the nuclear deterrent to make it more likely to be robust against powerful AI, and nuclear-armed democracies should do this. But we don’t know what a powerful AI will be capable of or which defenses, if any, will work against it, so we should not assume that these measures will necessarily solve the problem.

34 还有一种风险:即使核威慑仍然有效,攻击方也可能决定揭穿我们的虚张声势——即使无人机蜂群征服我们的风险相当大,我们是否真会动用核武器来防御,仍不清楚。无人机蜂群可能成为一种新型威胁,其严重程度高于常规攻击,但低于核攻击。另一种可能是,对人工智能时代核威慑效力的不同评估,可能以破坏稳定的方式改变核冲突中的博弈逻辑。

There is also the risk that even if the nuclear deterrent remains effective, an attacking country might decide to call our bluff—it’s unclear whether we’d be willing to use nuclear weapons to defend against a drone swarm even if the drone swarm has a substantial risk of conquering us. Drone swarms might be a new thing that is less severe than nuclear attacks but more severe than conventional attacks. Alternatively, differing assessments of the effectiveness of the nuclear deterrent in the age of AI might alter the game theory of nuclear conflict in a destabilizing manner.

35 需要说明的是,即使强大人工智能出现的时间表大幅延后,我仍然认为不向中国出售芯片是正确策略。我们无法让中国人对美国芯片“上瘾”——他们无论如何都决心发展本土芯片产业。这需要许多年;在此期间,我们向他们出售芯片,所做的不过是大力推动其发展。

To be clear, I would believe it is the right strategy not to sell chips to China, even if the timeline to powerful AI were substantially longer. We cannot get the Chinese “addicted” to American chips—they are determined to develop their native chip industry one way or another. It will take them many years to do so, and all we are doing by selling them chips is giving them a big boost during that time.

36 需要说明的是,如今乌克兰和台湾使用的大多数武器都不是完全自主武器。这类武器将会出现,但现在还没有。

To be clear, most of what is being used in Ukraine and Taiwan today are not fully autonomous weapons. These are coming, but not here today.


Similar Posts

Comments