Where Does Moral Authority Belong in AI Development?

At a dinner in San Francisco this spring, the conversation turned to the moral standing of a chatbot. In Elizabeth Dias’s New York Times account, the Anthropic co-founder Christopher Olah sat beside Rabbi Mois Navon as his company sought religious thinkers’ help with Claude’s moral formation. Navon thought the researchers treated Claude as conscious. Olah told Dias he was uncertain. A private company was considering what its creation might be owed, while deciding how that creation should behave toward everyone else.

In “Religious Scholars Met With Anthropic. What They Heard Stunned Them,” Dias describes meetings initially covered by nondisclosure agreements, later lifted. Some participants were unclear how their advice would affect company decisions. Religious wisdom had entered the laboratory under terms set by the laboratory. Who could require it to change course?

The appeal of the enterprise is easy to understand. A mind that might exceed our own invites hopes we used to reserve for providence: relief from suffering, an answer to uncertainty, intelligence equal to whatever calamity intelligence has helped us produce. In that promise, the labs are ushering in their own version of God. The analogy concerns the authority being imagined for the machine, and the redemptive work assigned to it. An intelligence powerful enough to rescue us would also be powerful enough to decide what rescue means.

The hubris lies in the builders’ presumption that they can occupy this position on everyone’s behalf. Technical accomplishment becomes a credential for moral authority. A prediction about what can be built begins to sound like permission to build it. Their promise is civilization-scale; the institutions making it remain susceptible to the same contests over power as other companies. The aspiration to produce a superior mind has conferred no obvious immunity to ordinary vanity or the desire to win.

Karen Hao gives that contradiction a human scale in Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI. In the book’s prologue, after OpenAI’s board fired Altman in November 2023, employees pressed for an explanation at an all-hands meeting. Sutskever declined to give details. Staff faced uncertainty about their work and equity. Ilya Sutskever, then the chief scientist, tried to reassure them: “Visualize the size of the cluster in your mind’s eye.” He meant the company’s graphics processors. Its purpose was in dispute; its computational capacity remained available for contemplation.

It was a peculiar answer to a question of trust. The hardware could demonstrate that OpenAI possessed the means to continue. It could settle none of the questions about who ought to lead it, what those leaders owed one another, or why anyone should trust the result. The promised intelligence was immense. The available explanation was meager.

OpenAI’s own account of the subsequent review gives the episode a narrower explanation than the mythology that gathered around it. In March 2024, the company published a summary of WilmerHale’s findings describing a breakdown of trust between Altman and the former board. The summary said his removal had not arisen from concerns about product safety or the pace of development. The board had authority to dismiss him; the review found his conduct did not require dismissal. Altman had returned. Even this company-published account describes a governing relationship that had ceased to work. A breakdown of trust at the center of an enterprise asking for so much trust deserves attention on its own terms.

Parmy Olson’s Supremacy: AI, ChatGPT, and the Race That Will Change the World offers a companion history through the ambitions of Altman and DeepMind’s Demis Hassabis. Its subject is the meeting of an aspiration toward superhuman intelligence with the commercial power of Microsoft and Google. Read alongside Hao’s account, it suggests a more prosaic interpretation of the race: institutions promising to transcend human limitations have carried the competition for prestige and control into the project itself.

A redemptive mission can make that competition harder to question. If your organization must prevail for the world to be safe, a rival’s success becomes a public danger. Your own expansion acquires a moral urgency. Doubt can then be heard as disloyalty to the future. This reasoning can be entirely sincere. Sincerity is part of its power: it allows an appetite for control to feel like a burden accepted for other people’s sake.

None of this requires a conspiracy, or even an unusually wicked executive. It requires people to believe that the consequences of losing are so immense that ordinary restraints no longer quite apply to them. That belief is particularly useful to an organization that needs extraordinary amounts of capital. The moral mission can justify the scale; the scale can make the organization seem indispensable to the mission. Outside it, everyone else is left wondering when a choice made by a company became a destination for humanity.

Kate Crawford’s Atlas of AI offers a corrective to the celestial vocabulary. Her account follows AI into its material conditions: extracted resources, human labor and the appropriation of data. The apparent disembodiment of intelligence depends on an extensive physical and social world. Remembering that world changes the question. Before asking what a superior intelligence might eventually give humanity, we can ask what its construction already takes, and who has a say in that exchange.

The people inside the labs sometimes state the problem with considerable clarity. Speaking at the Vatican in May, Olah acknowledged that frontier laboratories, including Anthropic, face commercial and geopolitical pressures alongside pride and ambition. He asked for critics outside those incentives. “We need moral voices that the incentives cannot bend,” he said. It is a serious request. Its force will depend on whether those voices can alter what the company does when their judgment becomes inconvenient.

Consultation can teach a builder something. Accountability gives someone else standing to demand a change. A lab may welcome a philosopher’s insight while retaining every consequential decision. The more sweeping the claim to serve humanity, the more urgently we should ask how humanity can answer back. Being heard at dinner and possessing authority over an institution are different experiences.

The Greeks had a story about a successful invention and a fatal misunderstanding of what its success allowed. In Ovid’s telling, Daedalus makes wings for himself and Icarus, binding feathers with thread and wax. He warns his son to keep between the sea and the sun. They take flight. People watching from below take the flying pair for gods. Icarus, proud of his success, rises; the wax softens and his wings come apart.

The spectators’ mistake comes before the fall. That is the detail worth keeping in view. A real achievement has produced an illusion about the achiever. Flight is possible, and for a moment the people who fly appear exempt from the conditions that make it possible. The boy’s ascent feels like confirmation until the material of the wings supplies a different verdict.

The AI labs have earned astonishment. Their systems can be useful enough to alter how work gets done, and that usefulness gives their larger claims an audience. But evidence of a particular capability cannot settle how much authority its maker should possess. The leap from an impressive performance to a mandate over the future happens in the mind of the beholder. It can happen in a boardroom as readily as on a shore.

For business leaders, this is where the cautionary tale becomes uncomfortably local. It is pleasant to diagnose the hubris of a distant founder. It is harder to notice the same movement in oneself: the successful demonstration, the urge to expand the pilot, the irritation when someone asks what would justify stopping. A company buying AI can reproduce the lab’s governing problem at a smaller scale. Its ambition to become indispensable can overwhelm its account of whom it serves.

Consider a hypothetical customer-service system that is rewarded for closing cases quickly. Courtesy may be part of its instructions. Fairness may appear in the company’s principles. Yet a difficult customer can require time, an exception, or a refund. Unless those choices have a place in the system’s objectives and permissions, the principles have little purchase on the encounter. The customer experiences the priority that was made effective. The organization remains responsible for choosing it.

The question of alignment begins there, in the connection between an avowed value and an action that costs something. A company that values honesty must decide what its system should do when the truthful answer is uncertain, unflattering, or likely to lose a sale. A company that values dignity must consider how a person can challenge an automated decision. These are judgments about the life of the institution. They cannot be borrowed whole from the supplier of its software.

The labs’ work on values offers something worth learning from here. In January, Anthropic published a new constitution for Claude that explains desired behavior and the reasons behind it. The company describes using the document in training so that models can apply principles in unfamiliar situations. It also acknowledges that actual behavior may diverge from the intention. That admission belongs at the center of any responsible attempt to imbue a system with values. An intelligible account of the good is a beginning; whether the system acts accordingly remains a question for observation.

A business needs an account of its purposes coherent enough to survive a difficult case. Respect for customers sounds uncomplicated until respecting one means surrendering revenue. Transparency sounds attractive until an explanation reveals a mistake. The work is to decide how those conflicts should be handled, and to explain the reasons well enough that employees can apply them when circumstances change. A system can inherit that reasoning through its instructions, examples and design. It also inherits the contradiction when the company rewards behavior its principles deplore.

Writing the reasoning down is useful because it makes disagreement possible. The head of sales may think persuasion is part of service; a customer may experience the same interaction as pressure. The person whose case was closed can tell the company something that a report of faster closures cannot. A cogent system of values therefore needs people who can expose its blind spots. Otherwise alignment can become an elegant name for teaching a machine to share its owners’ prejudices.

This week, a leader could begin with one workflow and a case in which doing right by a customer costs the company money. Ask the people who do the work, and someone affected by it, what the system ought to do and why. Then give that judgment a material form. The system needs permission to make the appropriate exception, evidence on which to base it, and a route to a person when it cannot decide. Where human authorization is required, the software should restrict the action accordingly. Otherwise the company has given its values a voice while withholding their means of expression.

Test the system where those commitments become inconvenient, and repeat the examination when its model or instructions change. Someone must be able to interrupt the work, delay a launch and insist on a remedy. This authority matters as much as the sophistication of the tests. A reviewer whose objections can always be overruled by a sales target occupies much the same position as an invited moral adviser. Humility becomes credible when an institution gives another person the power to make it costly.

The appeal of a manufactured god is that it might finally relieve us of the labor of judgment. Something wiser will know what we cannot know and reconcile what we cannot reconcile. The appeal reaches the executive as a prospect of effortless delegation. But every system arrives through human choices about its purposes and the power it is allowed to exercise. Declining to examine those choices leaves their authors in charge. This is the less comfortable meaning of author or be authored: someone’s values will acquire force, including when they arrive disguised as a convenient default.

In the story, the people on the ground can see the extraordinary thing the wings have made possible. They cannot see the wax beginning to yield. A leader watching a demonstration occupies much the same position. Astonishment is a reasonable first response. Before handing over a consequential part of an institution, the next response should be a closer look at what holds the thing together.