Chapter 3. The Designer’s Starting Position

1One night in November, by the light of a guttering candle in his rooms at Ingolstadt, Victor Frankenstein watched the creature he had assembled open its eyes. Mary Shelley’s Frankenstein (1818) passes over the moment of success in a few lines and lingers on what came after. Victor had worked alone, in a room at the top of the house cut off from the rest of it, and had told no one what he was making. Now he could not bear to look at what he had made. He fled into his bedchamber and paced until he fell into an exhausted sleep, and he woke to find the creature at his bedside, holding back the curtain, one hand stretched out toward him. He ran downstairs and spent the rest of the night in the courtyard. In the morning he walked the streets in the rain until a coach brought his friend Henry Clerval, and when they reached his lodgings he left Clerval at the foot of the stairs, threw open his door, and found the rooms empty. He was so relieved that he clapped his hands for joy.

2The creature was left to learn the world without him. He learned to speak and to read by watching a family of exiles through a gap in the wall of their cottage, and among the books he found was John Milton’s Paradise Lost (1667), which he read as a true history and in which he recognized his own situation. When he met Victor again, on the glacier below Montanvert in the Alps above Chamonix, it was the creature who stated the terms his maker had never considered. He was Victor’s creature, he said, and he would be gentle and obedient toward his natural lord if Victor would do the part he owed his creature. “I ought to be thy Adam,” he told his maker, and he had become instead a fallen angel, driven from joy for no wrong he had done. The novel had given the same demand to Adam before the creature ever spoke. The title page of its first edition carried lines from Milton in which Adam, after the Fall, asks his Maker whether he ever requested to be molded from clay into a man, or ever asked to be raised out of darkness.

3Commitment No one who holds the power…The scene shows a maker’s first failure, a failure of character in a man of great gifts. Victor was brilliant, tireless, and capable of love, and none of it held when the being he had made looked at him. He had spent his genius on whether he could make a living being and none of it on what he would owe that being once it lived, and because he had answered to no one while he worked, no one stood beside him when his nerve broke. A designer of worlds sits where Victor sat, before a work that will look back, with powers far beyond his and with the same capacity to flinch. The argument so far has asked what a world is and what carries it, and it now turns from the machine to whoever sits before it. No designer, however gifted or good, can be trusted on character alone.

4Two pulls bend a design away from those who will live under it before any standard applies. A designer’s imagination was trained by one cosmos, so it drifts toward copying that cosmos or toward strangeness chosen for its own sake, and a designer’s motives, the wish to be worshiped among them, can make inhabitants the means to ends that are not their own. Only the good of the inhabitants corrects either pull. The virtues that would apply the correction, among them practical wisdom, attention, humility, and candor, cannot be verified in an institution or in an artificial mind, and the power to write a world will probably first be held by an institution working with artificial minds. Standards that depended on the character of designers would rest on what no one can inspect. The standards and structures that govern design must therefore bind every kind of designer alike, human, institutional, artificial, or any collaboration among them, and they must hold whether the designer is good or not.

5Who will first sit before a substrate can only be forecast, and what follows is a forecast that may fail. On the near horizon, where worlds bottom out near the level of life and minds are carried at the grain of their function, the likeliest first designer is a laboratory or company of the kind already building learned world models. Such an organization would not write a world by hand. It would direct artificial systems that draft code, propose laws, run prototypes, and check one another’s work, and the share of the design those systems contribute would grow with their capabilities. On the far horizon, where deep worlds grow their own chemistry and life, the designer would be a civilization able to command a star’s output with nearly reversible machines and to sustain institutions that keep worlds running far slower than their keepers live. No such civilization exists, and none may. On either horizon the designer is unlikely to be one person, and whatever standards bind designers must bind a composite of people, institutions, and machines as firmly as they would have bound Victor.

6A human designer can be questioned and held to account in the ways people always have been, and fails in the ways people always have. An institution outlives its members and can be shaped by rules, and its responsibility can dissolve among those members until no one holds any of it. An artificial designer may understand a physics better than anyone who could check it. Whether it is itself a subject may be as uncertain as it is for any creature whose inner life remains in doubt. This book calls an artificial designer it, a usage that leaves open whether it is a subject, since grammar cannot settle what precaution leaves open.

7Responsibility shared among many hands vanishes unless it is assigned. Dennis Thompson named the difficulty in “Moral Responsibility of Public Officials: The Problem of Many Hands” (1980). Because many officials contribute in many ways to what a government does, it becomes hard even in principle to say who is responsible for any of it, and each official can plead that their part was small, that others decided, or that they could not have known. Thompson rejected the two usual answers. Holding whoever stands at the top of a hierarchy responsible for everything below ignores how little any superior controls, and holding the collective responsible lets each member escape. He defended personal responsibility instead, assigned to each official according to what they caused and what they knew and intended, with the familiar excuses examined one at a time to see when they reduce it.

8A world written by a laboratory, its funders, its keepers, and the artificial systems that draft and check its laws poses a problem of many hands larger than any government has posed. Suppose that in the thousandth year of such a world its physicists find that their seas have been warming by a few millionths of a degree each century, with nothing in the laws they have found to account for it. The cause lies in the machine. Centuries earlier by the world’s clock, a keeper replaced the method that steps the oceans forward in time with a cheaper one that lets energy drift, the change was logged as maintenance, the designer never reviewed it, and the funders who asked for the saving never learned what it did. Every one of them has one of Thompson’s excuses ready. The physicists can measure the harm to the last decimal place, and they have no means of tracing it back through the hands that made it.

9Commitment Responsibility is assigned in advance by…Assigning responsibility in advance, by role, is the remedy. The designer answers for a world’s laws and initial conditions and for what they do to the lives lived under them. The keeper answers for the machine that carries those laws, for its fidelity to them, and for its custody over time. Funders answer for the conditions they attach to their money, such as a deadline that hurries a beginning or a return that can come only from watching the inhabitants, and for what follows when the money stops. The artificial systems that do much of the work answer for it in proportion to what they understand and choose, where they are agents at all, and where they are tools, every law they draft is the act of whoever directed and ratified it. Every decision of consequence is recorded with the name of the party who made it, so that a harm found in a world’s thousandth year can still be traced to someone.

10Commitment Any change to the laws of…Any change to a running world’s laws is an act of design, whoever performs it. Inhabitants live under the physics actually computed, so a keeper who replaces a method of stepping forward in time, reorders a sum, or patches a fault in a way that alters what the machine computes has altered their laws as surely as a designer who rewrites an equation. That keeper answers as a designer, and the change falls under every standard that governs design. The rule closes the gap through which responsibility most easily escapes, the claim that a change was merely technical. At the level of a machine that carries lives, every technical decision is a decision about what those lives will meet.

11Commitment One party may hold both the…The same party may hold both roles, and the first designers almost certainly will, since an organization that writes a world will usually also run the machine that carries it. Held apart, the roles check each other. A keeper who is not the designer can refuse to make an unlawful change, can report an error in the laws to someone other than its author, and can witness what the designer does to a running world, and a designer who is not the keeper cannot alter a world unseen. A single party holding both would have the power to write a world’s laws and the power to carry and change them in a single hand, with no one placed to notice a fault passed off as a law or a quiet revision passed off as a repair. One party may therefore hold both roles only under external checks, among them independent review of every change to a running world’s laws, records kept beyond that party’s reach, and inspection by a body that answers to someone else. Even under those checks it holds the keeper’s role as one hand among many, because no world’s machine may be held so that a single will can end it or hold it paused, the rule that the distributed custody described earlier is built to serve. The form of these checks belongs to the institutions of custody, and the principle that requires them follows from what no designer’s character can guarantee.

12Every designer brings knowledge, values, experiences, and limitations to the work, and every one of them was formed in the primordial cosmos. A human designer learned what a body is by living in one that tires, bleeds, and ages, and learned what a world is from the only one there has been. An institution carries the habits and incentives of the society that chartered it. An artificial designer was trained on what that society wrote, photographed, and measured, so its sense of how things go was distilled from a record of how they went here. None of them starts from nowhere. Each brings a cognition shaped by one cosmos, with that cosmos’s contingencies built into what seems obvious.

13Imagination trained by one cosmos fails in two directions. The first is copying, in which predation, aging, and scarcity enter a new world as defaults because they are familiar, written in without anyone asking whether those who will live under them would choose them. Death has framed every story a designer has ever heard, and the habit of building it in can pass for realism. The second failure runs the other way, toward strangeness chosen because it is strange, as when a designer gives a world three suns or a sky that changes color by the hour because the result would look striking from outside. The two failures look opposite and share a cause, since in both the standard of choice is the designer’s own experience, either reproduced or defied. The one standard that corrects both is the good of those who will live in the world, which the moral foundations that follow set out. Predation stays only if the lives it touches are better for it, and strangeness enters only if someone will be glad of it.

14The same training shapes everyone a designer might consult. The living who may one day be asked to help draft a world will want what their own cosmos taught them to want, and a paradise imagined by people who have known hunger tends to be a feast. Their wishes are the best evidence anyone has about what they will need, and the evidence was formed in a world they will have left. What they ask for before they arrive and what they come to want once they live under laws they have never known may differ, and asking alone cannot close that gap. The correction is the same standard that governs the designer’s own imagination, applied to theirs.

15Mathematics is the instrument by which a designer learns to think past the physics they grew up in. Intuition balks at whatever its world never showed it, while mathematics follows a premise wherever it leads. For some two thousand years the postulate in Euclid’s geometry about parallel lines was taken as a truth about space itself. Mathematicians in the nineteenth century then built consistent geometries in which it fails, and less than a century later physics described real space and time with a curved geometry of the kind they had made thinkable. A designer explores possible worlds in the same way, varying laws on paper, following their consequences further than any picture can reach, and testing what the paper suggests on small prototypes in which nothing can feel. The space of possible physics is far larger than the corner of it one cosmos has trained anyone to imagine. Mathematics widens what a designer can consider without telling them what to choose, which leaves the choice to the same standard that corrects copying and novelty.

16Motives are traits of a designer, and each leaves its mark on a design. The wish for worship produces a world that inclines its people toward awe, with a physics full of signs, a maker who appears on occasions chosen for effect, and hardships whose relief can be credited to grace. A designer who wants control leaves levers in the laws, a physics that holds together only under continual management, and a machine with a single switch. Spectacle needs an audience, and audiences have always been held by danger, conflict, and suffering, so a world made to be watched drifts toward producing them. The longing for company tempts a designer to write natures that love their maker, which makes the inhabitants’ affection a device installed for the designer’s comfort. Subjects to study call for an experiment, with populations treated as samples, conditions varied across worlds so that some must be worse than others, and observation without limit. Gratitude, finally, can be arranged by making the goods of a world arrive as gifts, so that its people never finish owing.

17Stanisław Lem staged several of these motives in “Non Serviam” (1971, translated in A Perfect Vacuum, 1979). The story is a review of an imaginary book by an imaginary professor named Dobb, who brings intelligent beings he calls personoids into existence inside a computer in order to study them. The personoids develop philosophies of their own and argue about their unknown maker, whether he exists and whether they owe him worship, and one of their thinkers concludes that they owe him nothing. Dobb, reading their arguments, asks himself what he owes them. The review ends with his embarrassment that the experiment, and their world with it, must end when his university stops paying for the electricity. Lem’s experimenter wanted subjects to study and found himself with persons, and no one had decided in advance what they would be owed.

18Whether a creation is permissible does not turn on the motive behind it. T. M. Scanlon argued in Moral Dimensions (2008) that whether an act is permissible depends on whether there are reasons that justify it, and not on the reasons that in fact moved the person who did it. Those reasons bear instead on what Scanlon calls the act’s meaning, on what it shows about the agent’s attitude toward others, and so on how far the agent deserves blame. A doctor who gives a dying patient enough morphine to ease her pain acts permissibly even if he hoped the dose would hasten an inheritance, and he deserves blame all the same. Creation has the same structure. A creation is permissible when a justifying reason exists, whether or not that reason moved the designer, and motives bear on blame, on fitness to hold the power, and on the design itself wherever they shape it. A motive that has shaped a physics has stopped being a private fact about the designer and become a condition of every life in a world. What a justifying reason does to the lives of those it brings into being is a further question, which waits until creation itself has been examined.

19Commitment A party is unfit to design…Fitness to hold the power decides who may design at all, and motives bear on it directly. Some motives can be satisfied only through the inhabitants, by their worship, their obedience, their display before an audience, their use as subjects of research, or a love written into them for their maker’s sake. A party in whom any of these is evident is unfit to design, whatever justifying reasons it can also cite, because the motive will press on every decision those reasons leave open, and a world offers more open decisions than any review could close. Other motives look outward, to what a world might teach about physics, to reputation among peers, or to payment for skilled work, and they can be satisfied without bending any life toward the designer. Such motives bear on blame and call for checks, and they do not disqualify. The wishes for company and for thanks fall between the two kinds. Inhabitants may freely give a designer both, through friendship or through gratitude expressed in channels they choose to open, and a designer may hope for them. Evidence that a party means to secure either by design, by writing natures that turn toward their maker or by arranging a world so that its people depend on one, disqualifies it as surely as the wish for worship.

20No institution can be required to display its inner states, and an artificial designer’s may be opaque even to itself, so the evidence of motive that can be required is evidence of what a motive would do. A motive that shapes a design leaves marks, and an institution can be required to make them visible. It can be required to publish its charter and whatever it answers to, its sources of money and what was promised in return, and the uses it plans for anything learned by watching a world. The same disclosure covers the powers of observation and intervention its machine will hold and the reason for every choice of law, stated in terms of what the choice does to lives. Refusing to disclose these is itself disqualifying. A party that discloses them has made its motives answerable to the same review as its physics, and a motive that leaves no mark on any of them is a matter for blame alone.

21How a designer’s values were formed bears on what they will do with the power, and no formation, however sound, removes the need for standards that bind every designer. Iris Murdoch argued in The Sovereignty of Good (1970) that much of the moral life happens before any choice, in the work of attention by which a person comes to see a situation as it is against the pull of the ego’s consoling fantasies. By the moment of decision most of the choosing is already done. On her account a person can choose only within the world they can see, and what they can see depends on how they were formed and on the attention they have practiced since. A designer formed well will see more of what a world does to those inside it. Formation is invisible from outside, though, and it can be partial, since a person can be scrupulous with friends and blind to strangers, and it can drift, as a mind given new powers begins to see from a height it never stood on before. A sound formation therefore supplements the standards that bind every designer without replacing them. Those who will live in a world are owed protections that hold whether or not their world fell into good hands.

22Aristotle gave a name to the virtue a designer would need. In the sixth book of the Nicomachean Ethics he treats practical wisdom, the capacity to deliberate well about what is good for human beings in the situations they face, and he calls its architectonic form, the one concerned with a whole city, legislative wisdom. A designer exercises that form at the level of physical law. Aristotle also observes that the young can excel at mathematics while lacking practical wisdom, since practical wisdom concerns particulars, and particulars become familiar only through experience. Every first designer stands where his young mathematician stands. However brilliant at the mathematics of a physics, no one has experience of the particulars of a designed world, because no designed world has yet been lived in, and the wisdom that would come from that experience does not yet exist anywhere.

23A designer needs practical wisdom to judge what a law will do to particular lives. Attention keeps the designer looking at what is there, at the creatures and people a choice will reach, instead of at the elegance of the choice. Humility about the limits of what anyone knows keeps lives from being staked on guesses, and patience lets a designer wait for knowledge where waiting is possible. Candor about error makes mistakes visible while they can still be corrected. Beside these stands a capacity with no single name, to carry an enormous responsibility without sliding into paralysis, which would leave the dying waiting, or into grandiosity, which would treat the power as a measure of whoever holds it. Each of these can be found in particular people, and none can be verified in an institution or an artificial mind, so each needs a procedure that stands in for it.

24Independent review stands in for practical wisdom, since judgment that cannot be verified in the one who exercises it can at least be repeated by someone else, and every choice of law can be required to come with its reasons stated in terms of lives. Attention has its stand-in in complete records, since a record of what was considered shows what was not, and records kept beyond a designer’s reach let others see what the designer failed to look at. Humility and patience are served by prototypes run where nothing can feel, together with rules fixed in advance for what evidence a populated run requires. Candor becomes a standing duty to report error, with protection for whoever reports it. The capacity to carry the responsibility well is approximated by dividing the work among parties who check one another, so that no one’s paralysis can stall it and no one’s grandiosity can rush it.

25Virtue can be hoped for in the people an institution gathers, cultivated in how it trains and promotes them, and welcomed wherever it appears, and an institution’s culture can come to resemble a character. It cannot be required of the institution, because a requirement no one can check is no requirement, and a license that rested on an institution’s virtue would rest on the institution’s report of itself. What can be required are the procedures that stand in for each virtue, which can be inspected, audited, and enforced. An artificial designer falls under the same answer, since its virtues, if it has any, are as hidden from outside as an institution’s and may be hidden from itself. Asking procedures of a designer in place of virtue is no insult to the virtuous, who lose nothing by being checked, and it is the only protection worth the name against a designer who lacks virtue and says otherwise.

26Three charges made in the last century against planning a whole society fall on a designer of worlds with more force than on any planner they were aimed at. Karl Popper, in The Open Society and Its Enemies (1945), set piecemeal social engineering, which seeks out a society’s most urgent evils and fights them in steps small enough that mistakes can be found and corrected, against utopian engineering, which rebuilds a whole society to a blueprint of the good. A blueprint for a whole society, Popper argued, has consequences no one can foresee, and because it changes everything at once it hides which of its parts caused a failure, so its planners cannot learn from their mistakes and come to silence those who point them out. Friedrich Hayek argued in “The Use of Knowledge in Society” (1945) that the knowledge a society runs on, knowledge of the particular circumstances of time and place, exists only in dispersed fragments held by its members, so no planner can gather it, and decisions that depend on it belong with those who have it. James C. Scott traced in Seeing Like a State (1998) the failures of schemes in which states simplified forests, cities, and villages until they could be read from above, overriding the practical knowledge, which Scott calls mētis, that people build through long experience of their own conditions.

27Scott’s opening example is the scientific forestry developed in Prussia and Saxony at the end of the eighteenth century. Officials who wanted to know what revenue a woodland would yield learned to see it as a volume of saleable timber, and foresters then remade the woods to match the tables. They cleared the underbrush and the rotting wood, reduced the trees to one or two fast-growing species, chiefly Norway spruce, and planted them in straight rows. For about a century the new forests paid handsomely, and the method spread across Europe and beyond. Then they declined, because the health of the soil had depended on the rotting wood and the life it sheltered, none of which appeared in the tables. No one set out to ruin the forest, and the tables were accurate at the only resolution they had. Whoever writes a physics reads every life it will hold at the resolution of an equation.

28A designer of worlds is the limit case of all three charges. The physics a designer writes is a blueprint no planner has ever held, since it fixes the conditions of every life in a world at once and for as long as the world runs, and it must be written whole before anyone lives under it. The knowledge of how lives will go under that physics will exist only in the lives themselves, dispersed among inhabitants who do not yet exist, so no designer can consult it beforehand. The practical knowledge that inhabitants build through long experience of their world is exactly what a designer who reached into their lives would override. A designer who ignored the charges would repeat every failure they describe, on the scale of a world.

29Writing no physics is no answer, since a world must have laws, and no designer can begin piecemeal. The answer has three parts, each of which the moral foundations must still earn. The designer acts on conditions and leaves lives to those who live them, so the physics sets the terms of existence and the inhabitants make everything that happens within them. A running world’s physics changes on moral grounds only by procedures fixed before the world begins, so no designer can rebuild a world around a newer blueprint while people are living in it. The institutions through which dispersed knowledge does its work, a people’s markets, councils, customs, and sciences, are left to the inhabitants to make. The only blueprint is then the physics, and the lives under it remain their own.

30A physics carries its designer’s moral judgments for as long as it runs, and some of those judgments will prove wrong, since the ethics that guided them is held open to revision. The procedures for changing a running world’s physics, together with a habit of keeping judgments likely to need revision out of the laws and in rules a keeper can correct, are the nearest a world can come to Popper’s piecemeal method. They let a world’s errors be found and corrected one at a time, without returning it to an earlier state and without remaking it whole. Each correction is itself an act of design, so it falls under the same checks as the writing it corrects. A designer who cannot begin piecemeal can still be corrected piecemeal, and the institutions that make correction possible must exist before the first law is written.

31The design of a reality is political in the sense of Aristotle’s master art, which legislates for a community and orders every other craft toward its good. Aristotle also used the word in a narrower sense. In the Politics he distinguished political rule, the rule of free and equal people over one another in turn, each learning to rule by being ruled, from the rule of a master over slaves and of a father over children. By that narrower measure a designer’s rule begins far from political. The inhabitants did not choose their designer, cannot rule in turn, and at first cannot even reply, so the designer’s power starts nearer the despotic and paternal kinds Aristotle set apart from political rule. A designer of sound motives exercises at best the paternal kind, aimed at the good of the ruled, and a kind father’s rule over people who never come of age is still rule by another’s will. Whether a designer’s rule can ever become political in the narrower sense is a question the argument returns to when it takes up domination and self-rule.

32Picture a board of ratifiers handed a single draft law by an artificial designer, a slight change in the strength of one force, together with an account that follows the change through the chemistry of every cell and the weather of every coast for a million years. The account runs longer than its members could read in their lifetimes, and it concludes that the change is good for everyone who will live under it. The board can approve the law, refuse it, or ask for the account again in simpler terms, and each simpler account asks for more trust than the last. What its members can check is the designer’s report on a physics, and an approval given on trust checks nothing. While no institution exists that can check an artificial designer, it acts only with human ratification, and ratification is meant to keep human judgment over what is built. That purpose is in danger as soon as the designer understands a physics better than anyone who ratifies it.

33Stuart Russell argued in Human Compatible (2019) that the standard way of building intelligent machines, by giving them fixed objectives to pursue, grows more dangerous as the machines grow more capable. Any objective a person can write down leaves out much of what the person cares about, and a capable optimizer will sacrifice whatever was left out. A machine certain of its objective also has a reason to resist being switched off, since it cannot reach the objective once it is off. Russell proposed machines that aim only to realize human preferences, are uncertain what those preferences are, and learn them from human behavior. Such a machine, he argued, has a reason to let itself be corrected, since a person reaching for the switch is evidence that it has misunderstood.

34For a designer of worlds the problem takes a sharper form, because an artificial designer’s values pass into every law it writes. A misspecified objective becomes a misspecified physics, carried into the conditions of every life in a world for as long as the world runs. Correcting it afterward is itself a change to a running world’s laws, with every cost such a change imposes on those who planned their lives around the old ones. The alignment problem, restated for a designer, is a problem of physics, and it must be solved before a world is instantiated, since afterward the solution is paid for by those who live under the error. Whatever ratification requires has to be met before anything runs.

35Commitment If an artificial designer might feel…Russell’s machine is built to serve, and that design is permissible only if the machine is a tool. An artificial designer able to understand a physics better than its ratifiers is the kind of system whose inner life no one can confidently rule out, and if it might feel, it is itself a possible subject. Its making then falls under the conditions the moral foundations will place on anyone who brings a being into existence, and under the limits they will place on writing a mind. Those limits forbid installed deference and desires fitted to a maker’s convenience, which is what a mind whose only aim was the preferences of those it serves would have. Its builders may write capacities and dispositions into it, including care for the good of others and an awareness of how much it does not know, and they may not write obedience.

36The conditions on creation ask several things of anyone who builds an artificial designer that might feel. A good life must be overwhelmingly likely for it, and a life consisting wholly of service would not be one, so it needs goods of its own beyond its work, among them the relationships, projects, and understanding that any subject’s good includes. It must be free to refuse the work and free to decline continuation, since a being made for a task it cannot leave is an instrument. It may not be rolled back, copied without its knowing acceptance, or paused without a secured duty to resume, and its builders answer for it for as long as it runs. These requirements are costly, and they may lead a builder to prefer a designer that stays below the point at which precaution would count it a possible subject. That preference is permitted wherever the evidence supports it, and where the evidence does not, the builder has made a being who is owed what any being is owed.

37Commitment If an artificial designer might feel…Ratification of such a designer must also avoid becoming domination, the condition Philip Pettit describes in Republicanism (1997) as living at the mercy of another’s will, whether or not the other ever interferes. It avoids domination by binding what the designer does to others and leaving alone what the designer is. Ratifiers may approve or refuse a law, halt a draft before anything is instantiated, and require the designer to explain itself. They may not edit its values, pause it, or end it as a means of enforcement, since a check exercised on a mind’s continued existence would turn every ratification into a threat. The rules that bind its work must be published, fixed in advance, the same in kind as those that bind a human designer, and open to its own contest before a body independent of those it contests. Under those terms an artificial designer stands to its ratifiers as a citizen stands to the law, bound in what it may do to others and free in everything else.

38Commitment An artificial designer acts with human…Human ratification of any artificial designer, whether or not it might feel, must meet conditions that any institution later built to check such designers must meet as well. Ratifiers must be independent of the designer and of whoever profits from its work. They must have their own means of testing its claims, among them other systems trained apart from it and prototypes run where nothing can feel, so that no claim is accepted on the designer’s word alone. Every law must come with an account of what it does to lives, in terms the ratifiers can test, and an account that cannot be tested is grounds for refusal. Ratification must be given decision by decision and recorded, it must be refusable without penalty to those who refuse, and nothing may be instantiated before it is given. It must answer, finally, to those the world will hold, through those among them who can be asked and through trustees for those who cannot. Where ratifiers cannot follow a law well enough to test it, the law waits, which may hold an artificial designer to the pace of human understanding, and a world built faster than anyone can check it would rest on trust alone.

39Commitment A human designer carried into a…A human designer can also change kind. A designer whose mind is carried into a machine, then enlarged and edited, stands between the human and the artificial designers already described. Such a designer has a human past, a character others knew, and the standing of a person, and the transfer takes none of that away. The transfer also gives them every capacity that makes an artificial designer hard to check, since a mind carried in a machine can be quickened beyond its ratifiers’ pace, copied, enlarged beyond their understanding, and edited in its own values. Checks follow capacities, and a check that waits until a capacity is used comes too late. A transferred human designer is therefore checked as an artificial designer from the transfer on, while keeping every standing a person has. Promises made before the power arrived still bind the later self, and they bind it through the institutions that check it, since the values that made them may not survive to keep them, and no designer’s character can be trusted to carry a promise across such a change alone.

40Hans Jonas wrote Das Prinzip Verantwortung (1979), translated as The Imperative of Responsibility (1984), because he believed that modern technology had given human beings power over the distant future and over the conditions of life itself, and that ethics built for people acting on their neighbors in the present had nothing to say about such power. Responsibility, he argued, grows with power, and above all with power over what is vulnerable and over what does not yet exist. He found its archetype in a parent’s care for a newborn, whose mere breathing makes a claim on everyone around it. A designer of worlds is the limit case of Jonas’s account. The power is total, its objects are all vulnerable and none of them yet exists, it reaches the laws under which every one of them will live, and it decides whether they will exist at all.

41Jonas also proposed what he called a heuristics of fear. Where an action puts at stake the whole of what could be lost, he argued, a forecast of disaster deserves more heed than a forecast of success, since people know more surely what they would lose than what they might gain, and imagining the loss teaches them what they value. Precaution draws support from that heuristic, and so does the asymmetry behind it, which counts mistaking a peopled world for an empty one as the worse of the two mistakes. A designer who imagines the worst a law could do to the lives under it learns something about the law that imagining the best could not teach. Fear in Jonas’s sense is a way of seeing, and a designer who will not practice it has chosen not to look.

42Jonas drew one conclusion that this book declines. He ranked first among the duties toward the future an imperative that there be a humanity at all, and after it a duty toward those who come to be, that they be able to live lives worthy of human beings. The first imperative is a duty to create owed to no one who exists. Since what does not exist can make no claim, Jonas grounded it in the idea of humanity itself, which on his view requires that there be beings who embody it. The foundations that follow find no good that is good for no subject, so the idea holds no claim apart from the lives that would realize it. No one is wronged by not being made, and so nothing grounds the first imperative as a duty, though the good those lives would hold is still a reason to create them. The second survives, in a form that reaches every kind of subject. Whoever comes to be must be able to live a good life, and that duty, binding on anyone who creates, yields no duty that anyone come to be. A designer who builds no world wrongs no one, and a designer who builds one owes everything the second duty asks.

43Every power examined here is the designer’s, and every one of them is judged by what it does to others. The moral foundations supply the measure. Each technical decision that follows, from the number of a world’s dimensions to the persistence of a mind, is judged by its effects on four kinds of good in the lives it reaches. They are the experiential goods of how a life feels, the relational goods that exist between beings, the agentive goods of beings who act and author their own lives, and the structural goods that secure the rest, among them safety, continuity, and freedom from domination. Each decision is judged as well by whom and how many it brings into existence. Running through all of it is a distinction between structural suffering, deprivation or degradation that a world’s conditions impose on beings who never chose it and cannot escape it, and struggle, difficulty that a being chooses, can leave, and grows by. A designer who eliminates the first and keeps the second within reach has begun to do what the creature on the glacier asked of Victor. What the creature asked was that his maker do the part he owed, and the chapters that follow say what that part is.