crimbotSome serious moral dialogue about robots recently. Eric Schwitzgebel put forward the idea that we might have special duties in respect of robots, on the model of the duties a parent owes to children, an idea embodied in a story he wrote with Scott Bakker. He followed up with two arguments for robot rights; first, the claim that there is no relevant difference between humans and AIs, second, a Bostromic argument that we could all be sims, and if we are, then again, we’re not different from AIs.

Scott has followed up with a characteristically subtle and bleak case for the idea that we’ll be unable to cope with the whole issue anyway. Our cognitive capacities, designed for shallow information environments, are not even up to understanding ourselves properly; the advent of a whole host of new styles of cognition will radically overwhelm them. It might well be that the revelation of how threadbare our own cognition really is will be a kind of poison pill for philosophy (a well-deserved one on this account, I suppose).

I think it’s a slight mistake to suppose that morality confers a special grade of duty in respect of children. It’s more that parents want to favour their children, and our moral codes are constructed to accommodate that. It’s true society allocates responsibility for children to their parents, but that’s essentially a pragmatic matter rather than a directly moral one. In wartime Britain the state was happy to make random strangers responsible for evacuees, while those who put the interests of society above their own offspring, like Brutus (the original one, not the Caesar stabber) have sometimes been celebrated for it.

What I want to do though, is take up the challenge of showing why robots are indeed relevantly different to human beings, and not moral agents. I’m addressing only one kind of robot, the kind whose mind is provided by the running of a program on a digital computer (I know, John Searle would be turning in his grave if he wasn’t still alive, but bear with me). I will offer two related points, and the first is that such robots suffer grave problems over identity. They don’t really have personal identity, and without that they can’t be moral agents.

Suppose Crimbot 1 has done a bad thing; we power him down, download his current state, wipe the memory in his original head, and upload him into a fresh robot body of identical design.

“Oops, I confess!” he says. Do we hold him responsible; do we punish him? Surely the transfer to a new body makes no difference? It must be the program state that carries the responsibility; we surely wouldn’t punish the body that committed the crime. It’s now running the Saintbot program, which never did anything wrong.

But then neither did the copy of Crimbot 1 software which is now running in a different body – because it’s a copy, not the original. We could upload as many copies of that as we wanted; would they all deserve punishment for something only one robot actually did?

Maybe we would fall back on the idea that for moral responsibility it has to be the same copy in the same body? By downloading and wiping we destroyed the person who was guilty and merely created an innocent copy? Crimbot 1 in the new body smirks at that idea.

Suppose we had uploaded the copy back into the same body? Crimbot 1 is now identical, program and body, the same as if we had merely switched him off for a minute. Does the brief interval when his data registers had different values make such a moral difference? What if he downloaded himself to an internal store, so that those values were always kept within the original body? What if he does that routinely every three seconds? Does that mean he is no longer responsible for anything, (unless we catch him really quickly) while a version that doesn’t do the regular transfer of values can be punished?

We could have Crimbot 2 and Crimbot 3; 2 downloads himself to internal data storage every second and the immediately uploads himself again. 3 merely pauses every second for the length of time that operation takes. Their behaviour is identical, the reasons for it are identical; how can we say that 2 is innocent while 3 is guilty?

But then, as the second point, surely none of them is guilty of anything? Whatever may be true of human beings, we know for sure that Crimbot 1 had no choice over what to do; his behaviour was absolutely determined by the program. If we copy him into another body, and set him uip wioth the same circumstances, he’ll do the same things. We might as well punish him in advance; all copies of the Crimbot program deserve punishment because the only thing that prevented them from committing the crime would be circumstances.

Now, we might accept all that and suggest that the same problems apply to human beings. If you downloaded and uploaded us, you could create the same issues; if we knew enough about ourselves our behaviour would be fully predictable too!

The difference is that in Crimbot the distinction between program and body is clear because he is an artefact, and he has been designed to work in certain ways. We were not designed, and we do not come in the form of a neat layer of software which can be peeled off the hardware. The human brain is unbelievably detailed, and no part of it is irrelevant. The position of a single molecule in a neuron, or even in the supporting astrocytes, may make the difference between firing and not firing, and one neuron firing can be decisive in our behaviour. Whereas Crimbot’s behaviour comes from a limited set of carefully designed functional properties, ours comes from the minute specifics of who we are. Crimbot embodies an abstraction, he’s actually designed to conform as closely as possible to design and program specs; we’re unresolvably particular and specific.

Couldn’t that, or something like that, be the relevant difference?

14 Comments

  1. 1. Vicente says:

    They don’t mean robots, they are just referring to human beings, two more steps along that line and you’ll find yourself as a character in “minority report”.

    Ortega y Gasset said “I am I and my circumstances”, in your robotic example it would be “they are just programmes in their circumstances”. The point is not what AI means for AI it is what they want AI entail for humans. This process does not lead to robots with rights but rather to humans without them. Well, a few will keep them. Be careful.

  2. 2. Sci says:

    For what it’s worth, I’d just say the program is equivalent to the “soul” and thus whatever body the program is in can be justifiable punished so long as no other programs are stored there. That said, that equivalence between programs and souls hopefully shows how dualistic computationalism is.

    It seems to me that programs would have to shed light on our consciousness/memory/etc before we should confer upon them any rights. And as much as computationalists might bleat otherwise, demanding acknowledgement for “solving” the Hard Problem, this seems like a fair way to judge AI as conscious entities.

    “This process does not lead to robots with rights but rather to humans without them. Well, a few will keep them. Be careful.”

    As silly as I think computationalism is, I don’t think this is necessarily what computationalists want. It’s more that they want to advance a political cause and find minds being computers guts the idea of souls, or they want to believe their studies into computer science have conferred special insight into reality.

    Now corporations and governments might like eliminativism because it frees them from any moral restraint, but it seems to me one doesn’t have to be a computationalist to be an eliminativist. Sadly, given how easy it is to see computationalism as false whereas eliminativists like Bakker make a more serious case.

  3. 3. Sergio Graziosi says:

    Peter, I think that both you and Scott are missing a crucial trick, any your post makes it particularly evident.
    Our moral intuitions, like all the fast and largely unconscious judgements (I mean the judgements that we make without knowing how we make them, we are conscious of the result – the judgement itself – but are only able to produce post-hoc explanations) are, as Scott keeps saying, very effective heuristics that have been shaped through our evolutionary history.
    “Bad people deserve punishment” is probably the paradigmatic example of such an heuristic. Why do they deserve punishment? Because in human societies, punishment and ostracism are a very effective system to keep antisocial behaviour at bay. This is because we can’t directly rewire the faulty wetware between our ears, we can only work indirectly, via rewards and punishments.

    If we could easily switch off all of our undesirable impulses, would he have this obsession with retribution? I hope not, but I can’t be sure. To me, your example makes the locality (in the problem-space) of our almost ubiquitous “crime and punishment” disposition very apparent: it is our default attitude because it is the most effective general-purpose strategy in our normal conditions. In the case of robots equipped with digital processors and a clearly defined software layer, we won’t (or shouldn’t) ever think they deserve punishment because there are better ways of solving the problems they may cause: change the software. The urge to punish crimbot 1, 2 or 3 may be our knee-jerk reaction, but it should be easy to see why it isn’t the most effective strategy.

    From this starting point, I end up following Scott’s argument, but only up to a certain point. Truly general purpose artificial intelligence will expose the shallowness of some of our deeply held beliefs. One could take the catastrophic view and expect all of our thought-systems crumble as a consequence, but it doesn’t need to be like this. Call me a (provisional) optimist, but I see the intellectual challenge that the kind of scenarios you discuss pose as an opportunity to finally understand how limited our own intelligence is, and how it is hopelessly tied to deeply felt intuitions that can systematically misfire when they apply to unprecedented situations (situations that weren’t significant when our intuitions evolved). The problems introduced by AIs don’t need to be (destructive) pollution, they can be illuminating as well. Which one it will be has to depend on us, right? [Note: I do think strong, general purpose AI is an existential threat, because you know, natural selection: whatever is more rapacious and able to grab more resources will do exactly that. So all you need is a GAI to be a bit smarter than us, and slightly tilted towards trying to expand its disposable resources, or trying to reproduce, and we’ll suddenly face a formidable competitor, if not a flock of them. Given how blind we are to our own idiosyncratic limitations, I find the prospect very scary.]

    Conclusion: the trick I think you’ve missed is about realising that yes, the significant difference between humans and our hypothetical crimbots is the design neatness of the latter. However, this exposes the limited scope of our moral instincts, it shows how our deeply felt moral drives are finely tuned to our own peculiar and provisional circumstances. In itself it does not expose a profound difference between artificial and biological agents. I mean: suppose we could cure all sex offenders with one single intervention. Say we’ll eventually make a pill that transforms the sexual drive of a paedophile so that suddenly children are not sexually attractive, adults are, and the sexual impulse also never becomes overwhelming. Should the availability of this remedy change our attitude towards punishment? My take is that it should, and would, eventually; but our “deeply felt moral instincts” would make this change agonisingly slow, and thus allow more people to suffer as a consequence.

    The trick I think that Scott missed is the positive view. Being placed in front of a case that clearly exposes our intellectual limitations can be useful to learn how to overcome them. It may tear apart a lot of wishful philosophising, but whether it will only produce ruins has to be up to us. To be fair, Scott does mention this possibility, but doesn’t end up asking: how do we make sure we’ll get the positive outcome? I think it is a missed opportunity, because he is very well placed to help us getting a head-start.

  4. 4. Scott Bakker says:

    Sergio: “The trick I think that Scott missed is the positive view. Being placed in front of a case that clearly exposes our intellectual limitations can be useful to learn how to overcome them. It may tear apart a lot of wishful philosophising, but whether it will only produce ruins has to be up to us. To be fair, Scott does mention this possibility, but doesn’t end up asking: how do we make sure we’ll get the positive outcome?”

    I get upbraided for this on my own board too often for me not to think its true that I spin the doom and gloom. Some think I *am* describing positive outcomes! I think for me the big part is that I haven’t found any imaginative way to think around these issues that doesn’t lapse into quietism.

    Add the rank indifference of nature to the fact that it all seems to be happening so fast. ‘Bleak Theory’ Paul Ennis calls it.

  5. 5. Scott Bakker says:

    “The difference is that in Crimbot the distinction between program and body is clear because he is an artefact, and he has been designed to work in certain ways. We were not designed, and we do not come in the form of a neat layer of software which can be peeled off the hardware. The human brain is unbelievably detailed, and no part of it is irrelevant. The position of a single molecule in a neuron, or even in the supporting astrocytes, may make the difference between firing and not firing, and one neuron firing can be decisive in our behaviour. Whereas Crimbot’s behaviour comes from a limited set of carefully designed functional properties, ours comes from the minute specifics of who we are. Crimbot embodies an abstraction, he’s actually designed to conform as closely as possible to design and program specs; we’re unresolvably particular and specific.

    Couldn’t that, or something like that, be the relevant difference?”

    I think it’s fascinating the way posing the question like this lets us see how easily the analogy swings both ways. Is the ‘relevant difference’ the one that prevents humans from being treated like machines or is it the one that prevents machines from being treated like persons?

    But to be clear, are you also saying that if we made an AI as noisy as a human, restricted it to the functional capacities of a human, then we would be obligated to treat them as persons?

  6. 6. Mark O'Brien says:

    Hi Peter,

    I don’t agree with you that personal identity is the key difference. As you pretty much pointed out, almost the same points would apply to people if we had a transporter/duplicator setup.

    As Sergio explains, the thing to ask is why you want to punish the person or the robot in the first place. If it is because retribution appeals to you as a good in its own right, then I find your justification for punishment unconvincing. For me, the main justification for punishment is on consequentialist grounds. We punish people because they do not want to be punished, and this motivates them to avoid transgressing. The same logic would have to apply to robots. If the robots have desires we can frustrate to influence their behaviour, then all else being equal it is reasonable to do so. Whether we punish a “copy” or not depends on how this will affect the robot’s decision making at the time it is planning its actions, and this in turn depends on how its desires are programmed. If it cares what we do to its copies, then we can punish its copies. If it doesn’t care about anything that happens after it shuts down, then there is no point in punishing it after it shuts down.

    But, unless we have evolved the AI through some automated process so that it is a black box, it would seem that the better strategy would be, as Sergio suggests, simply to fix its programming.

    In the black box case, we may not know how the robot thinks well enough to have answers to all your questions out of the gate, but trial and error should let us figure out what kinds of punishment work eventually. If punishment has no effect, then of course we shouldn’t punish at all but find other means to control behaviour.

  7. 7. Sergio Graziosi says:

    Thanks Scott, I’ve been reading your views with increasing interest in the past 18-24 months, and I really think you are onto something. However, the doom and gloom side really puzzles me, specifically because I have my own views, which seem to be entirely compatible with yours (and very close to some of Metzinger’s), but lack a dark side: where you see disruptive consequences I see hope for an explanation, at last!

    I speculate that most of our differences come from how we approached the field, I started in neuroscience and developed an interest in the philosophical side only relatively recently, I would guess that your trajectory was exactly the opposite. In cognitive neuroscience, the fact that our own brains have to work around huge intrinsic limitations is the starting point (was my starting point, at least), making the general question “how come that our brains seem to do so much so well, given how limited they must be?”. In this context, explanations produce new possibilities. Philosophers tend to start from “hey, our minds are amazingly powerful, look how smart we are” (OK, I’m a little unfair here), so being shown the intrinsic limitations can feel like crushing cherished illusions.
    I may be wrong, it’s just speculation (and not a particularly informed one). Anyway, I do think and hope the two of us may develop some interesting discussions, don’t be surprised if I’ll try to make it happen one day or the other ;-).

  8. 8. Eric Schwitzgebel says:

    Terrific post, Peter! I’m thinking about turning my thoughts on these topics into a full-length essay, which is hard to do on this issue without some intelligent pushback like this!

    First: I’ve written a follow-up post at The Splintered Mind, which addresses the general idea (common to your and Scott’s posts) that AI cases are likely to cause serious trouble for the intuitions and principles behind our moral systems: http://schwitzsplinters.blogspot.com/2015/02/how-robots-and-monsters-might-break.html

    Second, you write: “in Crimbot the distinction between program and body is clear because he is an artefact, and he has been designed to work in certain ways. We were not designed, and we do not come in the form of a neat layer of software which can be peeled off the hardware. The human brain is unbelievably detailed, and no part of it is irrelevant.” Now it *might* turn out that if AI becomes possible, robots are designed such that what you say is true of human beings is also true of them — perhaps through evolving their minds up through evolutionary algorithms without explicit programming and in which hardware and software are linked. They might still, nonetheless, be duplicable, creating problems for attributions of identity. And, as other commenters have suggested, human “transporter” or “upload” cases might raise similar problems.

    So my inclination is to think that technology would have to play out in one specific way for a bright line to be maintained.

    Third: If the robots are android form, humanlike in behavior, and not in fact regularly wiped and duplicated (even if wipe and duplication are possible in principle) — like Data from Star Trek — then I suspect our intuitions will tend to support attributing rights and responsibilities. But if we grant rights in that case, then maybe a garden path of no-relevant-difference arguments can take us back into chaos in cases with more wiping and duplication.

  9. 9. Jochen says:

    I think people are taking the copyability of minds too easily for granted. Sure, the reasoning seems sound—present-day computers run with software that can easily be copied, so why should a future robo-mind not similarly be copyable?

    The problem with this reasoning is that, compared to anything approaching a mind, our present-day software is probably laughably simple, and thus, we can know the state of a present-day computer exactly (though even there are some limitations, given certain limitations in our access to a machine—broadly, it’s in general not possible to ascertain the state of a machine by providing it with input and observing its outputs, even if the machine is completely known to the experimenter; Moore termed this ‘computational complementarity’). But for a future robo-mind, I think there are good reasons that we won’t, in principle, be able to know its state exactly; but in this case, one can prove that it can’t be copied.

    Consider for simplicity some finite state machine with n states. Represent those states as the basis vectors of an n-dimensional vector space, say s1 = (1,0,0,…,0), s2 = (0,1,0,…,0), to sn = (0,0,0,…,1). Every computational step is then the transition from one state to another, mediated by an appropriate transition matrix.

    Now, copying the state si of this automaton means that you have a ‘blank’ system of at least n dimensions on which you want to copy that state, and a transition matrix C that acts on the combined stat of original and blank system, such that the blank ends up in the state of the original afterwards. Or, symbolically:

    C*(si x b) = si x si

    Here, b is the blank system’s initial state, x is the Kronecker product of two vectors, producing an n^2-dimensional vector in the joint state space of the original and the copy from the states of the original and the copy, * is ordinary matrix multiplication, and C is the n^2 * n^2-dimensional matrix that produces the transition from the two systems being in the stat [si, blank] to them being in the state [si,si], i.e. copying the original system’s state to the blank.

    As an example, if we look at two systems that can each be in either of two states, say A = (1,0) and B = (0,1), and we want to copy an arbitrary state of S1 to the system S2 which always starts out in the state A (for definiteness), we can do it the following way. First, let’s look at the case in which the original system starts out in state A. Then, the joint state is A x A = (1,0) x (1,0) = (1,0,0,0). In this case, we need not do anything to the joint state, since S2 is already in the same state as S1.

    The other relevant case is S1 being in state B, with S2 being in state A, meaning the joint state is B x A = (0,1) x (1,0) = (0,0,1,0). (The other two possible initial states, A x B = (0,1,0,0) and B x B = (0,0,0,1) won’t play a role, because S2 needs to start out in the blank state A.) In order to copy the state of S1 to S2, we thus need to effect the transition (0,0,1,0) –> (0,0,0,1). It’s not hard to see that the matrix ((1,0,0,0),(0,1,0,0),(0,0,0,1),(0,0,1,0)) can accomplish both, i.e. leave A x A invariant and transition B x A to B x B; thus, we can always clone the state of the system, and since any computer always is in one of these states, we can copy any computer program, and hence, a computer mind.

    But this conclusion is too quick. The above reasoning works only if we are able to fully determine the machine’s state. Once we admit the existence of intrinsic uncertainty about this state—either due to the machine being intrinsically probabilistic, depending on irreducible randomness sampled from the environment, imprecise measurement or computational complementarity—there is no general-purpose cloning process anymore. To see this, it suffices to realize that if we don’t know the precise states as above, we need to admit probability distributions over them—formally, convex combinations of the basis vectors, as in p*A + (1-p)*B = (p, 1-p), which corresponds to an automaton that is with probability p in state A, and with probability 1-p in state B.

    One could view this as a mind with certain dispositions towards producing certain behaviours—say, with probability p, I will eat a ham sandwhich, with probability 1-p, I’ll have tuna instead. There exists now, in general, no physical process that will, from having access only to me and a blank reference system, reproduce this disposition (and all possible other dispositions) in the reference system.

    The proof is easy: the transition matrices between states are linear, and they are completely general, being able to mediate all possible state transitions. But let’s look at the transformation we would need to reproduce the state (p,1-p) on S2: from the initial state (p,1-p) x (1,0)=(p,1-p,0,0), we want to produce the state (p,1-p) x (p,1-p) = (p^2,p(1-p),p(1-p),(1-p)^2)—which can obviously not be a linear transformation, since the resulting state is quadratic in p. This is the content of what is generally called the ‘classical no-cloning theorem’, classical because it was only discovered after its quantum analogue, which is thus generally known simply as the ‘no-cloning theorem’. (It can be proved in the setting of symplectic mechanics on phase space, which makes its physical applicability more obvious, but the above abstract discussion likewise is fully general.)

    What it means is, ultimately, the following: if you have a box, in which there is with probability p a red ball, and with probability 1-p a green one, then there exists no physical process that leaves you with that box and a second one in which again there is a red ball with probabiliy p and a green one with probability 1-p (nor even some isomorphic system, say a coin which yields heads with probability p, and tails with probability p).

    If we thus allow an irreducibly random element to minds (which it seems we have to—even macroscopic probabilities have their origin often at the quantum level, for instance, eight collisions between balls on a billard table suffice to make the uncertainty in their trajectories be dominated by quantum effects), then there can’t be any faithful copying of minds. Note that this does not mean you can’t transfer a mind to a different substrate, say from a brain to a silicon circuit—it merely means that the original must be destroyed, if such a process is to occur. ((p,1-p,0,0)–>(0,0,p,1-p) = (1,0) x (p,1-p) is a linear transformation.) But exact copies are forbidden.

    Of course, this invites the question of how exact a copy needs to be to still count as ‘me’. It doesn’t seem implausible that one could create a copy that’s as close to me now as I now am to me yesterday, and I certainly take myself to still be the same person in all relevant aspects. So given a set of imperfect copies, which one is me? I think this opens up an interesting avenue for investigating a ‘closest continuer’-kind of theory of personal identity, where one quantifies the difference between two minds regarding some appropriate distance measure on the space of probability distributions, such as the Kullback-Leibler divergence or relative entropy. But still, it seems to me that this discussion has the potential to nip some of the problems of cloning/copying in the bud (while introducing interesting new ones, no doubt).

  10. 10. Peter says:

    Thanks, folks. I think I may have muddied the waters a little by talking of punishment; it dramatises the issue of responsibility in a rhetorically useful way, but it brings in additional issues.

    Yes, I’d have to grant that a ‘robot’ which was a moral agent is conceivable if it has the right human-style properties.

    I don’t think a program-running-on-a-computer-robot could be like that, though.

    Digital computers are all like Turing machines, and Turing machines, as we know, are not real. They operate in discrete states, they have unlimited tape, and so on. They’re really a science fictiony way of talking about mathematical functions, whose existence is sort of Platonic. Human beings are not like that at all; we’re completely real, not at all abstract.

    I actually think Gerald Edelman was on to something here with his analogy with the immune system. You can’t tell in advance which bit of neurology is going to support a functional thought in the same way that you can’t tell which bit of chemistry on the surface of an antibody is going to lock with some disease-bearing invader.

    I might be drifting off the topic a bit.

  11. 11. ihtio says:

    The distinction between hardware and software, at least in the context of the current discussion, is only conceptual. Once a hardware runs some software, there are only instructions, bits, electrical impulses. Software is only a high level description of what a machine does. When software is deleted from the machine, the machine is essentially lobotomized.

    In the case of human beings, thoughts and experiences, goals and motives are also high level descriptions of what is going on between ensembles of neurons, electrical impulses, neurotransmitters. When a capacity for thinking in a certain way is removed, the person in question is essentially lobotomized.

    In both cases, the agent responsible for the act has been drastically modified. In fact, the agent may have been modified so much that the question of moral responsibility becomes mute, as there no longer exists an agent that performed the act (that is, we wouldn’t say that the before-agent and after-agent are the same).

    As technology progresses computer we use are becoming less and less similar to what standard theory of Turing machine says about universal machines. It is likely that future machines will work in a way that is more similar to how we work, that is like highly complex, interdependent dynamical systems (thinking machines), and not as simple Turing machines (simulation machines, as I call them). If that would be the case, the notion of deleting a program would be identical to deleting a portion of the brain, resulting in major disruption and modification of the machine.

  12. 12. Strong AI, utilitarianism and our limited minds | Writing my own user manual says:

    […] havoc in our moral systems, and leave the moral landscape littered with ruins. While I was reading Hankins’ post, I caught myself thinking “Oh my, is Peter constructing a pro-utilitarian case? This would be […]

  13. 13. Sergio Graziosi says:

    All: Thank you!
    Comment (pingback) #12 above links to my own attempt to show how one might start building a positive case: I’ve tried to back up my wishful suggestions with something more concrete. Hope it does make some sense.

  14. 14. Arnold Trehub says:

    Ng: “There’s a big difference between intelligence and sentience. Our software is becoming more intelligent, but that does not imply it is about to become sentient.”

    Sentience/consciousness is the key. See “Where Am I? Redux” and “A foundation for the scientific study of consciousness” on my Research Gate page, here:

    https://www.researchgate.net/profile/Arnold_Trehub

Leave a Reply