Session 5: Foundational Aspects of General Intelligence and AI
2:45 pm – 3:30 pm GMT
Keynote Talk
Vincent Mueller (Humboldt Professor and Alan Turing Institute) – AI, Ethics and Singularities
Transcript: –
Vincent Müller
Now for something completely different as Monty Python used to say. So I, what I want to do today is a really small thing. It’s a sort of combination of two different research activities that I have, which I thought were quite separate. And I now tend to think that maybe there’s a connection between the two. And I thought maybe other people would like to comment on whether I there is really a connection or whether I’m just fantasising there about that connection. So part one, is, I’ll give some really elementary introductions, because this is not a philosophy seminar. So explain a little bit what ethics is, in my view, which is different from the standard view. And then the main body of the talk is it is a reproduction of a paper that I wrote recently on, on the singularity. And on the importance of goals in that discussion, and then I’m coming back at the end of that discussion, to what it is to have goals and why that might be relevant for the understanding of cognition and ethics and how commission ethics are actually much more connected than people usually think they are. So the biographical background to this is that I have been working a little bit on philosophy of AI and ethics of AI for quite some time, about 15 years or so. And I’ve recently written this little survey article on it. Well, recently, last year. And at the time, the only survey on AI ethics there was. And one of the things I singled out there as interesting questions was the singularity at the end and the conceptual repercussions of all these discussions. So I, I think there is actually an interesting development in philosophy, which is ongoing, sort of similar to what I think Northoff is doing the million euro philosophy, I think we’re now doing some kind of machine philosophy. So we’re going back to the philosophy from what we’ve understood about machines. I’ll explain that in a minute. So very basic, this distinction, which I think is important for our discussion here is interpreting, descriptive and normative. So, we can describe how the world is, or we can say how the world should be. This is sometimes misunderstood. So for example, things like data is likely to change the world, or people have norms, and values, these are descriptive remarks. And nothing follows from that. But if you say something is more important than something else, where you should have done something like that, then these are normative remarks. And an ethics is about the normative remarks. Ethics traditionally, was just the word was just describing what people do the ethos, the customs, what one does, like in the slogan, you know, when in Rome do as the Romans do or something like that. The same applies to, to morals. So the way I see ethics, it is the systematic reflection on the normative. So the basic question of ethics, in good content tradition, is what should I do? But I’m taking that in a slightly broader sense than Kant. So Kant thought that certain things are part of the ethics which are not all the things which have this form, what should I do are parts of ethics. So for example, the question in the chess game there on the right, what’s the best move to win is not part of ethics is ethics involves some kind of immediate duty and impair a categorical imperative as equals that as opposed to a hypothetical one, which is, I should do something if I’m trying to achieve something else, for example, winning in the chess game. So So for my for my discussion, I will just assume ethics discusses that we think that’s, that’s normative. Now, now comes the second part, little rehash of some discussions on the singularity. I think the basic argument for existential risk from Ai goes something like this. There are two assumptions, namely, a super intelligent AI is a realistic prospect, it would be out of human control. I call that the singularity claim. Because it says that there is this point in time where this occurs, and it’s called the singularity and be any level level of intelligence can go with any goals. This is known as the orthogonality. Thesis. This is explicitly defended by several people. And it follows from that a super intelligent AI poses an existential risk for humanity.
It’s very easy to turn that into a formally valid argument if you want to do that you need be, because it could be otherwise that a super intelligent AI might just be super ethical, and do whatever is super ethical, which may, of course include killing all the humans. But we hope that that’s not the case. Now, in this discussion here, we think that we’ve discovered that there’s a trick in this argumentation. And the trick is that the two premises are using the word intelligence in a different way. So the argument from super intelligence to existential risks, and this the singularity claim uses some kind of notion of general intelligence, which is usually in this discussion, unfortunately, completely undefined, it sort of means the kind of thing that humans have, but more of that general cognitive abilities, with some kind of intention. In the orthogonality thesis, on the other hand, people have discussed what they mean by intelligence, and there, they talk about instrumental intelligence. So they’re talking about the kinds of things that people discuss in decision theory. That means that you have to have some kind of goal or utility function, specifically right there, then you see, what’s the best way to achieve that? Okay. And it seems to us that these are two quite different beasts. So in other words, instrumental intelligence and general intelligence are not the same thing. Here’s the explanation of that. So here is the instrumental intelligence, instrumental intelligence. That’s something that Nick Bostrom, who was one of the defenders, of course, of this kind of thesis has said, explained several times his capacity for instrumental reasoning, means and reasoning. And so, okay, so that’s the kind of thing that you could call problem solving, you have a problem, you find a way to solve it. General Intelligence involves all of those things can do the problem solving. It also involves problem defining. So the characteristic difference, it seems to us between normal instrumental intelligence and general intelligence is that general intelligence involves something that you might call metacognition. So you don’t just pursue some goals should also reflect on the goals whether these goals are relevant. So a chess computer, for example, will just try to win the chess game. A human will not necessarily have that. So if you play against your your seven year old niece, then you might think, well, it might not a good idea to actually crush her in the game, if provided that she’s not some kind of chess genius. But you should try to make sure that you’re having a good time and that she enjoys the game, and that maybe she wants to continue learning it. So maybe it’s a good idea to give her a chance to capture some of your pieces or something like that. So that is you’re widening the frame of your reflection from just winning the chess game, to seeing what’s, what matters here and what chords are actually important. Because you could do Biden them further, right? And say, you know, we should not even be playing chess, now we should take care of the invaders to Ukraine or something that’s completely different. So general intelligence, in other words, is a broader notion than instrumental intelligence. So we conclude from that, that we can’t have it both ways. That the singularity claim in the orthogonality thesis, these two premises of the argument might both be true, but only if we take them to be about two different forms of intelligence because orthogonality for general intelligence is extremely implausible. And the singularity claim for instrumental intelligence is also implausible. So I, we say that we don’t really see a way to interpret intelligence in such a way that we would we would get both
Sheri Markose
can you just go with the orthogonality? Why’s that? Can’t run counter to you know, singularity claims
Vincent Müller
so maybe I’ll just, I’ll just go back and see, see what we have the statements okay. So, the orthogonality thesis is B here, okay. So B means I can have any goal whatsoever. And it seems to us that that, that is not right if you’re taking general intelligence, because General Intelligence will allow us to reflect on goals. So it’s not the case that any goal is as good as any other. But we can have arguments on what is what, which goals are better than other goals. So whether, for example, winning by better performance is better than winning by cheating. And that, it seems to indicate that orthogonality is not true for general intelligence. So yeah, yeah. Good, maybe I think I’ll skip this bit, because that’s where we’re just confusing people. So, so the, the last bit here, Section five, is to try and wrap this together into some kind of understanding of, of, of normative metacognition. So it seems to me that intelligence therefore involves a number of different skills. One of them is instrumental intelligence. That is the rational choice to to maximise expected utility, subjective guilty, then there is the wider notion of input of intelligence that will involve embodied systems it will be involved in embedded and extended systems. So we’re not just using our bodies, but we’re also using other technical means, that are part of the explanation why humans are so successful as a species. So the artefacts creator, and the culture that we’ve created is a significant part of that. So it seems to us in other words, that, that the problem, the difference between whoops. Between general intelligence can, and instrumental intelligence that general intelligence involves metacognition is a quote from Doug Hofstadter. From that, he says, it’s an inherent property of intelligence that you can jump out, jump out of the task, which is performing and survey what it is done, it’s always looking for and often finding patterns. So this is what we call, widening the frame. And if we do that, then we can actually have goals, which are really our own goals, as opposed to just things which have been sort of defined for us. So it seems to us in other words, that the ability to reflect on goals is an elementary part of general cognition. In other words, the ability to do ethics is an elementary part of cognition. So that’s the opposite of the orthogonality thesis, right? The automatic thesis says that we have intelligence is one thing, and we have ethics is a completely different thing. And here, the suggestion is that the ability to reflect on goals to reflect in a normative way, is part of standard cognition and needs to be understood as that. And and that is the reason why, it seems to me that we now should should understand ethics in a slightly different way. And we should also learn something for ethics from the reflection on AI. So that’s why I’m suggesting that there is some kind of a digital philosophy or some kind of philosophy that Jen that generates insights from the reflection on AI. So this is a sort of programmatic beginning, kind of, I suppose, my little intervention,
Sheri Markose
right? So so Vincent, the singularities themselves, I think you may need to explain a bit more, I’m in the sort of, you know, nearly scenarios that people like curse will Bostrom and sign in. And this idea that suddenly an AI imbued with some adaptive capacity to find better and better solutions can suddenly I’m sorry, let’s, let’s take that, the AI that, you know, the paperclip AI, it’s just smells down on the metals. And then when when it senses because its goal is to produce paperclips it single mindedly pursues that uses all of its adaptive capabilities to achieve that. So it even senses that you’re going to be hostile to it and pull its plug and then it’ll sort of turn on you as well. That’s the sort of idea. So that’s a sort of singularity, you know, one of the similarities and the other one of course is going to be you’re going to go become more clever than us in many respects. Yes, but then we be used to that. I mean, no, you know,
Vincent Müller
I think I think this the singularity, so So that that’s a bit weird. Obviously, I didn’t explain in the slides now. So, the idea of the singularity is always that we are reaching a point at which the machines are more intelligent than humans, or at least as intelligent as humans. And at that point, the further development of AI will be up to the machines. So the machines are then also as good as humans on developing AI. Therefore, the machines will develop the next generation AI and they will be smarter, and so on. And this is often understood to be a possibility for an explosion soar and a very quick growth in a short period of time. Because the assumption is that some of that progress can then be achieved in a mechanised way, which will be way faster than what we’re doing now. And Nick’s Nick Bostrom, his idea is that this is dangerous record swans idea is that this will be beautiful. So the question is, which one is right about this. And, of course, is the whole concept, a sensible concept. So Nick says, this will be very dangerous, because these things will be out of our control, I think that’s clear that they would be out of our control, if there would be super intelligent beings in the world that would not be controlled by less intelligent beings. Our own experience shows that that’s exactly what we are doing on Earth right now with the other animals. So then he says, okay, therefore, these beings will end up doing whatever they think is the right thing to do. And that could be whatever. So it could be making paper clips, if it so happens that this was the kind of thing that was started. That was the utility function that was defined for it. So in other words, he assumes that even for a super intelligent being any goal is good, as good as any other goal. That yes, being will not sit down, why the hell should I make paper clips? What’s the point in doing that? Right, exactly. So so he says that you could be super intelligent, and yet unable to reflect on goals. Okay, so this is already dubious, because it seems hard to see how you could really understand everything about the world, including, for example, that you have certain intentions, which might run against my wanting to make lots of paper clips, and so I’d better get rid of you as an obstacle and so on. And so that would involve quite a lot of understanding of humanity, intentions, etc, etc. But, more importantly, the idea is that the paperclip Maximizer would actually be able to step out of the frame of the paperclip making and say, Hmm, in order to make paper clips, apparently, there’s another goal, which is important. For example, I need to maximise access to energy. And these humans are in my weight in my way, so I’ll get rid of them. So the idea is that you can do all of that. But you can’t, at the same time, reflect on goals. So that it seems to me that is a problematic notion. So the reason why,
Okay, the counter to the counter to it is really quite simple. My suggestion is that you’re making two assumptions in order to get that argument flying. And in the one assumption, you’re using an instrumental notion of intelligence, that’s the orthogonality thesis. And then the other assumption, namely that super intelligent machines would take over, you’re using a general notion of intelligence, and you can’t have it both ways.
Sheri Markose
So then, can we go to bed without having nightmares? Blood is basically saying it’s never gonna happen. These these scenarios are
Vincent Müller
not really I’m afraid. So there’s, there’s two reasons for not going to bed too quietly yet. One is, it could be that a purely stupid instrumental goal pursuing AI is bad enough. It will not be as bad as as most drum imagines it will not take over the world, etc. But it might do a lot of very stupid and detrimental things. And that is actually happening to some extent already. And of course, the other point is, there is a significant probability that I’m wrong. And, of course, if I’m wrong wrong, then then extension is from Ai really is a problem. And, and so there is a probability that that really is a problem. So that will not be the case there will only be the case if I were right. And we knew that I was right. But we’re not in that position. Definitely.
Sheri Markose
Any I mean, I don’t know. Willem, do you want to add? I mean, this thing about you must have heard of Capitani have and Moby Dick. Right? He said, all my methods are fine, you know, the rational, super rational, anything but by objective. So completely crazy bonkers, right? Chasing the white whale, and so on. So economists of Applegate have been up against this for a long time. I mean, you did I did. You’re a function in the tool, you instrumental rationality, but then we have this idea of, you know, when you talk about kind of arrows, non dictatorship. In other words, if each one of us are to be autonomous, and you know, goal oriented, then the system as a whole has to be open ended, right? The you know, the end neutral rules, the state that can talk about that means you can’t have both things that are purposive. You can’t have individuals that are purposive. And also the system itself, at the state level, or some sort of collective level, also having objectives except that I can’t do anything that would then violate your equal, right? To the same, you know, so but AI doesn’t work. We live in that world, it’s not constrained by the fact that, you know, what I call the universality principle. In other words, I can only do things that you can also do so then that becomes the counter there that I can’t perceive my objective without thinking about you as well. Your equal right to the same. So why would?
Vincent Müller
The difficulty is that you seem to assume that it is possible to have a super intelligent AI, which would have aims, but would be unable to reflect on those. That’s what I’m saying you is is not plausible.
Sheri Markose
All right. Anybody else? Adam? You remember we discussed what self improving AI? What was that? Not? No? We can the only problems we can think outside. You could be self improving. But but but the purpose remains the same, the objective remains the same. What point can you say maybe the objective is the wrong thing.
Adam Shroff
I’m here, I’ve seen different counters the orthogonality. thesis. And I tend to agree with what was just presented. Like in a lot of the scenarios, like most people talk about something like a treacherous turn, where there’s very high degrees of social intelligence, such that the system is capable of deceiving people. Well, this seems like you’ve now unfolded in a kind of like, now you have dependencies on things like the type of like, reflexive capabilities and meta awareness if you’re actually capable of fooling others in those ways. Further, I’d say like, I think there tends to be a kind of potential confusion about human intelligence, where like, we might try to reduce it to the algorithm, or to like, maybe even the architecture as a whole. But part of what that gets us as a gets us like, ability to basically do natural language and couple and basically inherit the Cultural Endowment. And so you so now you have like a meta brain of all of us interacting with the world, figuring things out over like, long periods of time. And that’s what makes us so smart. Like, you could take like von Neumann and like, but if he was feral, he would just be like, maybe a clever primate, he might not even be a clever primate, he might be like, a fairly dumb primate, if you did, not in culture, you know. And so now you have a whole nother set of path dependencies that are infusing value with intelligence. And so if something like natural language is part of the Bootstrap, which seems very likely that you would need this for something to pull off something like a singularity, well, now, you know, we didn’t like write out language and like the predicate calculus, you know, it’s a good old fashioned natural language, like even if you take like, the most like technical dry things, like from like, an AI company. It’s still just like, you know, laden with like metaphor and implicature. And all these like pragmatics that like, you need common ground, you need like some idea of like, like right now, like, I’m not enunciating very well, I’m kind of fumbling but you kind of get what I’m saying, because you have some sense for what I mean, because you’re all kind of like me, and so you have common ground and common sense. You’re able to disambiguate this like word salad I’m throwing at you a little bit, hopefully.
Sheri Markose
So Adam and Vincent so are we saying that because AI doesn’t have our, our framework for empathy, in other words, you know, I make your exam self making self self referential use of my own embodied self, right? That’s how I make you out reuse the same code, we believe in the mirror neuron system so on, whereas AI doesn’t have that capability. So another AI? Or how does it have a sense of self and the other or it has no sense of empathy? That’s that, you know, Can we can we read that? You know, can we breed that into AI? Because without that, how is it? How is it new any sense of have any sense of ethics? Because the other is not a projection of self. So you know, what do you care? What is the I care about others?
Vincent Müller
I thought Adam was making so I’m continuing along the lines, would you say what is it that I understood from that, from what you call the word salad, which wasn’t a salad, but anyway, is so I thought your point was that the Cultural Endowment, so to speak that, that, that explains our intelligence, which I totally agree with, is, or at least is a huge part of the expenditure for the intelligence of the human species would be not readily available to the AI, for use, so to speak, because it wouldn’t really understand what is going on there. So it’s sort of like a bit Kingstonian point that if a lion could speak, I wouldn’t be able to understand it. Yeah, the idea,
Adam Shroff
there would paraphrase it perfectly and like, and so then, like, the question would be like, even if we’re doing some something, okay, so if we’re talking about a system capable of advanced deception of human like agents, I find it implausible that it would be capable of that, without being able to be less lying, like and more human hypothetical, but like, I find that implausible, but even like something like engineering, like, like knowledge synthesis, from just like, you know, we want something or like, read the Internet. Well, it has to understand now, it wasn’t mean to understand it means to be able to unpack, like some kind of like, causal world model based on when it’s reading that it can then like, bring together all these facts and like, test out the different hypotheses, probably, it’s not just like, the next version of GPT, you know, whatever. So like, that seems to, and I’d argue, like, even like the lion, like, a, a souped up lion, still wouldn’t be able to do even basic engineering, because even that is an intensely social activity, where we’re like, inheriting know how via, like, a subset of the cultural dominance, but we’re still just, you know, neck deep and needing some kinds of empathy. I mean, there’s still like, a bullet to be been like psychopaths exist. Yeah. So it’s not like, it’s, maybe it’s not pure orthogonality. But it’s not like pure colinearity. Either. Like, you’re it’s like, there’s still like, some separability it’s like, you know, don’t piss them off, raise them? Well, however, you grow the system. But, you know, I would claim like, you know, just as a matter of practicality likely to make AI’s do the things we need to do, we’re going to need to give them some basic ability to somehow access basically empathy to some degree so that they can understand others enough to get to actually work with natural language.
Vincent Müller
Right. I think this, you know, it’s a there is a way in which I can interpret what you’ve said, as supporting what to say. So I will try to use it, though. So. So my suggestion was that we would need some kind of broader notion of cognition, which includes normative thoughts, right? And so So what you’re saying is that empathy with other beings is crucial. And it seems to me that these two things aren’t very far apart, because empathy presumably means that you then say, Oh, I know I can now feel that hitting the thumb is worse than not hitting the thumb of that other being there, because it will hurt just like my thumb would hurt if I would hit it with a hammer. So so close that norm that normative thought Why is it inherently bad if somebody feels pain? It might be awful, but something else right but but it is inherently something something negative. And so that that generates the normativity. And there and the real goals, it seems to me so you can reflect on that and say, Well, yes. But in this case, because I know that you’re really sick, I need to give you this injection and therefore, you know, you’ll have some pain. Forget election, but it’s really good for you. So it’s okay in the long run. But I would make a soul would make normative considerations about this, but they’re based on on this empathy. Thought. So. So in a sense, my my point is, is similar to that, that I say we need, in order to have an understanding or some kind of super intelligence that really works and is smart enough to generate something like world dominance, it would have to have access to something like that. That’s the
Sheri Markose
best. So the chances of having a sort of super intelligence, but that is up to us right now, because we haven’t yet got to the stage where any of the AI is anywhere near near to what Nick Bostrom Marcos will talks about? Surely it isn’t it is for us to then somehow stop that software, because it’s meant to be that software then sort of bootstraps itself in super intelligence. And somehow, we build the software right now for the AI that breed that would have these capabilities, right? Isn’t it own up to us to build a sort of soft AI that would have empathy. But, you know, that’s the whole
Vincent Müller
thing might be up to us. It might be up to us. But it might be that it happens anyway. So so when I was still at FHI, Nick, and I wrote a paper on when we, when you would expect super intelligence to occur. And so we we asked lots of people, what they thought when the probability would go over 50%. And the median response was 2014. At that time, it was like getting six, seven years ago or six years or something. And I think the assumption there is, is that you will, you might have developments of exponential growth, which could come at you pretty quickly. And even those of us who weren’t very familiar with exponential growth. Two years ago with Pratik, we’ve now all become quite familiar with these kind of developments in the pandemic, so in and it’s, I think the expectation is not that we will just sort of Potro away with programming and then and then boom, we get that thing but that we would produce systems that have the ability to do something by themselves more and more efficiently. And then, and then they would go out and learn more and more things. And, and so Nick thing is that machine learning is, is sort of speak on the right path for producing that well, which is then going to be a problem.
Sheri Markose
And recently, I was reading something that Daniel Dennett said to David Chalmers, but you know, you’re wasting your time thinking about superintelligence never gonna happen, right? I didn’t know that.
Vincent Müller
But that’s, that’s an interesting sort of you if you want this an interesting case. So these two superheroes in contemporary philosophy disagree on.
Sheri Markose
Yes. It’s a real problem. That is a real project, you should see what David Tennant said the job was was after him to comment on a paper where, you know, Thomas was direct talk with super intelligent, read that just just to sort of get up to speed York for your talk. I’m surprised that you’re wasting your time trying to talk about superinten It’s never gonna happen right.
Vincent Müller
Now, both of these people have actually written about, about AI and its similarity to human cognition and so on. And so, so it’s, it’s interesting seems to me that both of these people have influenced contemporary philosophy quite significantly. I’ve actually worked on AI.
