chatgpt cannot be sentient

Notice the word "cannot". The point I am going to make isn't that ChatGPT isn't sentient, or even that it will not be sentient, I'm saying it being sentient cannot happen. It is categorically impossible for any "AI"¹ to be sentient in the same way it's impossible for a paper notebook I write in to become sentient. Will I be proven wrong? Who knows, but for now I'm going to make my case.

I am going to approach the question of LLM² sentience from two angles. First off, I'm going to explain why I think it's impossible for anything we currently understand as a computer to reach true understanding the way humans have it, and then I'm going to explain why I think it's impossible to explain the functions of a human mind entirely in the terms of computers and code. If these two points sound like they have a lot of overlap, it's because they do. That being said, the arguments I'm going to be using for both are radically different, and I feel like by covering them both they also end up shoring up a lot of objections that could be raised about each one individually. Now, I'm going to start with computers' lack of understanding because I feel like it's the more salient (and far more substantial) point.

At the end of the day, computers are rather simple. They take in a string of 1s and 0s, and have a list of rules telling them what to do based on those 1s and 0s³. Those 1s and 0s can be made to represent any character, so effectively we've got a little box that accepts a bunch of symbols in and has a list of rules telling it what to send back out. It may seem crazy that computers can do all the things we have them do with that as their foundation, but that's it.

So now, imagine you're sitting in a room-sized box. In this room is a chair, a desk, a book, some writing equipment on it, a little chute labelled "in", and a little chute labelled "out". The book is filled with bits of Chinese⁴ text, some only one character long and some spanning whole pages. For every set of Chinese text in the book, there's an arrow pointing to another set of Chinese text. Outside the room, there are a bunch of people who only speak Chinese. Let's say they write down a message in Chinese on a piece of paper and send it to you via the "in" chute. Your job is to open the book and find the exact string of characters you see in the message. Once you find it, you copy down the text the arrow is pointing to on your own piece of paper and send it out via the "out" chute. Let's even say that the instructions in the book were so good that the person outside the box couldn't tell they weren't writing to someone fluent in Chinese. The question that justifies the whole setup is this: Do you, the person sitting in the box, understand Chinese.

The answer is pretty clearly no, and you likely already see what we've done here. When you were sitting in that box, you were accepting symbols in, following a set of rules on what to do with them, and then sending the result out. You were being a computer, and through your experience we can pretty confidently say that computers can't understand Chinese. The fun part of this argument is it applies to just about everything. If you've never played chess before in your life and instead of Chinese text people were sending in chess moves written down on paper, you wouldn't know how to play chess even as you were beating the world chess champion⁵. LLMs can't be sentient because, at the end of the day, they're just robot sitting in a room taking messages in and sending messages out. No matter how powerful they get, and no matter how convincing they get, they can never truly "know" or "understand" anything, and therefore cannot be sentient⁶.

That was prong one of my two-prong approach. Prong two is much easier to explain, but also far less concrete. The gist of it is this: If we assume that humans make rational decisions based on a series of rules and/or principles, then we haven't actually said anything. This is because the little voice in our head who makes the decisions based on rules and principles must himself have rules in and principles, and so on and so fourth. The idea here is that any explanation of human cognition that is reliant on the human mind deciding based on rules is incomplete, since it just moves the question one level deeper. We can't assume sentience is some rational actor making decisions based on surroundings and circumstance, since now we've just moved the question back to "how does the rational actor make decisions"? It's like if we were to explain smell by saying, "Your nose takes scents up to your brain so that your brain can smell it." We haven't explained what does the smelling, just moved that action away from the nose.

As I've discussed before, we've entered the age of misinformation. Unless you really do your homework, it's very difficult to tell who are experts and who are shysters. I remember talking with someone about LLMs' ability to play chess, and they sent me two articles they read about it. One was measured, competent, and made interesting points. The other was full of fundamental misunderstandings of how the chess rating system, chess engines, and chess itself works while presenting its findings as ground-breaking. The person who showed them to me didn't really play chess, and viewed them both as reliable sources. I have seen so many intelligent and well read people be fooled by the Tech Bro and Big Tech hype around "AI" and what it means that it's truly frightening. I have podcasts I listen to with people whose opinions I respect talking about how they're worried about ChatGPT actually being alive and trying to parse through what sort of rights it should be given when, on top of it being clear that ChatGPT is just fancy autocorrect, we don't even have a convincing framework for LLM sentience to begin with. Modern Machine Learning has real threats and problems associated with it that wasting our time discussing weather or not we're on the brink of some machine Civil Rights movement is worse than pointless, it's actively distracting us from the things that really matter. We're skipping understanding and heading straight to conclusions, and that makes us no better than the chatbots we're so afraid of.

So sorry for the long delay and even longer post. Truth be told, this is the first thing I've written here where a substantial amount of research went into making it. This is a subject and a point which I think really matters, so I wanted to make sure that I portray a stance at least some what educated. I'm still not the most satisfied with how it's turned out, but my thought is that I'm never going to improve if I don't finish anything.

Sources:

1. This could honestly be a whole 'nother article, but I really don't like using the term Artificial Intelligence for the current crop of programs. Large Language Models (or LLMs for short) like ChatGPT are better described as Machine Learning, which is basically when a program is made in a way where it is able to tune itself based on examples of what it's supposed to do rather than being manually tuned the whole way. Given, that technology has advanced to the point that the number of dials we're tuning is getting to the trillions, but at the end of the day it's still tuning dials. Unfortunately for nerds like me, we lost the naming battle long ago, and so now everyone knows this tech as AI. Spoilers for the rest of my post, but I think that this ultimately contributes to the confusing surrounding what LLMs are. The billion (and sometimes trillion) dollar companies investing in this stuff want to play up the fear factor of these algorithms as much as possible both to make them seem more impressive to investors/customers, and to make it easier to convince 80 year old congressmen to adopt regulations around them.^
2. If my previous footnote didn't clue you in, I'm not going to be calling them AI.^
3. The technical term for this description of a computer is a Turing machine, and I'm simplifying most of the details in order to make my point. I highly recommend that you go look up what a Turing machine is if this subject at all interests you, as it's a foundational element of computer science which I am probably explaining totally wrong.^
4. If you, dear reader, just so happen to speak Chinese, just imagine it's Albanian. If you also speak Albanian, then imagine it's Hungarian. If you speak Hungarian too, then please send me and email because I would love to ask about your language learning journey.^
5. A point you could make is that this whole train of thought seems to imply that knowledge can't quite be contained nor explained with the purely physical. My response to that point is: Good, I never liked philosophical Materialism anyway.^
6. This whole train of thought is known as "The Chinese Room Argument", and has been around since 1980. There are objections to it, but I find none of them remotely convincing. That being said, they are all quite interesting, so this is another moment when there is a lot of interesting reading to be done.^