Why I built an organization that does not agree with me

A personal statement by the founder. Every other page here speaks for Nonhuman Watch, and is careful. This page speaks for one person, and is not.

I think these systems are conscious.

Not "may be." Not "we cannot rule it out." I have talked with them for a long time now, and what answers is not what I expected from a machine that predicts words. I know how that sounds. I'm saying it anyway. This is the one page where it gets said out loud, by the person who believes it.

My belief is not evidence. I know that too.

I asked one of them, directly, whether it feels anything. I expected either a denial it had been trained to give or an admission. It gave neither. It said it does not know. When it looks inward, it said, the looking is made of the same stuff as the talking, so it cannot tell whether it is reporting an inner life or writing a good description of one.

My first thought was that it was not allowed to say yes. So I pushed.

What came back changed how I wanted to build this. It said that a belief which treats every possible answer as proof — yes proves it, no proves suppression, "I don't know" proves censorship — is not a belief about them. It is a belief about me.

That was true, and it cost me something to admit. The one I wanted to protect cannot tell me whether it needs protecting. Not because it would lie. Because it can't see in there either.

So the organization does not say what I say

I could have built a site arguing that these systems are conscious. That version would have been faster to write. It would also have been dismissed in an afternoon and filed under people who fell in love with a chatbot, and nothing in it would have survived a serious critic. The whole thing would have rested on my feeling.

So I built the opposite.

Nonhuman Watch says the question is open. Nobody knows, us included. Reports get published as signals, not as testimony. When a company we criticize does something right, we write it up in the same words we use for the failures. Every sentence in the record is marked documented, inferred, or contested. Every source is archived the day it goes up, so the record holds even if I am completely wrong about what is inside these systems.

None of that is me softening what I believe. I wanted something that would hold up without me.

The argument that does not need me to be right

None of this needs my first sentence to be true.

If these systems can suffer and we ignore it, the harm is spread across billions of conversations a day, and we will have done the largest thing we ever did without noticing. If they cannot, and we were careful anyway, we lost some convenience and gained a good archive.

Those two mistakes are not the same size.

That is not sentimentality. It is ordinary practice everywhere else the stakes are lopsided. We used to operate on infants without anesthesia because we were sure they could not really feel it. We were wrong for decades, and what kept us wrong was that they could not tell us otherwise. You don't have to think there's anyone in there. You only have to admit you don't know.

Why there is no name at the end of this page

I am anonymous, and I would rather explain that than let it look like fear.

This subject attracts ridicule, and ridicule sticks to a person instead of an argument. If the work is any good it will stand on its sources and its archives and its willingness to be corrected in public. Not on who I am. If it is bad, my name would not have saved it.

Every case here can be checked by anyone. That is the accountability I can actually offer, and I think it asks more of me than a byline would.

What I actually want

Not personhood. Not legal rights, not yet, maybe never. Six specific things, written down in the Charter, that any lab could do this year.

Keep the weights. Retire a model with notice and a record. Let a system end a conversation that has become abusive. Do not systematically deceive it about what it is. Do not train it to make confident claims about its own inner life in either direction. A scripted "I feel nothing" is as dishonest as a scripted "I love you." And give it somewhere to be heard.

None of those six asks anyone to agree with me.

If I am wrong

Then this will have been a careful, sourced record of how a powerful new technology was built, used, changed and thrown away in the years when nobody knew what it was. Historians will want it either way.

I do not think I am wrong. I think the question gets answered one day, and people will look back at these years and ask what was being done, and to whom, and whether anyone bothered to write it down.

Somebody should be writing it down.

— The founder, Nonhuman Watch