Roko's Basilisk

Necronic

Staff member
Is anyone familiar with this? It was referenced in the latest XKCD alt-text, so I looked it up. The premise of it is complicated and I won't do it proper service, so I'll just link the Slate article discussing it:

http://www.slate.com/articles/techn...errifying_thought_experiment_of_all_time.html

Before you read that let me, as a warning, point out that apparently even reading it or thinking about it may cause it to come true, according to the logic of the thought experiment (haha I warned you after the link so you probably read it anyways). Anyways, for those of you who are familiar with it or bravely chose to risk eternal damnation by reading the article I linked, what are your thoughts?

Personally, it just strikes me as pure and utter bullshit and the kind of mental games that will make your palms hairy. The kind of garbage metaphysics that people come up with to sound smart at parties. There are three fundamental flaws with the argument, and they are important:

1) The universe is NOT deterministic. This is incredibly important, and its not surprising to me that a group of intelligencia missed this NEARLY A CENTURY OLD FACT. We have known the universe is not deterministic since quantum mechanics was discovered, yet, for some reason, people often ignore this. Maybe its because the logic of a non-deterministic world is really hard to grasp, like a zen koan, or maybe it's because at larger scales the universe effectively IS deterministic/Newtonian.

However, this entire experiment is focused around the creation of an AI, a form of computer. Maybe it would be traditions circuitry, maybe it would be organic, but either way I can guarantee that it would be effected by quantum uncertainty.

This would mean that it would be impossible for it to do these accurate predictions of events that are necessary for the thought experiment to work.

2) Even if we ignored the deterministic flaw, for the basilisk to effectively create a perfect it would have to model the universe, which would require it to model itself, which would have to contain a model of the universe, which would have to have a model of itself. Ad infinitum.

I'm hoping you can see the problem with this. This is impossible. And its impossible because the prediction and the model has to be perfect to work, it requires omniscience. Any thought experiment that involves omniscience is going to always hit problems, because omniscience is very similar to the infinite, it simply doesn't exist in reality.

To me this whole thing just boils down to the old George Carlin joke where he asks the preacher "Can God make a boulder so large he couldn't lift it?"

3) Time travel. The AI requires time travel. Maybe this could really happen. Maybe not. But if your proposal requires time travel in its core it better be a new Terminator movie. Maybe you could remake terminator 3. Another version of it doesn't require time travel but requires an even more out there idea that we would be indivisible from a future simulation of ourselves, and that we might be living in the simulation right now.

Anyways. Anyone else familiar with this?
 

GasBandit

Staff member
Yudkowski needs to spend less time trying to reinvent Shrodinger's wheel and more time finishing HPMOR, IMO.
 
Would just take box B. Either I get a million dollars OR I become the first person to prove the infallible supercomputer wrong. Both outcomes sound like a "win" to me.
 

GasBandit

Staff member
Would just take box B. Either I get a million dollars OR I become the first person to prove the infallible supercomputer wrong. Both outcomes sound like a "win" to me.
That's just the setup, not the actual basilisk question. Box B gets you nothing or eternal torment.
 
Maybe I'm just slow today, but if the machine is always right, then no new information has actually been imparted to you and you should just make the choice you were always going to make.

If the machine is God of this reality, then either I have no free will (my choice doesn't matter), or God wants me to use my free will (the knowledge that God is always right doesn't matter).

If the machine is not always right, then it's no different from the kids who beat you up and take your lunch money when you do something they don't like. In which case, the Basilisk can take a number and line up, because we've already got a ton of bastards at various levels of power who can do that.

While the idea of a computer that created a universe trying to map the universe (Planetary did a nice job with that one) is a cool one, it makes for better science fiction than something to actually allow to affect my decisions.

'Cause really, when the Singularity God comes, he's going to be more pissed at Fox for cancelling Firefly than he will be at me for scoffing at a metaphysical thought experiment.[DOUBLEPOST=1416602859,1416602818][/DOUBLEPOST]
Would just take box B. Either I get a million dollars OR I become the first person to prove the infallible supercomputer wrong. Both outcomes sound like a "win" to me.
Oooh, nice, totally forgot about the fame and talk show angle. :D
 
I have other thinks to answer when I'm not tipsy from a couple beers on top of being exhausted but:
1) The universe is NOT deterministic. This is incredibly important, and its not surprising to me that a group of intelligencia missed this NEARLY A CENTURY OLD FACT. We have known the universe is not deterministic since quantum mechanics was discovered, yet, for some reason, people often ignore this. Maybe its because the logic of a non-deterministic world is really hard to grasp, like a zen koan, or maybe it's because at larger scales the universe effectively IS deterministic/Newtonian.

However, this entire experiment is focused around the creation of an AI, a form of computer. Maybe it would be traditions circuitry, maybe it would be organic, but either way I can guarantee that it would be effected by quantum uncertainty.
It is not obvious that an AI would be affected by quantum uncertainty in its decisions. The macroscopic world behaves mostly independent of quantum uncertainty. Computers do not do weird things because they work with electrons and our brains probably don't either (as appealing as it is to think that our decisions are influenced by quantum randomness to get a -false- appearance of free will, it is not clear that they are).
 
Okay if I leave this for tomorrow I will probably not post it so here it is:

I think Slate's account of what Roko's basilisk is isn't correct. Although I was already familiar with the idea, I had never read about it including infinite nested simulations (btw Necronic, I don't think it does in this version either). AFAIK (I may be wrong, since I haven't read the original post) the original Basilisk said that if you did the wrong thing (not supporting the AI), then the AI may torture you... or a simulation of you, in the future. So that if you know that this will happen already, you will act accordingly, or something. Saying that we are all simulations already or whatever doesn't make sense because, as in Newcomb's paradox, the AI only needs to simulate me up until the decision (in the Newcomb case, it will not simulate you enjoying the money, and here it won't simulate you suffering in hell just to know what you'd do).

Of course this original idea doesn't make a lot of sense either: if I have already decided, why should the AI punish me in the future? It will not change anything about its past.

BTW, EY (Eliezer Yudkowsky) now says that he erased all original posts because they were messing with people, not because he believed it (I'm not sure I buy it but whatever). If you look at the thread for the last xkcd comic in the kcd forums he also has some explanations as to why Roko's basilisk doesn't work (One is that, if you say that future torture won't change your decision, the AI has no reason to torture you).
 
2) Even if we ignored the deterministic flaw, for the basilisk to effectively create a perfect it would have to model the universe, which would require it to model itself, which would have to contain a model of the universe, which would have to have a model of itself. Ad infinitum.
Well, the quantum randomness is a good reason to have many simulations running in parallel to have a good sampling of everything that could happen. But the simulation doesn't need to have many levels because it is not simulating the AI, it's simulating you: in the Newcomb problem, it only matters what YOU do, not what's in the box, so the AI doesn't need to simulate its own decision (therefore needing to simulate you and itself again, and again)

3) Time travel. The AI requires time travel. Maybe this could really happen. Maybe not. But if your proposal requires time travel in its core it better be a new Terminator movie. Maybe you could remake terminator 3. Another version of it doesn't require time travel but requires an even more out there idea that we would be indivisible from a future simulation of ourselves, and that we might be living in the simulation right now.
[\QUOTE]
I have never heard that it requires time travel... It just requires that you act according to what some future AI may or may not do which doesn't make a lot of sense either (that or I'm not understanding it, but I browse LessWrong from time to time and I've never read that it requires time travel... Of course, it was censored for a long time so...)

Ah, and by the way, as far as I know most LessWrong users don't buy into the basilisk bulshit. Would be surprised if they did, actually was quite surprised when I thought that EY did
[DOUBLEPOST=1416606371,1416606205][/DOUBLEPOST]
He did the same thing I did. Quit after half the article.

Because really, the whole thing seems like the usual philosophical quandries, just with 'god' replaced by 'evil computer' so that atheists can have a crisis of faith.
It's like a negative version of Pascal's wager (I should believe in God regardless of wheter it exists, to make sure I don't miss on heaven ~ I should work towards IA regardless of wheter that particular one comes to be, to make sure I don't go to real or simulated hell)
 
This sounds like the premise of I Have No Mouth and Yet I Must Scream, except with a time travel angle. Lemme read this...

Okay...

If the computer is malevolent and can punish you for not helping it come to be, then why help it? Because you might suffer for it eternally? It already knew what choice you'd make... to offer you the choice instead of simply following it's own prediction is to admit that it is incapable of actually predicting anything and to punish you for not making the choice it wants (and knew you wouldn't make) is simply sadism. This is Divine Command Theory all over again: Does God order good because it is good or is it good because of God? Ether the AI can't do the very thing it says it can because it has to rely on you to make a choice or it's choice is completely arbitrary to your concerns and thus unworthy of cooperation.

So I choose neither box. The worst that can happen to me is the AI punishes me for a choice it already knew I would make (which is something I couldn't avoid) or I reveal that it can't do it's vaunted feat and it loses it's power over man.
 
Not choosing a box is choosing the second box.
No, it's forcing the machine to pick a box itself, which it could have done without my involvement if it's prediction abilities were sufficient for it to KNOW (rightly or wrongly) which I'd pick. To involve me in any way is to admit that my free will is a factor in the outcome and that it could not do what it claims (or has chosen not to for some inexplicable reason) or that it's simply toying with me and demanding I choose for it's own amusement or out of ritual. As a rational agent, the only outcome in which I preserve my free will is to not abide by the demands made of me.

 

Necronic

Staff member
My understanding of it is that the computer would be able to predict my choice based on a perfect model of who I was made by creating a perfect simulation of me. First, this would require a perfect simulation of me, physically, from which you could, based on determinism, predict my actions. This is the first flaw of determinism in the model, as I would argue there is good reason to believe that nueral pathways are heavily affected by quantum uncertainty. However, remember, nature vs nurture. A perfect simulation of me would have to include the things that affected me. To do that you would have to make a perfect simulation of my surroundings. To do that you would have to model the earth, then the solar system, then the universe itself, and you would have to do that for every living human throughout time that has to be hit by this blackmail. The scale of computation here is where the secondary affects of quantum uncertainty could come into play. Quantum uncertainty has almost no chance to ever affect computational stuff. However, the scale of the model and the necessary calculations would quite likely hit a tipping point here and have said uncertainty disrupt the model.

I'm not sure where I got that it would need to model itself, that doesn't seem to be necessary

Anyways, the key flaw here is that the model requires perfect models and simulations, which is where you get to these insane scopes where you have to model whether the gravitational disruption caused by some far flung galaxy on my chest hairs will affect whether or not I decide to buy cherios or captain crunch. But the requirement, as far as I can tell, is that it IS required for the model to be perfect, because if it is not then it cannot accurately predict what I will do, which invalidates the decision crisis. Statistical averages of weaker models will yield "close-enough" results, but that is not really acceptable in this premise.[DOUBLEPOST=1416609349,1416609284][/DOUBLEPOST]Oh yeah and I did spend some more time reading up on this and LessWrong doesn't seem to buy it at all anymore, but...I have yet to hear any refutation of the initial comments made by EY, which sort of make him sound like a crazy person.
 

GasBandit

Staff member
The boxes are the rewards, not the choice. If you decide to help create the Basilisk AI, you get both boxes, if you decide not to, you get box B.
 

Necronic

Staff member
why would I want both boxes, one has torture in it.[DOUBLEPOST=1416609498,1416609449][/DOUBLEPOST]Which, btw, was the not the hit unboxing video on youtube I expected it to be
 
Is anyone familiar with this? It was referenced in the latest XKCD alt-text, so I looked it up. The premise of it is complicated and I won't do it proper service, so I'll just link the Slate article discussing it:

http://www.slate.com/articles/techn...errifying_thought_experiment_of_all_time.html

Before you read that let me, as a warning, point out that apparently even reading it or thinking about it may cause it to come true, according to the logic of the thought experiment (haha I warned you after the link so you probably read it anyways). Anyways, for those of you who are familiar with it or bravely chose to risk eternal damnation by reading the article I linked, what are your thoughts?

Personally, it just strikes me as pure and utter bullshit and the kind of mental games that will make your palms hairy. The kind of garbage metaphysics that people come up with to sound smart at parties. There are three fundamental flaws with the argument, and they are important:

1) The universe is NOT deterministic. This is incredibly important, and its not surprising to me that a group of intelligencia missed this NEARLY A CENTURY OLD FACT. We have known the universe is not deterministic since quantum mechanics was discovered, yet, for some reason, people often ignore this. Maybe its because the logic of a non-deterministic world is really hard to grasp, like a zen koan, or maybe it's because at larger scales the universe effectively IS deterministic/Newtonian.

However, this entire experiment is focused around the creation of an AI, a form of computer. Maybe it would be traditions circuitry, maybe it would be organic, but either way I can guarantee that it would be effected by quantum uncertainty.

This would mean that it would be impossible for it to do these accurate predictions of events that are necessary for the thought experiment to work.

2) Even if we ignored the deterministic flaw, for the basilisk to effectively create a perfect it would have to model the universe, which would require it to model itself, which would have to contain a model of the universe, which would have to have a model of itself. Ad infinitum.

I'm hoping you can see the problem with this. This is impossible. And its impossible because the prediction and the model has to be perfect to work, it requires omniscience. Any thought experiment that involves omniscience is going to always hit problems, because omniscience is very similar to the infinite, it simply doesn't exist in reality.

To me this whole thing just boils down to the old George Carlin joke where he asks the preacher "Can God make a boulder so large he couldn't lift it?"

3) Time travel. The AI requires time travel. Maybe this could really happen. Maybe not. But if your proposal requires time travel in its core it better be a new Terminator movie. Maybe you could remake terminator 3. Another version of it doesn't require time travel but requires an even more out there idea that we would be indivisible from a future simulation of ourselves, and that we might be living in the simulation right now.

Anyways. Anyone else familiar with this?
Fwiw: I ardently disagree with the interpretation of quantum mechanics as meaning the universe is probabilistic in nature. The reason we cannot develop deterministic models at the quantum level is because we cannot measure things that small without affecting them (thus Heisenberg and his principle). This doesn't mean the universe must roll probabilities to function but that we have no other way to model them but with probabilities.

And to points two and three: all models, including simulations, are reductive. Perhaps we are in one and cannot know it. They can still be highly predictive even with such reductions. Perfect simulation is unnecessary. But we need only be in a single one, because that is all we can be aware of. Thus no time travel is necessary. Only simulation of the past. Ooo...Matrixy.

All that said I am unconcerned.
 
My understanding of it is that the computer would be able to predict my choice based on a perfect model of who I was made by creating a perfect simulation of me. First, this would require a perfect simulation of me, physically, from which you could, based on determinism, predict my actions. This is the first flaw of determinism in the model, as I would argue there is good reason to believe that nueral pathways are heavily affected by quantum uncertainty. However, remember, nature vs nurture. A perfect simulation of me would have to include the things that affected me. To do that you would have to make a perfect simulation of my surroundings. To do that you would have to model the earth, then the solar system, then the universe itself, and you would have to do that for every living human throughout time that has to be hit by this blackmail. The scale of computation here is where the secondary affects of quantum uncertainty could come into play. Quantum uncertainty has almost no chance to ever affect computational stuff. However, the scale of the model and the necessary calculations would quite likely hit a tipping point here and have said uncertainty disrupt the model.

I'm not sure where I got that it would need to model itself, that doesn't seem to be necessary

Anyways, the key flaw here is that the model requires perfect models and simulations, which is where you get to these insane scopes where you have to model whether the gravitational disruption caused by some far flung galaxy on my chest hairs will affect whether or not I decide to buy cherios or captain crunch. But the requirement, as far as I can tell, is that it IS required for the model to be perfect, because if it is not then it cannot accurately predict what I will do, which invalidates the decision crisis. Statistical averages of weaker models will yield "close-enough" results, but that is not really acceptable in this premise.[DOUBLEPOST=1416609349,1416609284][/DOUBLEPOST]Oh yeah and I did spend some more time reading up on this and LessWrong doesn't seem to buy it at all anymore, but...I have yet to hear any refutation of the initial comments made by EY, which sort of make him sound like a crazy person.
There is not very good reason at all to believe quantum effects neural change! It might be nice to believe such a thing but there is virtually no data to support it.
 

Necronic

Staff member
What about unsheathed neurons cross chatter? That deals with electron level interactions doesn't it? Fwiw I agree that quantum uncertainty could just be a measurement problem, but from what I can tell determinism vs uncertainty is just a coin toss, and I'll stick with the one that doesn't leave me with a free will paradox.

As for the reductive models, in the real world this is absolutely true. But this is a philosophical logic trap which means that it has unrealistic requirements of perfection. A perfect model can not be reduced. At least, that's my guess seeing as no one has ever made a perfect model for anything, ever. Remember, this isn't a probabilistically predictive model, this needs to be a completely perfect model.
 
What about unsheathed neurons cross chatter? That deals with electron level interactions doesn't it? Fwiw I agree that quantum uncertainty could just be a measurement problem, but from what I can tell determinism vs uncertainty is just a coin toss, and I'll stick with the one that doesn't leave me with a free will paradox.

As for the reductive models, in the real world this is absolutely true. But this is a philosophical logic trap which means that it has unrealistic requirements of perfection. A perfect model can not be reduced. At least, that's my guess seeing as no one has ever made a perfect model for anything, ever. Remember, this isn't a probabilistically predictive model, this needs to be a completely perfect model.
That is a lot less definitive of statement than the OP.

As for neurons, quantum fluctuations are orders of magnitude too weak to affect the electrochemical nature of neurons, including any cross chatter that may occur.
 
The boxes are the rewards, not the choice. If you decide to help create the Basilisk AI, you get both boxes, if you decide not to, you get box B.
The reward is analogous to the choice. If it has already decided that I would not help it, it is going to punish me no matter what. In which case there is no point in humoring the AI. If it has decided that I WILL help it, then it should have already given me the reward. That it hasn't means it's ether unsure of if I will help it (in which case why bother? It's a flawed AI) or has chosen not to predict my choice for some unknown reason (which makes it completely arbitrary).

There are really only three options: ether it can't do what it's supposed, it's not doing it on purpose, or it can and I'm getting torment no matter what. The only way I can preserve my own dignity is to simply refuse to play and FORCE IT to make a choice by itself.

Ironically, this isn't a logic problem. It's an ethics problem.
 

Necronic

Staff member
Man, I should not be trying to read that article while driving[DOUBLEPOST=1416613237,1416613053][/DOUBLEPOST]What's really cool about this paper is that it's not even about electrons, it's talking about uncertainty in molecular collisions, which is far more easily understandable. I think it's based on chaos driven tunneling...I think. I vaguely remember studying that in school.
 

Necronic

Staff member
Oh yeah, also forgot about the best argument I know against the measurement argument for uncertainty. Electron orbitals, or more generally the particle in a box. If you solve the square of the wave function (think of a sin curve squared) you get the probability density function, which will have nodes in it. Think of a mcdonalds sign, those points are the nodes, where the probability is zero.

This is severely problematic, because the particle spends 1/2 it's time on either side of that node, but can never cross the node in between them. How can this be possible. There are a two possibilities:

1) probabilistic superposition: the particle exists at all points simultaneously

2) teleportation: the particle teleports from one position to the next passing over the node.

Uncertainty in measurement is still possible, but it doesn't explain the problem of passing nodes.

Anyways, just something I remembered
 
for the basilisk to effectively create a perfect it would have to model the universe, which would require it to model itself, which would have to contain a model of the universe, which would have to have a model of itself
This was what confused me. The AI doesn't need to model itself, only the ways in which it affects the universe. Also, I thought they said that on the article but the problem with imperfections in the simulations is solved by doing many of them and taking the collective result.
--------------------------
On the uncertainty of neuron behaviour:
I think that paper confuses determinism with predictability. Chaos is deterministic but becomes unpredictable because you can't know to an infinite precision its initial conditions (they say that this means not being deterministic). A system is stochastic (not deterministic) when, even with complete information you cannot predict with 100% certainty what will happen.

I say this but, on the other hand, cells are indeed effectively stochastic due to the reasons they say in the paper. Low number of molecules makes it hard to predict when reactions will happen et cetera. But this stochasticity is due to the level of description that we look at (the cell), I'd have thought that molecule trajectories still were deterministic. Maybe I'm wrong, I'll have to think about it (I actually work on this: stochasticity in gene expression etc, so it's relevant to me).

In any case, I don't see the philosophycal advantadge of having our brains be stochastic. In the end if theres nothing 'extra' or metaphysical (such as a soul) influencing what you do, either it is predetermined (shit) or it is random according to laws you don't control (shit), that randomness does not restore the illusion of free will does it?[DOUBLEPOST=1416617755,1416617323][/DOUBLEPOST]
Fwiw: I ardently disagree with the interpretation of quantum mechanics as meaning the universe is probabilistic in nature. The reason we cannot develop deterministic models at the quantum level is because we cannot measure things that small without affecting them (thus Heisenberg and his principle). This doesn't mean the universe must roll probabilities to function but that we have no other way to model them but with probabilities.

.
I thought Bell's inequalities (http://en.wikipedia.org/wiki/Bell's_theorem#Importance_of_the_theorem) proved that there are no hidden variables that can predict what the outcome of a 'probabilistic' measurement.
Also, I'm pretty sure the uncertainty principle is not really about modifying that which you are measuring but something more fundamental. I think trying to get Quantum Mechanics to be deterministic is imposing on it how the world should be. The probabilities we use work so magnificiently that... why shouldn't they be true?
 

Necronic

Staff member
Does the stochastic model give us free will? Not...really... I guess...

I thought about it a lot. What it does is gives us a randomized in our actions, like a random number seed (or seeds, but it's the same thing). So our "soul" is basically a unique random number seed. Pick a number folks! I'm going with 6942069
 
Oh yeah and I did spend some more time reading up on this and LessWrong doesn't seem to buy it at all anymore, but...I have yet to hear any refutation of the initial comments made by EY, which sort of make him sound like a crazy person.
There is some of that here. I haven't read all of it but for instance he says:
EY said:
Why I yelled at Roko: Because I was caught flatfooted in surprise, because I was indignant to the point of genuine emotional shock, at the concept that somebody who thought they'd invented a brilliant idea that would cause future AIs to torture people who had the thought, had promptly posted it to the public Internet. In the course of yelling at Roko to explain why ths was a bad thing, I made the further error---keeping in mind that I had absolutely no idea that any of this would ever blow up the way it did, if I had I would obviously have kept my fingers quiescent---of not making it absolutely clear using lengthy disclaimers that my yelling did not mean that I believed Roko was right about CEV-based agents torturing people who had heard about Roko's idea
 
This was what confused me. The AI doesn't need to model itself, only the ways in which it affects the universe. Also, I thought they said that on the article but the problem with imperfections in the simulations is solved by doing many of them and taking the collective result.
--------------------------
On the uncertainty of neuron behaviour:
I think that paper confuses determinism with predictability. Chaos is deterministic but becomes unpredictable because you can't know to an infinite precision its initial conditions (they say that this means not being deterministic). A system is stochastic (not deterministic) when, even with complete information you cannot predict with 100% certainty what will happen.

I say this but, on the other hand, cells are indeed effectively stochastic due to the reasons they say in the paper. Low number of molecules makes it hard to predict when reactions will happen et cetera. But this stochasticity is due to the level of description that we look at (the cell), I'd have thought that molecule trajectories still were deterministic. Maybe I'm wrong, I'll have to think about it (I actually work on this: stochasticity in gene expression etc, so it's relevant to me).

In any case, I don't see the philosophycal advantadge of having our brains be stochastic. In the end if theres nothing 'extra' or metaphysical (such as a soul) influencing what you do, either it is predetermined (shit) or it is random according to laws you don't control (shit), that randomness does not restore the illusion of free will does it?[DOUBLEPOST=1416617755,1416617323][/DOUBLEPOST]
I thought Bell's inequalities (http://en.wikipedia.org/wiki/Bell's_theorem#Importance_of_the_theorem) proved that there are no hidden variables that can predict what the outcome of a 'probabilistic' measurement.
Also, I'm pretty sure the uncertainty principle is not really about modifying that which you are measuring but something more fundamental. I think trying to get Quantum Mechanics to be deterministic is imposing on it how the world should be. The probabilities we use work so magnificiently that... why shouldn't they be true?
I am not familiar enough with Bell's theorem but it looks like it hasn't been fully tested as of yet. As to your question: what better option could we have? Alternately: perhaps the probability mathematics is a good approximation. Of something physical (like subatomic energy states or something we cannot observe) and so probabilities serve as a very good proxy. We are spinning out of reach for me now, but I am always interested to read more. Thanks for the link.
 
Top