26 September, 2021

Honoring Petrov Day by NOT Pressing the Button

Thirty-eight years ago, Stanislav Petrov disobeyed orders that may have caused a nuclear attack. I'll quote from Yudkowsky's retelling of Petrov's story:

On September 26th, 1983, Lieutenant Colonel Stanislav Yevgrafovich Petrov was the officer on duty when the warning system reported a US missile launch.  Petrov kept calm, suspecting a computer error.

Then the system reported another US missile launch.

And another, and another, and another. 

What had actually happened, investigators later determined, was sunlight on high-altitude clouds aligning with the satellite view on a US missile base.

In the command post there were beeping signals, flashing lights, and officers screaming at people to remain calm.  According to several accounts I've read, there was a large flashing screen from the automated computer system saying simply "START"….

Petrov decided that, all else being equal, he would prefer not to destroy the world.  He sent messages declaring the launch detection a false alarm, based solely on his personal belief that the US did not seem likely to start an attack using only five missiles.

Petrov was first congratulated, then extensively interrogated, then reprimanded for failing to follow procedure.  He resigned in poor health from the military several months later.

Each year, I and many others take a moment to think back to the day when the world as we know it almost died. Of all the traditions I follow, this is perhaps the most solemn. (In 2018, I attended a ceremony where the Future of Life Institute posthumously presented Stanislav Petrov the $50,000 Future of Life Award.)

From Petrov Day 2020.
This year, I have been invited to take part in an experiment of mutually assured destruction. LessWrong and the Effective Altruism Forum have decided to honor Petrov day by creating buttons on each site which, if pressed with the appropriate arming code, will take the other site down for the duration of the day. I was chosen by the EA Forum as one of the people trusted with the launch codes capable of taking down LessWrong's site.

To outsiders, this exercise may seem silly. It has the appearance of a mere game, but I think it is much more than that: it is a serious ritual, one where the stakes involve thousands of visitors to each site, one where defection will be public, one where we practice the very real act of not causing wanton destruction due to mistrust, carelessness, or flippancy. But yes, it is also a game: one with stakes we should not callously risk.

Last year, this experiment failed. LessWrong user Chris Leong pressed the button, taking down the site during Petrov Day 2020. The failure, I believe, was not entirely on his part, but also due to a poor choice of who would be entrusted with the launch codes. I am hopeful that the decision to trust me with the codes this year will not be in vain.

At the same time, I am cognizant that the concept of mutually assured destruction here is supposed to incentivize the other team to not press their button. This presents a dilemma to me: I honestly do not want to press a button that will take down LessWrong's site. But should I keep open the possibility, should LessWrong press their button to take down the EA Forum? In order for the threat of MAD to work, I must precommit to taking an action that might not make sense in the moment when I have to take it. But I abhor the idea of precommitting myself to such an action.

Homepage of the EA Forum today.
I'm not going to strike first. That much is certain. But I'm less sure about my stance on a retaliatory strike. I want to say that even if they fire first, I will not fire back. What use is there in additional destruction? But this intellectually seems like the wrong stance to take. This exercise is repeated each year; Tit for Tat does seem like the better policy. That requires precommitting to MAD. At the same time, I don't take precommitting to anything lightly.

So here I stake my claim: if the EA Forum goes down due to LessWrong pressing their button, I may press in retaliation. This is not an idle threat. I do think that I may press, just to ensure that future Petrov days don't undergo the same terrible defection. But I'm not precommitting. Hopefully, LessWrong will understand this to be a credible threat, even if not entirely likely. I am hopeful that this small amount of threat will be sufficient to prevent them from deciding to press their button.

(If you are reading this on Petrov Day, Sept 26, after 11 a.m. ET, you can see the button on LessWrong and the EA Forum's home pages if they are still up. Or, if one side has already defected, you will see that the other side's site will be taken down.)





38 years ago, Stanislav Petrov saved the world. This year, I was chosen by the Effective Altruism Forum as steward of...

Posted by Eric Herboso on Sunday, September 26, 2021

No comments:

Post a Comment