I've been thinking a bunch lately about designing systems to prevent misuse, ensure appropriate use, etc.
We talk all the time about dual-control systems and separation of duties and when they are strictly necessary to ensure security or some desired system property. It reminds me of a scene from Dr. Strangelove........
General "Buck" Turgidson: Mr. President, about, uh, 35 minutes ago, General Jack Ripper, the commanding general of, uh, Burpelson Air Force Base, issued an order to the 34 B-52's of his Wing, which were airborne at the time as part of a special exercise we were holding called Operation Drop-Kick. Now, it appears that the order called for the planes to, uh, attack their targets inside Russia. The, uh, planes are fully armed with nuclear weapons with an average load of, um, 40 megatons each. Now, the central display of Russia will indicate the position of the planes. The triangles are their primary targets; the squares are their secondary targets. The aircraft will begin penetrating Russian radar cover within, uh, 25 minutes.
President Merkin Muffley: General Turgidson, I find this very difficult to understand. I was under the impression that I was the only one in authority to order the use of nuclear weapons.
General "Buck" Turgidson: That's right, sir, you are the only person authorized to do so. And although I, uh, hate to judge before all the facts are in, it's beginning to look like, uh, General Ripper exceeded his authority.
Makes you think hard about designing systems to prevent what could happen vs. what you expect to happen, as was nicely pointed out again by Rob Newby.
Sometimes we do go overboard with dual-control and such, sometimes though I'm pretty happy that we design systems with that built in.
Now if we just didn't set the Permissive Action Link codes to all zeros we'd be fine.