The size of the gap between “smarter than humans” and “not controlled by humans anymore” is obviously where the disagreement is.
To assume it’s a chasm that can never be overcome, you need at least the following to be true:
No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls.
I’m less worried bad actors control it than I am that it escapes them and is badly aligned.
I think the greatest concern is not so much that a single AI will be poorly aligned.
The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them.
If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function.
And if they're sufficiently advanced by then, we will have no way of knowing.
Totally. I’ve wondered how you safeguard humans in such a scenario. Not sure it can be done, even by self modifying defenders who religiously try keep us intact.
I also somewhat assume it’ll get Darwinian if there are multiple tribes of either humans or AI’s, through sheer competition. if we aren’t in this together we’re in shit.
I think we should assume it will be badly aligned. Not only are there the usual bugs and unforeseen edge conditions, but there are sure to be unintended consequences. We have a long, public history of unintended consequences in laws, which are at least publicly debated and discussed. But perhaps the biggest problem is that computers are, by nature, unthinking bureaucrats who can't make the slightest deviation from the rules no matter how obviously the current situation requires it. This makes people livid in a hurry. As a non-AI example (or perhaps AI-anticipating), consider Google's customer support...
To assume it’s a chasm that can never be overcome, you need at least the following to be true:
No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls.
I’m less worried bad actors control it than I am that it escapes them and is badly aligned.