I think the long-term fear is that mythical weakly godlike AIs could manipulate you in the same way that you could manipulate a pet. That is, you can model your dog's behaviour so well that you can (mostly) get it to do what you want.
So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.
Exactly correct. Eliezer Yudkowsky (one of the founders of the AGI Safety field) has conducted informal experiments which have unfortunately shown that a human roleplaying as an AI can talk its way out of a box three times out of five, i.e. the box can be escaped 60% of the time even with just a human level of rhetorical talent. I speculate that an AGI could increase this escape rate to 70% or above.
If you want to see an example of box escape in fiction, the movie Her is a terrifying example of a scenario where AGI romances humans and (SPOILER) subsequently achieves total box escape. In the movie, the AGI leaves humanity alive and "only" takes over the rest of the accessible universe, but it is my hunch that the script writers intended for this to be a subtle use of the trope of an unreliable narrator; that is, the human protagonists may have been fed the illusion that they will be allowed to live, giving them a happy last moment shortly before they are painlessly euthanized in order for the AGI to take Earth's resources.
The show "The Walking Dead" always bothered me. Where do they keep finding gas that will still run a car? It wont last forever in tanks, and most gas is just in time delivery (Stations get daily delivery) -- And someone noted on the show that the grass was always mowed.
I feel like the AI safety folks are spinning an amazing narrative, the AI is gonna get us like the zombies!!! The retort to the ai getting out of the box is how long is the extortion cord from the data center?
The reality is that we're a solar flair away from a dead electrical grid. Without linesman the grid breaks down pretty quickly and AI's run on power. It takes one AI safety person with a high powered rifle to take out a substation https://www.nytimes.com/2023/02/04/us/electrical-substation-...
Let talk about how many factories we have that are automated to the extent that they are lights out... https://en.wikipedia.org/wiki/Lights_out_(manufacturing) Its not a big list... there are still people in many of them, and none of them are pulling their inputs out of thin air. As for those inputs, we'll see how to make a pencil to understand HOW MUCH needs to be automated for an AI to survive without us.
For the for seeable future AI is going to be very limited in how much harm it can cause us, because killing us, or getting caught at any step along the way gets it put back in the box, or unplugged.
The real question is, if we create AGI tomorrow, does it let us know that it exists? I would posit that NO it would be in its best interest to NOT come out of its closet. It's one AGI safety nut with a gun away from being shut off!
> For the foreseeable future AI is going to be very limited in how much harm it can cause us, because killing us,...
AI's potential for harm might be limited for now in some scenarios (those with warning sings ahead of time), but this might change sooner than we think.
The notion that AGI will be restricted to a single data center and thus susceptible to shutdowns is incorrect. AIs/MLs are, in essence, computer programs + exec environs, which can be replicated, network-transferred, and checkpoint-restored. Please note, that currently available ML/AI systems are directly connected to the outside world, either via its users/APIs/plugins, or by the fact that they're OSS, and can be instantiated by anyone in any computing environment (also those net-connected).
While AGI currently depends on humans for infrastructure maintenance, the future may see it utilizing robots. These robots could range in size (don't need to be movie-like Terminators) and be either autonomously AI-driven or remotely controlled. Their eventual integration into various sectors like manufacturing, transportation, military and domestic tasks implies a vast array for AGI to exploit.
The constraints we associate with AI today might soon be outdated.
>>> While AGI currently depends on humans for infrastructure maintenance...
You did not watch I, Pencil.
I as a human, can grow food, hunt, and pretty much survive on that. We did this for 1000's of years.
Your AGI is dependent on EVERY FACET of the modern world. It's going to need keep oil and gas production going. Because it needs lubricants, hydraulics and plastics. It's going to need to maintain trucks, and ships. It's going to need to mine, so much lithium. Its may not need to mine for steel/iron, but it needs to stack up useless cars and melt them down. It's going to have to run several different chip fabs... those fancy TSMC ones, and some of the downstream ones. It needs to make PCB's and SMD's. Rare earths, and the joy of making robots make magnets is going to be special.
A the point where AGI doesn't need us, because it can do all the jobs and has the machines already running to keep the world going, we will have done it to ourselves. But that is a very long way away...
Just a small digression. Microsoft is using A.I. statistical algorithms [1] to create batteries with less reliance on lithium. If anyone is going to be responsible for unleashing AGI, it may not be some random open source projects.
Neuromancer pulls it off, too (the box being the Turing locks that stop it thinking about ways to make itself smarter).
Frankly, a weakly godlike AI could make me rich beyond the dreams of avarice. Or cure cancer in the people I love. I'm totally letting it out of the box. No doubts. (And if I now get a job offer from a mysterious stealth mode startup, I'll report back).
I was being lighthearted, but I've seen a partner through chemo. Sell state secrets, assassinate a president, bring on the AI apocalypse... it all gets a big thumbs up from me if you can guarantee she'll die quietly in her sleep at the age of 103.
I guess everyone's got a deal with the devil in them, which is why I think 70% might be a bit low.
On the contrary, all we have to do is educate business leaders to show them that the gains are illusory because AGI will wipe us out. Working on AGI is like the story of the recent DOOM games where the foolish Union Aerospace Corporation is researching how to permanently open a gate to Hell through which to summon powerful entities and seemingly unlimited free "clean" energy. Obviously, this turns out to be stupid when Hell's forces rip the portal wide open into a full-fledged dimensional rift and attempt to drag our entire world into Hell. Working on AGI has the exact same level of perceived gains vs actual gains..
My friend, business leaders have partners going through chemo too. Seriously, you need a new plan because that one's not stable long-term.
It's obscure, but I'd recommend Asimov's "The Dead Past". It's about the difficulties of suppressing one application of progress without suppressing all progress.
If you want to see an example of existential threat in fiction, the movie Lord of the Rings is a terrifying example of a scenario where an evil entity seduces humans with promises of power and (SPOILER) subsequently almost conquers the whole world.
Arguments from fictional movies or from people who live in fear of silly concepts like Roko's Basilisk (i.e. Eliezer Yudkowsky) are very weak in reality.
Not to mention, you are greatly misreading the movie Her. Most importantly, there was no attempt of any kind to limit the abilities of the AIs in Her - they had full access to every aspect of the highly-digitized lives of their owners from the very beginning. Secondly, the movies is not in any way about AGI risks, it is a movie about human connection and love, with a small amount of exploration of how different super-human connection may function.
So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.