> I can't even picture a functional world where humans are cut out of most professions that don't involve manual work
Many people do manual work. It may very well be the case that general fine motor skills are a far more complex and difficult operation than the entire edifice of human intellect. Philosophically, it would be an immense blow, the mother of all existential crises. But regardless, it suggests that the first AGI would be incapable of independent survival and we'd still be relevant and in control for a while.
> where any amount of acquired knowledge and skills will be surpassed by machines that can produce better results at a thousandth of the cost in a thousandth of the time a human can.
I don't think we can necessarily extrapolate it to be that cheap. It could be. But it is also possible that the increases in cost and resources to scale these models bigger and bigger will outstrip hardware progress and that this technology will run into a dead end. To put it differently, I think it is far from clear that the kind of hardware that we build is actually better than an organic substrate for this sort of computation. Imagine an optimized organic neural implementation of ChatGPT, for instance. Would it be slower or more expensive than ChatGPT? Perhaps not. Likewise, the very best that the current paradigm can offer may not be faster or cheaper than humans are at quite many valuable tasks.
Many people do manual work. It may very well be the case that general fine motor skills are a far more complex and difficult operation than the entire edifice of human intellect. Philosophically, it would be an immense blow, the mother of all existential crises. But regardless, it suggests that the first AGI would be incapable of independent survival and we'd still be relevant and in control for a while.
> where any amount of acquired knowledge and skills will be surpassed by machines that can produce better results at a thousandth of the cost in a thousandth of the time a human can.
I don't think we can necessarily extrapolate it to be that cheap. It could be. But it is also possible that the increases in cost and resources to scale these models bigger and bigger will outstrip hardware progress and that this technology will run into a dead end. To put it differently, I think it is far from clear that the kind of hardware that we build is actually better than an organic substrate for this sort of computation. Imagine an optimized organic neural implementation of ChatGPT, for instance. Would it be slower or more expensive than ChatGPT? Perhaps not. Likewise, the very best that the current paradigm can offer may not be faster or cheaper than humans are at quite many valuable tasks.