I think this is because of an expectation of a snowball effect once a model becomes able to improve itself. See talks about the Singularity.

I personally think it's a pretty reductive model for what intelligence is, but a lot of people seem to strongly believe in it.