They actually did scale it down from the original plans in ~2015.

I'm not sure if it's worth scaling down substantially from where it is. There is a certain scale that's necessary for the economics (payload vs dry-weight vs fuel) to work out, and ultimately they want to be able to put very significant amounts in orbit. They're currently targeting 150 tons, would it be worth targeting 50 tons? Probably not.

Like single stage? I don't think the booster can reach orbit by itself (much less carry something and come back).

I doubt it, the size is what makes it economical in the long run. For anything smaller there's F9 or Heavy.