It is true that NYC is a popular place to live, but the higher apartment costs are caused by zoning density restrictions and overuse of historic landmark status. These are "economic rents" -- a market failure that creates inefficient markets through use of politics by special interests -- in this case landlords that want to extract higher rents from tenants.
In general this is true (and certainly given that many HN readers live in the Bay area, it probably falls within people's personal experience), but NYC is not a great example. Parts of NYC have some degree of density restriction, but much of it (pretty much all of Manhattan, say) is among the highest-density residential areas in the US, and still very expensive, because even given all the density, there's a finite amount of land, which still means supply is constrained as compared to demand. Even given no density restrictions at all, some places would still be very expensive. Tokyo is another good example of this.
Yeah, the buildings aren't very tall, but people are in them right now.
Before you can build a newer, bigger building, you first need to buy a large enough contiguous area from potentially several owners who don't particularly want to move. That adds friction and years of extra work to any new building, which will discourage new construction, lower supply and raise prices regardless of the zoning.
Unless you're proposing that the government begin seizing tracts of land in Manhattan on behalf of high-rise developers?
I don't think this is as much of a problem as you are making it out to be. If the east village, which is currently mostly zoned R8B[1] were tomorrow to be substantially upzoned (to say R10) you'd see lots of new construction within a year. There'd be so much money to be made that deals would happen.
