The short version is this: Property owners now have shocking amounts of control over what their neighbors do with property. Owners, especially of single-family houses, elect officials who restrict development through zoning and similar means. Height limits and parking requirements effectively mandate single, detached housing in most of the U.S.
In the meantime, much of the population growth has shifted to Sun Belt cities in Texas, Arizona, Georgia, and Florida where development is easier and/or simply sprawls more.
