Also, counties often have their own sheriffs with a separate set of powers and responsibilities. Sometimes court systems have their own enforcement arms, separate from other agencies.
The US often seems to have more layers of administration and law-making than some other places. Making individual officers work through which set of powers and laws they're dealing with in a given moment seems like it might at times be a daunting prospect. You could in theory organize them into a single bureau, but I suspect you'd inevitably wind up with specialized sections to deal with the particular laws in given cities.
And France and Germany are each twice as populous as the most populous US state.
So you're not really correct.
There's absolutely no reason why having a single bureaucracy wouldn't work well.
It actually seems like it would be much easier to have a single bureaucracy.