I agree with 100% of what you said, but if we didn't have access to the genes and were merely going by fragments of a few fossils - what then? Can one quantify the magnitude of difference between H. nalendi and H. erectus fossils, and compare it to the magnitude of difference between hypothetical Polynesian and Pygmy fossils?
You'd still notice a difference between two lineages diverging for a few hundreds or thousands of years and a two lineages diverging for hundreds of thousands of years or more.
It's not just size or deformity, but other features that will diverge.
