Pygmy populations are ordinary humans with conserved genetic mutations, due to inbreeding, that limit the expression of things like growth hormones. They are anatomically modern humans, just much smaller. The ease with which these mutations can occur is evidenced by the existence of unrelated Pygmy populations in different parts of the world. There is no evidence of speciation or material genetic divergence in these cases. While we can't say anything for sure, the genetic divergence between modern humans and these other hominid species is significantly larger.
I agree with 100% of what you said, but if we didn't have access to the genes and were merely going by fragments of a few fossils - what then? Can one quantify the magnitude of difference between H. nalendi and H. erectus fossils, and compare it to the magnitude of difference between hypothetical Polynesian and Pygmy fossils?
You'd still notice a difference between two lineages diverging for a few hundreds or thousands of years and a two lineages diverging for hundreds of thousands of years or more.
It's not just size or deformity, but other features that will diverge.
