> how could colonial historians know that "these groups had diminished" before colonialism when they weren't there?

is this a serious question? what defines history as a subject is precisely that it is not the present.