I agree that humans and AI are in the same boat here.

It's valid to take either position, that both can be aware of truth or that neither can be, and there has been a lot of philosophical debate about this specific topic with humans since well before even mechanical computers were invented.

Plato's cave comes to mind.