We have the right to turn them away granted by our ability to use force, and we absolutely should if they are making our lives worse, period, end of story
"Might makes right" is not only a pretty heartless ideology but one that I don't think works well most of the time. It seems like the prior application of that policy caused much of our immigration today.
> if they are making our lives worse
They (legal and illegal immigrants, for this discussion specifically from Latin America but I suspect more broadly) certainly are making my life better and I'm not convinced that they're making the life of the average American worse.
I've seen a lot of convincing arguments for why immigration helps us, and they seem to mesh with my understanding of immigration to the US being historically a good thing. I can't say the same for arguments from the other side. Often the claims are totally baseless, such as claims that illegal immigrants are as a whole dangerous, despite having a lower incarceration rate than US-born citizens. Other times they're just myopic (most of the claims related to jobs imo). Sometimes they're just blatant racism (white-replacement theories).