You have to limit the traffic to that pool to prevent people accessing your server directly. But that's not enough on its own, because other people can use CloudFlare's IPs to scan you too, so you need some kind of auth on top or use the tunnel.

Yes, this is correct. If you're using IP address allowlists then you also have to check the Host HTTP header (Cloudflare won't allow their other customers to forge that header). Or, you can use mTLS (as another commenter pointed out), or tunnels (as I pointed out): https://news.ycombinator.com/item?id=26690388

Typically my servers is behind NAT and it has no public address, one can only reached the service through the CF tunnel and my access is through VPN, this should be safe, right?

they also provide certs for mTLS between cloudflare and your origin, which you can layer in along with IP restrictions

(the term they use is “authenticated origin pull”)