If you're using solely the flow labels to do load balancing, malicious clients can force traffic to come through only one load balancer by setting the same flow label.

You need to add the source IP/port into the mix. But they alone are in practice enough for decent load balancing.