This is your host idling the connection due to the silence. Just keep something playing (like a stream of almost-silence) on loop and you won't have this problem.
Most reviewers are already utterly unable to measure "normal" latency. In the very ridiculous chance you'll find a reviewer measuring wake up latency (which has little to do with the codec used), I wouldn't even trust it.
Yes, wake up latency is going to be a crapshoot, but you can eliminate it by playing a silent file at the expense of battery life. Actual latency though there's nothing you can do about it, so it's more important imo.
