> The paper talks about using this to save power, but that’s probably not worth it for a few reasons

It's probably not worth it to do this in software. But in hardware, it might be!

A similar approach with a hardware prototype. https://research.nvidia.com/publication/2022-12_lns-madam-lo...