AFAIK Snips.ai supports recognizing wake words on the pi zero w, and then have the actual speech recognition happen on some other machine.
I guess something similar should be possible with Rhasspy?
I was part way through a "smart speaker" project and planned to use Snips.ai, but I see now that they've been bought by Sonos so Rhasspy is looking pretty tempting now.
However my plan was to use pi zero's at the speaker end, with my beefier HA machine doing the speech recognition.