Your intuition is right, in that most of the heavy math is simply unnecessary or irrelevant at the practical level. Complex number notation doesn't fall into that category, though. The notion of frequency is intimately tied to the notion of vector rotation, and just as you can't represent Cartesian translation without negative numbers, you can't represent Cartesian rotation without complex ones.
The Smith book is definitely for you; also check out Rick Lyons's books. Bo Pirkle and Julius O. Smith are good for audio-specific theory and applications.
Saying that you can't represent cartesian rotation without complex numbers seems a bit much? You do need two numbers. But you can teach a trigonometry class without complex numbers, using either cartesian coordinates or polar notation. And matrices do work.
The Smith book is definitely for you; also check out Rick Lyons's books. Bo Pirkle and Julius O. Smith are good for audio-specific theory and applications.