Oddly enough I was rocking 100% accuracy by ignoring the face and looking at the background.

Maybe a face-centric ML isn't so good at backgrounds. Shocker!