I agree that will have a bigger effect, and boo Brexit. But it's actually only transactions on EU shops by UK cards that face higher fees, which is a really small percentage of purchases. Definitely not every transaction.
I'm sure they'll raise the other fees later though. Probably just dipping their toes.