So now I probabilistically spam a ton of different orders to on average get my desired fill... This just turns it into a "whoever is best at DoS'ing the exchange" game. As the orderbook fills with competitor orders it makes sense for yourself to also spam orders so each of your orders maintains the same probability of being filled

Exchanges have requirements imposed on HFTs to prevent this kind of abuse. This one would be no different.

Impose a small order fee.

That will tend to discriminate against smaller traders, like 'retail' traders.

> will tend to discriminate against smaller traders, like 'retail' traders

Retail rarely hits an exchange.

Retail usually has larger fees unless you mean Robin Hood which is monetizing you in other ways.