The problem is downvotes mean different things to different people. IMO a point based system for comments particularly one with downvoting is a sort of an aggressive one that may steer certain folks away when really good community shouldn't. It might even create a bad culture and attract hyper aggressive and deter more passive folks. There are other gamification systems that don't allow downvotes (e.g. Quora) which I think are better.

When I have been downvoted in the past for opinion things I'm immediately concerned that I have offended someone. It actually is painful not to know... did I say something factually wrong... is my comment being misinterpreted... am I coming off as offensive?

I'm sorry I'm not an "adult" but I need explanation as to why a comment is incorrect or why you might disagree otherwise I'm not going to sugar coat... it doesn't feel good to be downvoted and I'm sure others feel the same.