I'm usually not the 'reduce it to math' person but this seems like a perfect example. Facial recognition in policing fails at a mathematical level because of Bayes.
Let's say I have a database of 10,000 people's faces and I'm looking to match 1 accused criminal from a good quality image. Let's also say my algorithm is 99.99% accurate with 100% specificity (i.e., I always ID the bad guy as the bad guy).
So I get a true positive, I can correctly ID the guy. But in doing so I am 10 times as likely to get a false positive. at 99.99% accuracy. Scale it to 1,000,000 people and that likelihood of a false positive is 100x. Because you are looking for a small number of people as your database gets larger the value of a positive match (i.e., your positive predictive value) rapidly approaches zero.
Yes this is a simplification but the basic idea stands. These algorithms used at scale are flat out unacceptable because the concept of their use makes false positives more likely than true positives.
Let's say I have a database of 10,000 people's faces and I'm looking to match 1 accused criminal from a good quality image. Let's also say my algorithm is 99.99% accurate with 100% specificity (i.e., I always ID the bad guy as the bad guy).
So I get a true positive, I can correctly ID the guy. But in doing so I am 10 times as likely to get a false positive. at 99.99% accuracy. Scale it to 1,000,000 people and that likelihood of a false positive is 100x. Because you are looking for a small number of people as your database gets larger the value of a positive match (i.e., your positive predictive value) rapidly approaches zero.
Yes this is a simplification but the basic idea stands. These algorithms used at scale are flat out unacceptable because the concept of their use makes false positives more likely than true positives.