However, it also becomes a human intelligence vs system problem, where people are exchanging notes about the model offline in terms of how to get it to offer the most favourable outcome.
If the insurance company models loss-causing outputs as Bernoulli trials (i.e. each time the LLM is used, it is an independent and identically distributed event - equal chances of an error), but they are actually correlated due to information sharing, then that could make it harder for them.
If the insurance company models loss-causing outputs as Bernoulli trials (i.e. each time the LLM is used, it is an independent and identically distributed event - equal chances of an error), but they are actually correlated due to information sharing, then that could make it harder for them.