Question 6/11 fast.ai v4 lecture 3

What is one of the most challenging things (and why) when deploying ML models?

Answer

Feedback loops - because they can take a really minor issue and explode it into a major one. For instance, consider predictive policing. You police the areas the model suggests, you find crime as a natural byproduct of focusing on these areas, you feed that new data into the model and you got yourself a feedback loop! The model is now even more likely to suggest that these areas are to be policed, and if unchecked this model would lead to suggesting the entire police force should focus their efforts on a very small area. To quote a paper on this subject: "predictive policing is aptly named: it is predicting future policing, not future crime".

Relevant part of lecture