It is a fundamental, structural issue with the first step of the data generation process and can exist even given perfect sampling and feature selection. Gathering more data is usually not the way to fix it as the problem is with the method that generates the data, that mechanism is biased (for instance, the justice system - if it is biased, anything we trained on its outputs will inherit the bias).

One way to mitigate this would be to talk to domain experts and those impacted.