Just Machine Learning

• A program learns from experience E (with respect to task T and performance measure P), if it's performance on the task T improves from experience E.
• Many failures of ML are due to E not being right for T.
• Friedman + Nissenbaum 1996 on sources of bias:
  • Preexisting bias,
  • Technical bias (from tech constraints and considerations),
  • Emergent bias (from context of use).
• How do we define fairness in ML context:
  • Probabilistically,
  • Independence of some "sensitive" attributes,
  • Many possible senses, it's complicated.
  • Measure via confusion matrix: True positives, False positives, Precision.
• Impossibility results:
  • Kleinberg, Mullainathan, Raghavan 2016
  • Chouldechova 2016
• There a technical solutions and policy soultions (e.g. right to explanation).
• Often formulating the right objective function is big part of the solution. What do we actually care about?
• Take-away: Estimating risk assessment scores is bad, we need a different problem formulation.
  • Return the list of better and worse cases and let the judge decide. Basically we will learn the ordering this way, instead of a mapping to a probability.
    • Regression is suboptimal because the target variable is often heavy-tailed and regression assumes a more normal distribution.
• We want to keep the decision with a human so that someone is liable for the decision.
• Learning from pairwise comparisons seems to be more robust, perhaps because humans are better at pairwise comparisons.

• http://eliassi.org/tina_justML_july2018.pdf