Machine learning · Python

A fraud detector designed around the cost of being wrong.

How do you detect a rare event without creating an expensive stream of false alarms?

Role
End-to-end analysis and modelling
Format
Independent project
Tools
Python · Pandas · scikit-learn · Random Forest
  • 284,807Transactions
  • 0.17%Fraud rate
  • ~84%F1, 5-fold CV
  • 0.4Suggested review threshold

The problem

Rare-event accuracy was the wrong target.

Fraud made up less than 0.2% of the transaction data. That imbalance made overall accuracy nearly useless. The useful question was whether the model could catch fraud while keeping manual review manageable.

A model can look excellent by predicting every transaction as legitimate. I needed evaluation that reflected the cost of missed fraud and the operational cost of false positives.

The approach · 4 steps

Evaluate the model around the review decision.

  1. Cleaned and standardized the anonymized transaction features.
  2. Used five-fold cross-validation with F1 as the primary metric.
  3. Trained a Random Forest and reviewed precision, recall, ROC-AUC, and the confusion matrix together.
  4. Compared decision thresholds based on the review burden they would create.

The result

The threshold sets the workload.

  • ~84%Mean cross-validated F1
  • V17 / V12 / V14Strongest model features
  • 0.4Suggested review threshold

The model score was only the start. The threshold determines how risk policy becomes day-to-day review work, so it should be chosen with the people handling the alerts.

What I would do next

  • Add cost-sensitive evaluation using fraud loss and review cost.
  • Compare boosting methods and resampling strategies.
  • Package the model as a real-time scoring service.