Things I’m building
Case study 02
Machine learning · Python
A fraud detector designed around the cost of being wrong.
How do you detect a rare event without creating an expensive stream of false alarms?
- Role
- End-to-end analysis and modelling
- Format
- Independent project
- Tools
- Python · Pandas · scikit-learn · Random Forest
- 284,807Transactions
- 0.17%Fraud rate
- ~84%F1, 5-fold CV
- 0.4Suggested review threshold
The problem
Rare-event accuracy was the wrong target.
Fraud made up less than 0.2% of the transaction data. That imbalance made overall accuracy nearly useless. The useful question was whether the model could catch fraud while keeping manual review manageable.
A model can look excellent by predicting every transaction as legitimate. I needed evaluation that reflected the cost of missed fraud and the operational cost of false positives.
The approach · 4 steps
Evaluate the model around the review decision.
- Cleaned and standardized the anonymized transaction features.
- Used five-fold cross-validation with F1 as the primary metric.
- Trained a Random Forest and reviewed precision, recall, ROC-AUC, and the confusion matrix together.
- Compared decision thresholds based on the review burden they would create.
The result
The threshold sets the workload.
- ~84%Mean cross-validated F1
- V17 / V12 / V14Strongest model features
- 0.4Suggested review threshold
The model score was only the start. The threshold determines how risk policy becomes day-to-day review work, so it should be chosen with the people handling the alerts.
What I would do next