02RanClassification

Online payment fraud

I encoded the transaction type as a number and trained a decision tree on type, amount and the sender's balance before and after the payment, then scored it on a shuffled 20% of the rows.

Original tutorial by Aman Kharwal: Online Payments Fraud Detection (opens in a new tab)

What it printed in 2022

Numbers read from the saved notebook outputs. Cell numbers count every cell in the notebook, Markdown included, starting at 1.
Output	Value	Cell #
Test accuracy (decision tree)	99.97%	16
Transactions	6,362,620	6

Cell # counts every cell from the top of the notebook, Markdown included, starting at 1.

Accuracy is the wrong yardstick when only about one transaction in 775 is fraud. A model that never flags fraud already scores about 99.87%.

The data

PaySim, a synthetic mobile-money log of 6.36 million transactions (Kaggle, needs a login). The CSV is not in the repository.

What I'd do differently

Report recall and precision for the fraud class, not accuracy.
Drop the balance columns. The dataset's own notes say fraudulent transactions were cancelled, so those balances describe what happened after detection.
Use a stratified split and a precision-recall curve, then pick a threshold from the cost of a missed fraud versus a false alarm.

Re-checked in the audits

02
InteractiveCan't tell from 2022
99.97% accuracy on a 1-in-775 problem
99.97% is 381 mistakes on 1,272,524 test rows. That is consistent with catching every fraud, or with missing almost a quarter of them, and the features include balances the dataset's author says must not be used.
Read the audit

What it printed in 2022

The data

What I'd do differently

Re-checked in the audits

99.97% accuracy on a 1-in-775 problem

What it printed in 2022

The data

What I'd do differently

Re-checked in the audits

99.97% accuracy on a 1-in-775 problem