A surprisingly easy way to multiply an AI model’s profit is to drive decisions via expected value instead of predictive scores. Here’s how, illustrated with fraud detection.
getty
I want to share with you what I consider to be a true breakthrough for predictive AI. It’s the most underutilized – yet shockingly simple – way to multiply the value you get from your machine learning projects.
Typically, when a business deploys a predictive AI model, it drives each decision with a raw model score. For example, the model might tell you there is a 30% chance a particular transaction is fraudulent. To make a decision, businesses usually compare that probability to a fixed threshold, like 50%, to decide whether to block the transaction.
But here is the paradigm shift: Instead of driving decisions with the raw model score, your model deployment should drive each decision with the expected value.
This very rare practice represents a straightforward “no-brainer,” both technically and conceptually. Here’s how it works, using a concrete example of payment card fraud detection.
The Flaw With Raw Risk Scores
Imagine you are running fraud detection for a bank. Your predictive model outputs a risk score (a probability) for every single transaction.
If you rely purely on the raw risk level to make each decision, you are treating a $100 transaction exactly the same as a $5,000 transaction. But a $5,000 fraudulent transaction costs your business significantly more than a $100 one. For larger transactions, the downside of undetected fraud is… larger. On the flip side, however, the cost of a “false positive” – wrongly blocking a legitimate transaction and inconveniencing the customer – typically doesn’t increase at the same steep rate, even for large purchases. For a given risk level (predictive model score), the upside can quickly outweigh the downside when deciding whether to block a higher-dollar transaction.
To account for this, many banks and businesses currently use clunky “if-then” business rules, setting different risk thresholds for different transaction sizes. They might say, “If the transaction is over $1,000, lower our risk tolerance.” However, these rules are discrete and rigid. They treat all transactions within a potentially wide range the same, so they don’t serve to optimize the “numbers game” that is fraud detection.
The Fix: Decisioning With Expected Value
To maximize the bottom line, we need a smooth, continuous approach. Rather than using rigid rules, we simply do a little bit of grade-school arithmetic before applying a decision threshold.
Step 1: Use your machine learning model to calculate the probability of fraud.
Step 2: Multiply that probability by the size of the transaction.
This simple multiplication gives you the expected value – which, in this context, is the expected amount of money you would save by blocking the transaction.
Let’s look at the math for an example:
- Transaction A: The model says there is a 20% (0.2) chance that the transaction is fraudulent. The transaction is for $100. Multiply 0.2 by $100 and your expected value of blocking this transaction is $20.
- Transaction B: The model says there is only a 5% (0.05) chance of fraud. The transaction is for $5,000. Multiply 0.05 by $5,000 and your expected value of blocking this transaction is $250.
Step 3: Instead of thresholding on a raw percentage (the risk probability), you draw a decision threshold based on the monetary expected value.
If you set your decision threshold at an expected value of $15, you would block both of the transactions above. Even though Transaction B is technically “low risk” at only 5%, the sheer size of the transaction means your risk tolerance should go down. Driving your decision based on the $250 expected value captures this perfectly.
Visualizing The Bottom-Line Impact
To truly understand why this is a massive win for your business, we need to look at a profit curve (or, similarly, a savings curve).
A profit curve (savings curve) depicting the money saved with fraud detection, in relation to the proportion of highest-risk transactions blocked. The Y-axis represents total “savings in dollars,” and the X-axis represents the “percentage of transactions blocked,” moving from the highest risk on the left to the lowest risk on the right. For this view, calculated across an assumed pool of 10 million transactions, each correctly-blocked fraudulent transaction (true positive) is assumed to save the bank the amount of that transaction and each wrongly-blocked legitimate transaction (false positive) is assumed to cost $75.
Eric Siegel
In a standard deployment based purely on the model’s risk score, you prioritize transactions from left to right based on the chance that they are fraud. The curve rises as you block the riskiest transactions and save money, eventually hitting a peak. After the peak, the curve drops because you start blocking too many legitimate transactions and the cost of false positives (inconveniencing customers) eats into your savings.
That curve looks pretty damn good. It peaks at more than $20 million. But what happens when we re-order our cases and prioritize them by expected value instead?
A comparison graph showing the original curve versus a new “expected value” curve. The new curve’s peak is significantly higher on the Y-axis (greater savings) and shifted further to the left on the X-axis (fewer transactions blocked).
Eric Siegel
What a difference! When you switch to an expected value approach, two incredible things happen on this chart:
- The peak is higher: You achieve a higher maximum savings than you ever could driving decisions based on the raw model score alone.
- The peak moves to the left: This is the best news of all. It means you reach those massive maximum savings while blocking a smaller number of transactions.
By more highly prioritizing the big-dollar transactions, you don’t need to disrupt commerce as much to maximize the savings. You need only block the smaller handful of transactions on the far left, those most deserving of intervention.
Beyond Fraud Detection
This methodology isn’t just for credit card fraud; it can elevate various predictive AI projects where the magnitude of the cases varies.
For example, in nonprofit fundraising, you can multiply a donor’s likelihood to give by their estimated wealth capacity to decide who receives an expensive outreach campaign. And the expected value approach has also been proven to dramatically improve decisioning for churn modeling, targeting collections and credit scoring.
This approach is as impactful and straightforward as it is rare. I mean, I’ve basically never heard others in the industry even bring up the notion! And senior practitioners to whom I’ve presented the idea respond, “Oh yeah! Great idea! I’ll have to try that.”
So take a moment to look at your current predictive AI initiatives. By taking this simple step – transitioning from pure model probabilities to estimated monetary value – you stand to multiply the returns of your machine learning model deployments.
To view a YouTube video with me visually illustrating this approach in greater detail, click here.