How to assign costs to the confusion matrix

Question

How to assign costs to the confusion matrix

PicaR

2022年6月2日 07:45

I am trying to assign costs to the confusion matrix. That is, in my problem, a FP does not have the same cost as a FN, so I want to assign to these cases a cost x so that the algorithm learns based on those costs.

I will explain my case a little more with an example:

When we want to detect credit card fraud, it does not have the same cost to predict that it is not fraud when in fact it is than the other way around. In the first case, the cost would be much higher.

What I wanted to know is if there is a library in R in which I can assign costs to these wrong decisions (i.e. give a cost to each possible value of the confusion matrix) or if there is an algorithm that learns based on a cost/benefit matrix.

I could also use some way to implement this without the use of a library.

Thank you very much.

Topic cost-function evaluation r machine-learning

Category Data Science

German C M · Accepted Answer · 2022年6月2日 07:45

In this case, you can consider a payoff matrix to use average profit as your evaluation metric. It is important to mind the difference between the cost function (used in the learning process of the algorithm during training, which must be also differentiable with gradient-based optimization) and the performance metric (which is what I think you should consider in this case, your average profit for instance).

In case you have the following confusion matrix:

you can assign a benefit/loss per quadrant of your matrix:

and in this case, your evaluation metric (to select your best model):

$ Average Profit = \frac{(TN number*TN benefit + TP number*TP benefit - FN number*FN losses - FP number*FP losses)}{Number Of Predictions}$

This metric can also be used to select the optimal threshold of your model:

In case you want to feed you training algorithm to an XGB in R for instance, have a look at the documentation, where you have an argument called feval, to include your custom evaluation metric function (documentation link):

feval --> customized evaluation function. Returns list(metric='metric-name',value='metric-value') with given prediction and dtrain

Nicolas Martin · Accepted Answer · 2022年6月2日 06:56

What defines the penalty value of each FP or FN? It is the final cost due to those decisions compared to the TP and TN.

Having a false positive of 1M dollars of credit card fraud, has not the same importance than having a false positive of 10 dollars.

That's why the cost function shall be directly linked to the amount of the frauds, and hence the objective function of your algorithm.

Therefore you can multiply each decision result (TP,TN,FP,FN) by the amount of each payment: Remember that you aim to improve the whole model, including the right result, not just the bad ones.

Consequently, the objective function is to try to find the maximum of the sum of all payments, taking the good payments (TP,TN) as positive, divided by all the amounts (TP,TN,FP,FN).

For exemple:

TP= [2500, 200, 60]
TN= [300,5000]
FP= [1500,30,600]
FN= [300, 200]

Score = (2500+200+60+300+5000) / (2500+200+60+300+5000 + 1500 + 30 +600 +300 +200)

Score = 0.75

Which means that 75% of the total money has been correctly assigned, and the aim is to reach 100% as close as possible.

If you want to take into account the quantity of frauds as well, you just have to add and artificial penalty (let's say 100 dollars) to each transaction, so that you can also value the quantity of frauds, instead of the final raw cost result.

Score = ( 2600+300+160+400+5100) / (2600+300+160+400+5100 + 1600 + 130 +700 +400 +300)

Score = 0.73

This option is usefull when trying to reduce the importance of big transactions, in favor of the small ones (= considering more the small customers).

You can find more information about cost sensitive cost function here: https://towardsdatascience.com/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9

How to assign costs to the confusion matrix

About