How to compare alternative models based on a positive measure of fitness
I am still trying to define this question precisely, so please indicate any feedback and I will edit my question.
I have $M$ alternative models that need to be compared. The only measure that needs to be taken into account is a positive value $n$ that indicates how many independent sub-items (independent and with same weight on the total value) are supported by the data. The total number of items $N$ is not fixed, but it needs to be taken into account.
A basic approach might be the following: We have a default model $m$ that is selected either if:
- it has the largest $n$ of supported items AND $p\%$ (of total items $N$) more than the second best model
- no model among the $M$ alternatives has the largest number $n$ of supported items AND the $p\%$ advantage
Otherwise, the best model (largest $n$ and $p$) is selected.
How can this approach be improved?
Topic counts
Category Data Science