How to compare alternative models based on a positive measure of fitness

I am still trying to define this question precisely, so please indicate any feedback and I will edit my question.

I have $M$ alternative models that need to be compared. The only measure that needs to be taken into account is a positive value $n$ that indicates how many independent sub-items (independent and with same weight on the total value) are supported by the data. The total number of items $N$ is not fixed, but it needs to be taken into account.

A basic approach might be the following: We have a default model $m$ that is selected either if:

  • it has the largest $n$ of supported items AND $p\%$ (of total items $N$) more than the second best model
  • no model among the $M$ alternatives has the largest number $n$ of supported items AND the $p\%$ advantage

Otherwise, the best model (largest $n$ and $p$) is selected.

How can this approach be improved?

Topic counts

Category Data Science

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.