Blog

Learn About Our Meetup

4500+ Members

[D] Which performance metric should be used for cases of *minor* class imbalance?

There has been a lot of discussion with regards to choosing an appropriate performance metric to use for model training and evaluation for classification problems with a moderate to large amount of class imbalance present (e.g. 1, 2, 3, 4, 5).

In these cases, it is generally suggest that one uses a metric like cohen’s kappa or PR AUC in place of accuracy or AUROC, perhaps along with up-sampling or SMOTE.

What are people’s thoughts on cases where there is only a minor class imbalance present in the data? For example, something like a 3:1, 2:1, or even 1.5:1 ratio of major to minor class members? Is it still beneficial (and what would be the cost?) or using a metric geared at addressing larger imbalances in these cases?

Also, somewhat tangential, but are there ever any scenarios where you might want to use one metric for an outer CV loop / model performance evaluation, but a different metric for inner CV (e.g. feature selection / hyperparameter optimization)?

submitted by /u/user381
[link] [comments]

Next Meetup

 

Days
:
Hours
:
Minutes
:
Seconds

 

Plug yourself into AI and don't miss a beat

 


Toronto AI is a social and collaborative hub to unite AI innovators of Toronto and surrounding areas. We explore AI technologies in digital art and music, healthcare, marketing, fintech, vr, robotics and more. Toronto AI was founded by Dave MacDonald and Patrick O'Mara.