But in most cases your evaluation metric itself is different. Hard to beat that for highly imbalanced problems. If your evaluation metric really is the Hamming loss, then you might actually be better off predicting just the majority for each binary problem. I take it you have reduced the problem into 100, binary classification problems? That would be a standard way to do things in the multilabel setting. You can evaluate your predictions after the fact using Hamming loss, but it may be informative to compare your results to the Hamming loss of always predicting 0. Use -loss_function="logistic" or -loss_function="hinge" for binary classification problems in vowpal wabbit. Rows table(ifelse(predict(lin_mod, type='response')>0.50, 1, 0)) Here's a demonstration of the same "problem" with a linear regression model in R: set.seed(42) Vowpal Wabbit can't do magic: if you ask it to minimize squared error loss, it will indeed minimize squared error loss, as will any other regression program. Think about the loss function: if 99% of observations are zero, predicting 0 in all cases gives a squared error of 0.01. I've tried ratios from 1:1, to 1:25, with performance degrading abruptly at the 1:2 ratio.Īny linear model will handle class imbalance "very badly" if you force it to use squared loss for a binary classification problem. vw handles imbalance very badly, at least for highly-dimensional, sparsely-populated text features. I'm wondering if anyone else have any suggestions.Įdit: I finally had the chance to test out class-imbalance, specifically for vw. I was thinking of using cost-sensitive one-against-all classification to try to balance the classes, but reducing multi-label to multi-class is unfeasible since there exists 2^100 label combinations. which is one way of minimizing loss, I guess. I'm using the squared loss because it closely resembles the Hamming Loss.Īfter training, when testing on the same training set, I've noticed that all examples were predicted with the '0' label. I'm using the default options provided by VW (which I think is online SGD, with the squared loss function). To: 1 |healthy bananas oranges jack fruit I've reduced this problem down to binary classification for Vowpal Wabbit by taking advantage of namespaces, e.g.įrom: healthy fruit | bananas oranges jack fruitĮvil monkey | bipedal organism family guy ![]() As mentioned in the title, each example has at least one label, and there are only 100 possible unique labels. I have a set of around 300k text examples. However, I have a multi-label problem, so how would you deal with it in this case? I've seen a few questions on class imbalance in a multiclass setting.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |