Vectorize One line text data

Question

Vectorize One line text data

Payal Bhatia

2022年6月3日 03:08

How to vectorize one-line text data? I have used tf-idf including bigrams and trigrams but I am not able to get good results. I have purchase order descriptions which are one-liners and I need to classify. It is a multi-class imbalanced data and I have a small dataset to train around 700 PO descriptions. The number of classes is 7 and the class distribution is similar to exponential. One class is dominating. My take is that TF IDF should not work since the term frequency and the IDF frequency will be very small. Also, can we make some user-defined functions to create vectors? If yes, what should be it?

Please suggest some alternative approaches as well.

Topic tfidf nlp

Category Data Science

tehem · Accepted Answer · 2020年8月24日 15:23

1

tehem answered at 2020年8月24日 15:23

You could alternatively use a pretrained embedder like word2vec or glove to vectorize your data into fixed length vectors.

Erwan · Accepted Answer · 2019年11月22日 12:21

Using bigrams and trigrams is likely to generate a high number of features, but with a small dataset the traditional approach would be to reduce the number of features. You could start by removing the least frequent words/n-grams (e.g. less than 3 occurrences), and/or use feature selection with InfoGain. It might not be very accurate but at least you avoid overfitting.

Piotr Rarus · Accepted Answer · 2019年11月22日 11:12

1

Piotr Rarus answered at 2019年11月22日 11:12

Check my answer to this question. Nowadays there're many pretrained embedders to choose from. They'll give you fixed-size numerical vector of features. You don't even have to go DNN way, xgboost will work just fine.

Vectorize One line text data

About