TF-IDF to find technical terms
I have some sentences and I want to see whether or not they contain words that are technical terms. I was thinking of working with Wikipedia texts: finding the most common words in a certain article, and if those words are rare among most of the other articles, then they are most likely technical terms. Does this make sense?
I tried it using 3 specialized texts from my computer, from different areas, and the results were quite bad. I got lots of numbers and common verbs and nouns (though let's say I can filter some of them in advance). And even so, I don't know how to apply this to the short, separate sentences I want to work with.
Any advice or hint is very much appreciated!
Topic bag-of-words tfidf word-embeddings nlp machine-learning
Category Data Science