Calculating confidence score in NER
I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end-user. Now the ask is end-user needs a confidence score along with the extracted entity. For example, the given text is: XYZ Bank India Limited is a good place to invest your money - Our model is detecting XYZ Bank as an Org, but India as a Location (which is wrong - the whole XYZ Bank India Limited is the name of the organization). Our model also gives a probability score for each token it classifies. But the end-user wants to know the confidence of the model that it did not mistake to detect the subsequent tokens as the parts of the organization name.
Question is - how can we efficiently measure that in a given sequence our model is detecting a certain sub-sequence as an Organization name (or a Location or something else) correctly or not? How can we say that it did not miss out on any subsequent or preceding token which actually a part of the named entity (like it missed India Limited in the above example)?
Topic sequence-to-sequence named-entity-recognition deep-learning nlp python
Category Data Science