Detect change in performance of new rasa model
I have a rasa nlu model running to detect intents and entities. Whenever a new model is loaded by rasa after training, I want to know how good or bad it is from previous model.
What I am planning is , whenever a model is loaded first time, I can test it against some gold standard data and save the output metrics to compare with previous saved metrics.
Questions:
- Is there any event which is loaded first and one time only when a model is loaded by rasa nlu? I can write test and save metric logic in that event.
- Do you have a better alternate suggestion to achieve this purpose?
Topic rasa-nlu performance
Category Data Science