Few shot learning and object detector

I have a dataset with a lot of classes (~10000+) but few examples by classes (~15-). I want to classify these classes, but there are some specificities. My examples provide from a video stream. Currently, I use a detector (like YOLO) to do object detection and then I apply a siamese/triplet network to obtain embeddings. Finally, I classify by using a Kneighbor classifier with one neighbor.

The real specificity, is that I want to detect when a picture provided by my detector is in a sublist of my classes. For example, on day 1, I have a list of 100 classes, and my goal is to know if the object (provided by the detector) is in my 100 classes, and which one. I don't know how to have the best use of this information. Do you have any idea about it?

For now, I precalculate all of my embeddings, then for each object, I have only a forward pass to do and a computation of the distance. Maybe I can regroup all my other embeddings as an other class and keep 101 classes (my 100 classes of my sublist and the other class) but I don't know if it is really efficient.

To conclude, have you some tricks to obtain a higher score with siamese/triplet networks? I use ResNet with pretrained weights on imagenet, but there may exist some technics to score better.