Transfer learning on siamese network with limited data

This may be a silly example, but it should be similar enough to my true research question without giving specifics. Let's say I have a pretrained Siamese neural network that tells you similarity between image pairs of dogs of different breeds.

Now I want to use transfer learning on the last layer to ask a slightly different question. Now the dogs may be wearing hats, and I want to consider a regular picture of a Husky and a Husky wearing a top hat to be similar. Meanwhile, I want a regular Husky and a Husky wearing a beanie to be different. However, in my training set, I only have a Husky wearing a top hat (a positive pair, if you will). I do have a picture of a corgi wearing a beanie (a negative pair), but no pictures of top hat corgis.

Is it reasonable to assume that even if each dog breed does not have both positive and negative (top hat and beanie) examples, that the neural network will learn the general idea of the hats and generalize that across all breeds? Is it common for an input into a Siamese neural network to have no positive pairs, and it can only be defined as dissimilar from a negative example?

Also, are there any papers out there that have tried something similar, with different modifications of the input being assigned different similarities?