Image similarity: Similarity of mixed vector

In order to identify the similarity between images (products) I want to use a neural network approach similar to TiefVision. This pre-trained neural network is basically translating the images into a feature vectors and then creating a similarity measure between the images using a distance measure between the vectors. To make it more tangible have a look at a 2D visual representation below.

I want to take it one step further: When a single user "likes" multiple images, I want to average their feature vectors. This will result in a new vector, which I want to use to calculate recommendations. My assumption is that images close to the combined feature vector will possess similar features towards all "liked" images together.

Now I wonder: Is my thinking flawed - maybe because averaging the vectors will simply lead to entirely different images, or will it actually produce images with similar features?

_{(source: indico.io)}