Learning similarity of representations

I am interested in a framework for learning the similarity of different input representations based on some common context. I have looked into word2vec, SVD and other recommender systems, which does more or less what I want. I want to know if anyone here has any experience or resources on a more generalized version of this, where I am able to feed in representations on different objects, and learn how similar they are.

For example: Say we have some customers we are sending different advertisements to, and I would like to create a system to map offers to customers. I am thinking in the lines of creating a customer representation, and a representation of the offers, and feeding them in parallel to a neural network that has a label of whether they acted on the advertisement or not. The idea is that I should be able to locate the best offer for any customer given these representations.

I have looked into siamese networks and word2vec, both are close to what I want. The problem differs slightly in that for the siamese networks, there are identical parallel networks, which I don't want because my inputs are not equivalent. Word2vec type methodology is also close, but I would want a model to process the inputs on "both sides". A combination of the two, is kind of what I am looking for.

If anyone has any resources on a similar problem statement, I would be very interested in it.

Thanks