Working with three types of data: numeric (integer, floats), images, and text for prediction

So I have three types of data (in title) and am wondering how I can combine the data. The target is numeric (price).

My idea is to perform feature extraction on both the images and text, which would result in a 1 dim row vector of size n. So this would produce n features. After this, combine all the data and normalize. The model would be trained on this combined dataset.

I have really only worked with one data type or the other, so I am unsure if this is the right approach. I haven't found many relevant resources for this type of problem, but let me know if I am missing something.

Topic real-ml-usecase image-preprocessing prediction text preprocessing

Category Data Science

1
answered at

Based on the assumption that you can correctly assign a text, image and numeric features to each instance of your dataset, I would try as a first approximation:

  • numeric features: as is (applying usual preprocessing on numeric features if necessary)
  • text: treat, at the beginning, the texts in your dataset as an independent set, build your bag of words, and find the most frequent terms (uni or bi-grams) so that you can add the frequency of such terms as new features (adding these ones to the numeric features of point 1)
  • images: if you have in advance an idea of which kind of objects you will have in those images, you can build an image classifier (or even an object recognition model), so the output of this model would be the value of the additional features (categorical values) of which type of image/objects your image is (here an introductory source in case it's needed).

In the end, you would end up with something like:

id numeric_1 numeric_2 numeric_3 term_1_freq term_2_freq term_3_freq term_4_freq car cat beach dog
1 3 0.4 20 0 0.3 0 0.6 0 0 1 0
2 3 0.2 25 1 0.2 0.05 0.7 0 0 0 1
3 3 0.4 10 0 0.2 0 0.6 0 1 1 0

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.