Clarification about Octave data size limit

I'm just starting to work on a relatively large dataset after ML course in Coursera. Trying to work on https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD. Got an accuracy of 5.2 in training and test set with linear regression using gradient descent in octave.

I tried adding all possible quadratic features (515345 instances and 4275 features), but the code just won't stop executing in my HP Pavilion g6 2320tx, with 4GB RAM in Ubuntu 14.04.

Is this beyond the data size capacity of Octave ?

Topic octave

Category Data Science

1
answered at

You have about 4GB of RAM on your machine and Octave is an in memory application.

If you want to work with 515345 instances and 4275 features, assuming that you are using double precision (i.e. 8 bytes), you would need a memory of 515345*4275*8/1000000/1024 bytes ~ 17.6 GB. Even if you were using 4 bytes for each data point, you would require at least 9 GB for the computation to go through.

This issue might not be the Octave memory restriction in this case. See here for further details on Octave's memory usage.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.