How many images can be trained in Google Colab?

I am using the ResNet50 pretrained model to train my images using TensorFlow. I have 70k images and upgraded to Google Colab Pro, but still I am facing a memory error. So how many images I can train in Google Colab? And how much RAM is needed for 70k images?

This is how I labeled and loaded images from the drive.

labels = []

imagePaths_generater = paths.list_images(Config.DATASET_PATH)
imagePaths = []
for item in imagePaths_generater:
  imagePaths.append(item)

for imagePath in imagePaths:
  label = imagePath.split(os.path.sep)[-2].split(_)
  #print(label)
  image = load_img(imagePath, target_size=(224, 224))
  image = img_to_array(image)
  image = preprocess_input(image)
  data.append(image)
  labels.append(label)
 

data = np.array(data, dtype=float)
labels = np.array(labels)```

Topic google-cloud colab keras tensorflow predictive-modeling

Category Data Science

1
answered at

The real problem is that you should not try to fit all your images in memory. Instead, you should small groups of images, normally called "minibatches", which can fit in the GPU/CPU memory.

For that, tensorflow offers the function tf.keras.preprocessing.image_dataset_from_directory that loads images from a directory. I suggest you take a look at Tensorflow's tutorial for image loading, which guides you to the image loading process, as well as the training of a model with those images.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.