SageMaker - mini_batch_size what is it and why can't I get it higher than 5?

I'm following this tutorial but I keep getting the error:

The number of input images must be bigger or equal to the mini_batch_size.

I've tried a series of different permutations of data and hyperparameters but I keep getting this error.

# The algorithm supports multiple network depth (number of layers). They are 18, 34, 50, 101, 152 and 200
# For this training, we will use 18 layers
num_layers = 18
# we need to specify the input image shape for the training data
image_shape = 3,640,640
# specify the number of output classes
num_classes = 2
# we also need to specify the number of training samples in the training set
num_training_samples = num_classes * 500
# batch size for training
mini_batch_size = 8
# number of epochs
epochs = 1
# learning rate
learning_rate = 0.01
# report top_5 accuracy
top_k = 5
# resize image before training
resize = 640
# period to store model parameters (in number of epochs), in this case, we will save parameters from epoch 2, 4, and 6
checkpoint_frequency = 2
# Since we are using transfer learning, we set use_pretrained_model to 1 so that weights can be
# initialized with pre-trained weights
use_pretrained_model = 0

5 seems to be the magic number. I'm struggling to understand the issue and why the error message does not seem to match the problem. The default is 30, and the tutorial says 128.

At the moment, the number of classes is 2, and epochs is 1, because I'm trying to isolate this issue.


Addition information:

Using a data augmentator to generate several validation images has allowed me to go higher: to 10 (so far). Perhaps the validation images are also related?

More info from Cloudfront

[07/24/2021 12:59:04 ERROR 140201375991616] [12:59:04]
/opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.3.x_ecl_Cuda_10.1.x.6753.0/AL2_x86_64/generic-flavor/src/src/io/iter_image_recordio_2.cc:318:
Check failed: !overflow number of input images must be bigger than the batch size

Topic sagemaker image-classification

Category Data Science


How many images do you have as input? Maybe you haven't load the images as expected (seems about 30k). Check if the download process has been done properly by getting the images'quantity.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.