How to load several numpys in batch generator from directory
I have a deep learning model which has to be feed with a huge amount of data (200k of 256x256 images), so it runs out of memory. I have divided my data in several numpy arrays that are in an specified directory, but I do not know exactly how to create the bacth generator from diferent numpy arrays so all the numpy work as X_train,and then it is load to the model in batches.
I tried coding the following lines obtain from other responses, but it does not work, the kernel starts working but it never ends or does something, it is busy but that is all.
def load_data(ids):
X = []
Y = []
for i in ids:
# read one or more samples from your storage, do pre-processing, etc.
# for example:
part1y = np.load('set1target.npy')
part2y = np.load('set2target.npy')
part1x = np.load('set1.npy')
part2x = np.load('set2.npy')
X_train=np.concatenate((part1x, part2x))
Y_train=np.concatenate((part1y, part2y))
return np.array(X_train), np.array(Y_train)
Then:
BATCH_SIZE=32
def batch_generator(ids, batch_size = BATCH_SIZE):
batch=[]
while True:
np.random.shuffle(ids)
for i in ids:
batch.append(i)
if len(batch)==batch_size:
yield load_data(batch)
batch=[]
train_generator = batch_generator([0,1], batch_size = bs)
history= model.fit_generator(train_generator ,epochs = 2,steps_per_epoch=48160//bs ,
validation_data=(X_test, Y_test ))
Any recommendation?Thank you!
Topic numpy keras memory deep-learning dataset
Category Data Science