Splitting and training multiple datasets at the same time

I've got 15 different datasets at about 10GB each. Each dataset comes with a binary 2D ground truth (10486147ish, 1) that I pull from it. I'm trying to figure out how to load each dataset, split them all with scikitlearn's train_test_split, then iterate over all 15 datasets per epoch. Under normal circumstances, the datasets would be shuffled as well, but I cannot figure out how to even do that since the data is too large to load all at once to shuffle (as such shuffling them is on the back burner for now).

Here's what my code looks like for one dataset.

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import sequence
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection imporrt train_test_split



arr = np.load ('source/dir/dataset1.npy', allow_pickle = True, fix_imports = True)
arr[arr == -inf = -9999]
rehape = arr.reshape(((arr.shape[0])*(arr.shape[1])), (arr.shape[2]))
drop = reshape[~np.all(reshape == -9999, axis = 1)]
#additional work done with -9999 here
truth = drop[:,46]
data = drop[:,0:45]

#callbacks deleted in code sample

encoder = LabelEncoder()
encoder.fit(truth)
Y = encoder.transform(truth)
Y = Y.reshape(10486147, 1)
X = data.reshape(10486147, 45, 3)

seed = 7
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = seed)

model = sequential()
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3)))
model.add(Dense(1, kernel_initializer = 'normal', activation = 'sigmoid'))

model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 500, batch_size = 1000, callbacks = [deleted callbacks])

So that makes sense for one dataset, but as I've said before I've got 15 datasets to iterate through, and I don't think retraining on new data is the right step. Is there a way through dataset1.npy through dataset15.npy while properly splitting the ground truth as well.

Any suggestions?

Topic stacked-lstm keras tensorflow scikit-learn python

Category Data Science

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.