Don't know how to preprocess my dataset for image classification

Question

Don't know how to preprocess my dataset for image classification

aguythatneedshelp

2021年4月17日 14:59

I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy.

I believe the issue might be with my data preprocessing. My dataset directory contains a Training and Test folder but no validation folder (I have to split the dataset myself) and they are layed out like this:

Training
├── class0
│   ├── image1
│   ├── ....
│   └── image20
│
├── ....
│   ├── image1
│   ├── ....
│   └── image20
│
└── class9
    ├── image1
    ├── ....
    └── image20

Here is my AlexNet model, the code is below:

model = tf.keras.models.Sequential([
    #Conv_1          #original model was built for input shape of 224X224
    tf.keras.layers.Conv2D(96, (11,11),strides=4, padding='valid', activation='relu', input_shape=(224, 224, 3)),
    # Pooling_1
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    # Batch Normalisation_1
    tf.keras.layers.BatchNormalization(),
    # Conv_2
    tf.keras.layers.Conv2D(256, (11,11),strides=1, padding='valid', activation='relu'),
    # Pooling_2
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    #Batch Normalisation_2
    tf.keras.layers.BatchNormalization(),
    # Conv_3
    tf.keras.layers.Conv2D(384, (3,3),strides=1, padding='valid', activation='relu'),
    # Batch Normalisation_3
    tf.keras.layers.BatchNormalization(),
    # Conv_4
    tf.keras.layers.Conv2D(384, (3,3),strides=1, padding='valid', activation='relu'),
    # Batch Normalisation_3
    tf.keras.layers.BatchNormalization(),
    #conv_5
    tf.keras.layers.Conv2D(256, (3,3),strides=1, padding='valid', activation='relu'),
    #pooling_3
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    #Batch Normalization_4
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Flatten(),
    #Dense layer_1
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    #Dense layer_2
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    #Dense layer_3
    tf.keras.layers.Dense(1000, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(1, activation='sigmoid')
    ])

As for what I did to preprocess my data:

train_dir = '/content/gdrive/My Drive/x/Training'
test_dir = '/content/gdrive/My Drive/x/Test'

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest',
      )

test_datagen = ImageDataGenerator(rescale=1./255, validation_split=0.4)


train_generator = train_datagen.flow_from_directory(
        train_dir,  
        target_size=(224, 224),  
        batch_size=20,
        class_mode='binary')


validation_generator = test_datagen.flow_from_directory(
        test_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary',
        subset='validation')

model.compile(loss='binary_crossentropy',
              optimizer=RMSprop(lr=1e-4),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,  
      epochs=10,
      validation_data=validation_generator,
      validation_steps=10,  
      #callbacks = [checkpoint],
      verbose=2)

Any and all help will be immensely appreciated! Thank you to anyone who may reply.

Topic alex-net cnn keras deep-learning python

Category Data Science

aguythatneedshelp · Accepted Answer · 2021年4月17日 14:59

Since someone said that if I solved my answer I should answer my own question: What I did was:

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=20,
        # Since we use binary_crossentropy loss, we need binary labels
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=20,
        class_mode='categorical')

model.compile(loss='categorical_crossentropy', ## this code has been changed as categorical from the original code
              optimizer=optimizers.RMSprop(lr=lr_first), # 2e-5
              metrics=['acc'])

Basically replaced the binary labels with categorical.

Peter · Accepted Answer · 2020年5月31日 10:29

Find a working Keras setup for a pretrained model with custom data generator here.

Note that validation data should not be augmented!

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

# Validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=20,
        # Since we use binary_crossentropy loss, we need binary labels
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=20,
        class_mode='binary')

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=lr_first), # 2e-5
              metrics=['acc'])

Don't know how to preprocess my dataset for image classification

About