Training Inception V3 based model using Keras with Tensorflow Backend

I am currently training a few custom models that require about 12Gb GPU memory at the most. My setup has about 96Gb of GPU memory and python/Jupyter still manages to hog up all the gpu memory to the point that I get the Resource exhausted error thrown at me. I am stuck at this peculiar issue for a while and hence any help will be appreciated.

Now, when loading a vgg based model similar to this:

from keras.applications.vgg16 import VGG16
from keras.models import Model

import keras

from keras.models import Model, Sequential
from keras.models import Input
input_shape = (512, 512, 3)
base_model = VGG16(input_shape=input_shape, weights=None, include_top=False)

pixel_branch = base_model.output
pixel_branch = Flatten()(pixel_branch)

new_model = Model(inputs=base_model.input, outputs=pixel_branch)

text_branch = Sequential()
text_branch.add(Dense(32, input_shape=(1,), activation='relu'))

# merged = Merge([new_model, text_branch], mode='concat')
merged = keras.layers.concatenate([new_model.output, text_branch.output])

age = Dense(1000, activation='relu')(merged)
age = Dense(1000, activation='relu')(age)
age = Dense(1)(age)

# show model
# model.summary()
model = Model(inputs=[base_model.input, text_branch.input], outputs=age)

When I just run a jupyter cell with this code and monitor the GPU usage using nvidia-smi, it is 0% . However, I replace the code in the above Jupyter cell with the following:

from keras.applications.inception_v3 import InceptionV3
from keras.models import Model
import keras
from keras.models import Model
from keras.models import Sequential

from keras.models import Input
input_shape = (512, 512, 3)
base_model = InceptionV3(input_shape=input_shape, weights=None, include_top=False)

pixel_branch = base_model.output
pixel_branch = Flatten()(pixel_branch)

new_model = Model(inputs=base_model.input, outputs=pixel_branch)

text_branch = Sequential()
text_branch.add(Dense(32, input_shape=(1,), activation='relu'))

# merged = Merge([new_model, text_branch], mode='concat')
merged = keras.layers.concatenate([new_model.output, text_branch.output])

age = Dense(1000, activation='relu')(merged)
age = Dense(1000, activation='relu')(age)
age = Dense(1)(age)

# show model
# model.summary()
model = Model(inputs=[base_model.input, text_branch.input], outputs=age)

The GPU usage goes crazy and suddenly almost all the memory is over in all the GPUs even before I do model.compile() or model.fit() in Keras!

I have tried both allow_growth and per_process_gpu_memory_fraction in Tensorflow as well. I still get the resource exhausted error the moment I run model.fit when using the Inception based model. Please note that I do not think this is a GPU memory error as I have about 96GB of GPU memory using an instance with 8 Tesla K80s .

Also kindly note that my batch size is 2.