How can i make a process using several small keras models execute faster?

I have a keras model that is being trained using a genetic algorithm. This means that the performance of the model needs to be tested many times. To test the model performance the model is used as a control system in a simulation. In this simulation the model is evaluated in a tight loop so the speed of the forward pass of the model is very important.

I am using a very small model. I used to be a dense network of 60-80 parameters but the current version being used is a recurrent network of ~350 parameters. Switching to the recurrent network made the simulation process 10 times slower which poses a problem as the genetic algorithm went from a day to ten days to get a decent amount of generations in.

Worth mentioning is also that I maintain a population of 100 individuals meaning 100 keras models are being tested in parallell in the same simulation. I have not been able to do this on multiple cores as keras does not allow easy multiprocessing.

I tried to boost performance by enabling GPU acceleration but this lead to a 33% slowdown. I'm assuming this is as the data being generated by the simulation have to be put moved to the GPU-memory as mentioned in this post, but I'm not sure.

What could I do to speed up execution of my keras model?

Here is the part of the code (inside a loop) where the keras model is being called:

def apply(self, agents, agents_data, params):
    turning_idx = np.nonzero(self.levy_timer = 0)[0]
    for i in turning_idx:
        alpha = agents_data.network_containers[i]
           .get_next_alpha(agents_data.memory[i])

    return agents, agents_data


def get_next_alpha(self, compressed_memory):
    compressed_memory = np.reshape(compressed_memory, 
       (1, len(compressed_memory)//net_input_size, net_input_size))
    return self.network(compressed_memory) + 1

My graphics card is: NVIDIA GeForce GTX 1660 Ti. My processor is: Intel Core i7-9750 CPU @ 2.60GHz.

Training on GPU is normally faster for regular backpropagation training.