How to perform crossover between models of different sizes in deep genetic algorithms?

I'm working building a genetic algorithm that will learn to play snake. I've worked out how to add/remove layers and neurons in the model, allowing the model's size to change through mutation. But this brings the problem of how to perform crossover when models will have different numbers/sizes of layers.

If it was just the size of each layer that could change, it would be one thing. I can handle that. But I just can't figure out how to do crossover between a model with, say 5 layers and one with 2.