The effect of removing pooling layers in the model's accuracy

I know that removing pooling layers will lead to an increase in dimensionality and subsequently, make the training to be more time-consuming. But I'm wondering if it worth it to remove pooling layers or not? does it lead to a higher accuracy?

Have you ever seen any relevant papers, articles, etc. about this issue? (I've searched and couldn't find much things except for this paper)

I even don't know how many pooling layers should I remove.