ML model shooting up prediction probabilities

I have a scikit-learn logistic regression binary classifier and tried training it on my dataset. My model does extremely well at a threshold of 0.95 instead of 0.5 and all my predictions on example cases are above 0.8 for both classes. I cannot figure out why my machine learning model is shooting up predictions so much. I would appreciate some potential work arounds for this.