A lot of guys where I live (Florida, USA), use an immersion or counterflow chiller to get the beer down to about 80-90F (which is about as low as it will go with 70F+ groundwater), and then rack to a carboy and put in a fridge for 6-8 hours to hit pitching temps. This makes me nervous, but they seem to have good success with the plan.
I couldn't tell from your question if you want to remove the chiller entirely, or if you just don't want to run it endlessly. But its definitely not a good idea to rely on a refrigerator to take a 5 gallon batch down from boil temps to pitching temps without any extra cooling help from a chiller, ice bath, etc.
This is because a carboy full of hot wort is incredibly well insulated and dense, and will take many, many hours to cool down to pitching temps. During that time, your aroma and flavor hops will continue to "cook" and become more bitter.
Also, once the wort drops beneath about 140F, its going to be spending many, many hours inside the temperature zone thats PERFECT for bacteria before you pitch. This is a serious risk to your beer.
I would suggest using the chiller to knock the temp down lower than 100F at least, and then using an ice bath to get the carboy to pitching temps, or putting it into your fridge for 6-8 hours before pitching, tops. I do the former, and 20lbs of ice will take a carboy from 90F-ish to 65F with some swirling in about 20min.