admin-ajax.php calls fail if referrer is sent with 500

calling admin-ajax.php via jquery ajax calls were failing from my site but postman calls to the same url were successfull.

After some looking at headers it seems that the referer being set is the difference that calls the failure (500).

What would possibly cause admin-ajax.php to care about whether or not a referer is being sent and fail if it is?

The error logs just say that memory is running out but why would the referer header change how much memory is being used?