YouTube API 403 error when website restriction set

I am loading the latest videos from a YouTube channel to WordPress, using the YouTube API:

function load_feed() {
    $url = add_query_arg(
        array(
            'part' = 'snippet',
            'channelId' = '[YouTube channel ID]',
            'maxResults' = 3,
            'order' = 'date',
            'type' = 'video',
            'key' = '[YouTube API key]'
        ),
        'https://www.googleapis.com/youtube/v3/search'
    );

    $response = wp_remote_get(esc_url_raw($url));

    return json_decode(wp_remote_retrieve_body($response), true);
}

Everything works well when in the API settings on Google Developers I have:

Application restrictions: None

But when I set:

Application restrictions: HTTP referrers (web sites)

Website restrictions: www.mysite.com/*

I get the following response from the API:

{
    error: {
        code: 403,
        message: Requests from referer https://www.googleapis.com/youtube/v3/search?part=snippetchannelId=[Channel ID]maxResults=3order=datetype=videokey=[API key] are blocked.,
        errors: [
            {
                message: Requests from referer https://www.googleapis.com/youtube/v3/search?part=snippetchannelId=[Channel ID]maxResults=3order=datetype=videokey=[API key] are blocked.,
                domain: global,
                reason: forbidden
            }
        ],
        status: PERMISSION_DENIED
    }
}

It looks as though the API detects the request was made from googleapis.com and not from mysite.com, where I have the WP installation and am running the above code.

Why is this error occurring and what can I do to fix it?