switch_to_blog() and wp_nav_menu() not functioning properly

I have an MU plugin active on the network with this small function to query menu 4 from the main site in my network. On the main site in the network (1) things work as expected.

On the sub sites in the network (2,3,4) no menu gets displayed (since I have the fallback set to false, and no menu is found).

I'm wondering if I'm doing something wrong here, or if something larger is going wrong.

/**
*   Multisite menu
*/
function wp_multisite_nav_menu() {
    global $blog_id;
    $args =  array(
        'menu'            = 4,
        'fallback_cb' = false,
        'menu_class'      = 'main-nav',
        'depth'           = 0,
    );
    if ( ! is_multisite() || 1 == $blog_id ) {
        wp_nav_menu( $args );
        return;
    }
    // switch to blog, and query
    switch_to_blog( 1 );
    wp_nav_menu( $args );  
    // switch back to original blog
    restore_current_blog();
}

I can see that switch_to_blog is properly switching, as I can retrieve data from the main site using get_bloginfo( 'name' ); and get_the_title( 123 ); without any issues.

Alternatively, if this isn't the best way to go about it - I'm up for additional solutions.