Display all categories but only if they have posts in them that have a specific tag assigned

I have a WooCommerce site and am working on the product tag pages. I want to display every product that is assigned to that tag, and break them up into their individual categories (this bit is working).

The only issue I am having is it is displaying every single category, even if 0 products in the current tag have that assigned.

?php $tag_id = get_queried_object(); ?

  div id=product-list class=product-list

    ?php // Get all products inside the current tag ?
    ?php $collectionProducts = array(
      'post_type' = 'product',
      'order' = 'ASC',
      'hide_empty' = true,
      'parent' = 0,
      'taxonomy' = 'product_tag',
      'field' = 'slug',
      'terms' = $tag_id-slug
    );

    // Get list of categories, but only display them if they have products with the current tag assigned 
    //(THIS BIT ISN'T WORKING)

    $child_terms = get_categories( array(
      'parent' = 0,
      'hide_empty' = true,
      'taxonomy' = 'product_cat',
      'tax_query' = array (
        'taxonomy' = 'product_tag',
        'field' = 'slug',
        'terms' = $tag_id-slug,
        'operator' = 'IN'
      )
    ));
    ?

      ?php // List products, broken down by category ?
      ?php foreach($child_terms as $child_term) { 
        $shortcode = do_shortcode('[products tag=' . $tag_id-slug .' category=' . $child_term-slug . ' per_page=-1 columns=3 orderby=date]');
      ?

        div class=product-list-single-category id=?php echo $child_term-slug; ?
          div class=title
            h3?php echo $child_term-name; ?/h3
          /div
          ?php echo $shortcode; ?
        /div
      ?php } ?

    /div