Tableau: Trying to determine the category of one table based on the dynamic aggregate of another table for a Tableau Dashboard

I have one table that contains unique rows for all my quote requests:

Quote_ID
1234
1235
1236
1237
1238

in a second table that I've joined (1-0*) with a relationship, I have referrals. These referrals represent reasons why the quote should be referred to an expert, but could represent any other attribute for the sake of the problem.

Every referral has a key to the Quote_ID, a unique Referral_ID and a name:

Quote_ID    Referral_ID     Referral_name
12345       1               too many X
12346       2               too many X
12346       3               not enough Y
12348       4               price over Z
12348       5               too many X

Essentially what I need to do is visualise quote counts based on the number of referrals they have:

Where I'm getting stuck is making this dynamic for my dashboard. I want the user to be able to filter out different referrals to simulate different results. Filtering out the 'too many X' referral should lead to updated categorisation for the quote:

When trying to make this happen in tableau, I can't seem to find a way to filter out the referral without filtering out the quote it's related to. In this example, quote '12345' would be filtered from the dashboard instead of moving to the '0 Referrals' category and I end up with: