Breadcrumbs and Categories

I have a post with two/three categories associated to it.

Category 1 (1) and Category 2 (2). Number two also has a sub category (2.1)

I select the sub category (2.1) to be the primary category. However, in the bread crumbs the category 1 always shows. I don't understand how I can make the category sting 2 to show in the bread crumbs instead. I use the Yoast plugin to make the (2.1 category to be my primary category). Should't that make that string appear in the breadcrumbs string?

Thanks!