Rack to Secondary for Vanilla addition?

I'm planning a Vanilla Coffee Porter this weekend but I'm unable to decide when, or even if I should, rack to a secondary. My original plan was this:

Primary for 7 days in bucket, then add 2 vanilla beans to secondary carboy and rack over. Let it sit in secondary for 2 weeks. Add cold brew coffee and priming sugar to bottling bucket, rack over and bottle.

I'm unsure now whether to do a secondary though. I've been reading that removing the beer from the yeast cake in the primary too early won't allow the yeast to clean up after itself. But racking too late and the yeast won't clear the oxygen out of the secondary.

So I'm thinking of doing a primary for 2 weeks, then 1 week in the secondary with the vanilla beans. However I'd love to hear some advice on the subject.

Thanks!