Can you disable typography panels for just the paragraph block?

I'm wondering if there is a way to remove the typography and custom color panels from the sidebar for just the paragraph block.

I've currently written a function that I've added add_theme_support to for the typography panel and custom color picker but it's affecting every block that can utilize those things. Here is said function:

    public function typography_custom_color_theme_support() {
        // Disable Custom Color Picker
        add_theme_support( 'editor-color-palette' );
        add_theme_support( 'disable-custom-colors' );
        // Disable Font Size and Custom Font Size Dropdowns
        add_theme_support( 'editor-font-sizes' );
        add_theme_support( 'disable-custom-font-sizes' );
}

If possible, could I add a conditional to this along the lines of if paragraph block is selected run add_theme_support actions?

Thank you.