Disable "auto embed" feature when pasting text and links

I want to disable the auto embed feature of Wordpress, when pasting arbitrary text that includes links in Gutenberg. I tried this with something like:

add_filter('allowed_block_types', 'remove_default_blocks');
function remove_default_blocks($allowed_blocks){
// Get all registered blocks
$registered_blocks = WP_Block_Type_Registry::get_instance()-get_all_registered();

// unset embed block
unset($registered_blocks['core/embed']);

// Get keys from array
$registered_blocks = array_keys($registered_blocks);

// Merge allowed core blocks with plugins blocks
return $registered_blocks;

}

This works - but unfortunately this disables all embed blogs and I want the user to manually embed e.g. YouTube. Best would be, if just the auto embedding could be disabled, but if the user could still choose the embed blocks manually and embed thinks.

Can anybody help me here?

Thanks

Harald