Filter the title to only affect the_title() template function

I am using the_title filter to add an HTML snippet under the title, as follows:

I only want the output to happen for the main title in the posts or page view.

That's why I added an early return to make sure that the filter only applies if is_singular() returns true:

if ( ! is_singular() ) { return; }

But there is an issue, since the title may be loaded in other places inside the single view template.

For example, the twentytwentyone theme displays the post or page edit link as follows:

edit_post_link(
sprintf(
     esc_html__( 'Edit %s', 'twentytwentyone' ),
     'span class=screen-reader-text' . get_the_title() . '/span'
),
'span class=edit-link',
'/span'
);

So I am having trouble with the HTML being output in places that I don't want.

How do you suggest to handle this situation (without touching the theme files) to make sure the snippet is only added under the main title area?