How to change the seperator in the title

How document title is generated has changed since Wordpress v4.4.0. Now wp_get_document_title dictates how title is generated:

/**
 * Displays title tag with content.
 *
 * @ignore
 * @since 4.1.0
 * @since 4.4.0 Improved title output replaced `wp_title()`.
 * @access private
 */
function _wp_render_title_tag() {
    if ( ! current_theme_supports( 'title-tag' ) ) {
        return;
    }

    echo '<title>' . wp_get_document_title() . '</title>' . "\n";
}

Here is the code from v5.4.2. Here are the filters you can use to manipulate title tag:

function wp_get_document_title() {
    /**
    * Filters the document title before it is generated.
    *
    * Passing a non-empty value will short-circuit wp_get_document_title(),
    * returning that value instead.
    *
    * @since 4.4.0
    *
    * @param string $title The document title. Default empty string.
    */
    $title = apply_filters( 'pre_get_document_title', '' );
    if ( ! empty( $title ) ) {
        return $title;
    }
    // --- snipped ---
    /**
    * Filters the separator for the document title.
    *
    * @since 4.4.0
    *
    * @param string $sep Document title separator. Default '-'.
    */
    $sep = apply_filters( 'document_title_separator', '-' );

    /**
    * Filters the parts of the document title.
    *
    * @since 4.4.0
    *
    * @param array $title {
    *     The document title parts.
    *
    *     @type string $title   Title of the viewed page.
    *     @type string $page    Optional. Page number if paginated.
    *     @type string $tagline Optional. Site description when on home page.
    *     @type string $site    Optional. Site title when not on home page.
    * }
    */
    $title = apply_filters( 'document_title_parts', $title );
    // --- snipped ---
    return $title;
}

So here are two ways you can do it.

First one uses pre_get_document_title filter which short-circuits the title generation and hence more performant if you are not going make changes on current title:

function custom_document_title( $title ) {
    return 'Here is the new title';
}
add_filter( 'pre_get_document_title', 'custom_document_title', 10 );

Second way uses document_title_separator and document_title_parts hooks for the title and the title seperator that are executed later in the function, after title is generated using functions like single_term_title or post_type_archive_title depending on the page and about to be outputted:

// Custom function should return a string
function custom_seperator( $sep ) {
   return '>';
}
add_filter( 'document_title_separator', 'custom_seperator', 10 );

// Custom function should return an array
function custom_html_title( $title ) {
   return array(
     'title' => 'Custom Title',
     'site'  => 'Custom Site'
    );
}
add_filter( 'document_title_parts', 'custom_html_title', 10 );

UPDATE for WordPress 4.4

Since WordPress 4.4 the wp_title filter doesn't work because wp_title() function is not used in core anymore. That function was marked as deprecated, then reinstated until new notice but theme authors are discouraged from using it. Because of that, wp_title filter still will work if you continue using wp_title() function directly in your theme but it is not recommended.

There are new filters to customize the document title when theme support for title-tag is enabled:

1. pre_get_document_title
2. document_title_parts
3. document_title_separator.

As you only want to customize the separator, you can use document_title_separator as follow:

add_filter( 'document_title_separator', 'cyb_document_title_separator' );
function cyb_document_title_separator( $sep ) {

    $sep = "-";

    return $sep;

}

Previous answer

You can use wp_title filter to customize the <title> tag.

add_filter( 'wp_title', 'customize_title_tag', 10, 3 );
function customize_title_tag( $title, $sep, $seplocation ) {


    // Customize $title here.
    // Example taken from https://generatepress.com/forums/topic/title-tag-separator/
    $title = str_replace( '|', '-', $title );

    return $title;

}

A more complex example of how to use this filter (taken from TwentyTwelve theme):

function twentytwelve_wp_title( $title, $sep ) {
    global $paged, $page;

    if ( is_feed() )
        return $title;

    // Add the site name.
    $title .= get_bloginfo( 'name' );

    // Add the site description for the home/front page.
    $site_description = get_bloginfo( 'description', 'display' );
    if ( $site_description && ( is_home() || is_front_page() ) )
        $title = "$title $sep $site_description";

    // Add a page number if necessary.
    if ( $paged >= 2 || $page >= 2 )
        $title = "$title $sep " . sprintf( __( 'Page %s', 'twentytwelve' ), max( $paged, $page ) );

    return $title;
}
add_filter( 'wp_title', 'twentytwelve_wp_title', 10, 2 );