Sorting taxonomy columns by meta value numeric

Slowly going crazy over trying to make taxonomy admin columns sortable by a custom field number. I have a custom taxonomy called "typer" and these taxonomies have a custom field called "prioritet".

I managed to get my code to show the column, and make it sortable. Only issue is, that the fields are sorted alphabetically, despite all being numbers. thats means that a fieldset of [1, 3, 14, 22] would show as:

1

14

22

3

My code so far

function create_date_column_for_issues($issue_columns) {
  $issue_columns['prioritet'] = 'Prioritet / sortering';
  return $issue_columns;
}
add_filter('manage_edit-typer_columns', 'create_date_column_for_issues');

function populate_date_column_for_issues($value, $column_name, $term_id) {
  $issue = get_term($term_id, 'typer');
  $date = get_field('prioritet', $issue);
  switch($column_name) {
    case 'prioritet': 
        $value = $date;  


    break;
    default:

    break;
  }
  return $value;    
}
add_filter('manage_typer_custom_column', 'populate_date_column_for_issues', 10, 3);

function register_date_column_for_issues_sortable($columns) {
  $columns['prioritet'] = 'prioritet';
  return $columns;
}
add_filter('manage_edit-typer_sortable_columns', 'register_date_column_for_issues_sortable');

add_filter( 'terms_clauses', 'filter_terms_clauses', 10, 3 );

/**
 * Filter WP_Term_Query meta query
 *
 * @param   object  $query  WP_Term_Query
 * @return  object
 */
function filter_terms_clauses( $pieces, $taxonomies, $args ) {

    global $pagenow, $wpdb; 

    // Require ordering
    $orderby = ( isset( $_GET['orderby'] ) ) ? trim( sanitize_text_field( $_GET['orderby'] ) ) : ''; 
    if ( empty( $orderby ) ) { return $pieces; }

    // set taxonomy
    $taxonomy = $taxonomies[0];

    // only if current taxonomy or edit page in admin           
    if ( !is_admin() || $pagenow !== 'edit-tags.php' || !in_array( $taxonomy, [ 'typer' ] ) ) { return $pieces; }

    // and ordering matches
    if ( $orderby === 'prioritet' ) {
        $pieces['join']  .= ' INNER JOIN ' . $wpdb-termmeta . ' AS tm ON t.term_id = tm.term_id ';
        $pieces['where'] .= ' AND tm.meta_key = "prioritet"'; 
        $pieces['orderby']  = ' ORDER BY tm.meta_value '; 
    }

    return $pieces;
}

My knowledge on MySQL is next to nothing, so i have no idea where to go width this.

As an extra:

Right now the sorting will exclude all taxonomies that has an empty field. Would be nice to have these shown in the bottom, but i do realize i requires a way more complicated query. So only join in on that if just so happen to have the solution on the top of your head.