Order terms by term_order

term_order isn't an ordering for terms within a taxonomy. It's an ordering for terms within an object (post).

https://developer.wordpress.org/reference/classes/wp_term_query/__construct/ says:

Unless $object_ids is not empty, 'term_order' is treated the same as 'term_id'.

For people who are trying to order terms for an object by term_id, you need to provide object_ids to the query.

For everyone else who asks this question, who is looking to order a list of terms, don't use term_id. Instead add a term meta value, and order by that.

get_terms([
  'taxonomy'   => 'my_taxonomy',
  'meta_key'   => 'order',
  'orderby'    => 'meta_value_num'
]);

This is an idea
But I did it thinking of ordering a listing on the front end
You need to change the table "{prefix}term_taxonomy", creating the "term_order" field. See the function O_term_update_db():

In your function.php

<?php
function O_term_update_db(){
    global $wpdb;
    global $term_db_version;
    $term_db_version = "1";

    $charset_collate = $wpdb->get_charset_collate();

    if(get_site_option( 'O_term_db_version' ) < $term_db_version):

        require_once( ABSPATH . 'wp-admin/includes/upgrade.php' );

        $wpdb->query( "ALTER TABLE $wpdb->prefix .'term_taxonomy' ADD `term_order` INT (11) NOT NULL DEFAULT 0;" );

        update_site_option( 'O_term_db_version', $term_db_version);
    endif;
}
add_action('init','O_term_update_db');
?>

This function refers to the database:
In your functions.php

<?php
function O_taxonomy_filter_by_order($taxonomy){
    global $wpdb;
    $terms = $wpdb->get_results(" 
        SELECT t.*, tt.*
        FROM ".$wpdb->prefix."term_taxonomy AS tt
        JOIN ".$wpdb->prefix."terms AS t
        ON t.term_id = tt.term_id
        WHERE tt.taxonomy = '".$taxonomy."'
        ORDER BY tt.term_order ASC
    ",ARRAY_A);
    return $terms;
}
?>

This function is used to update the table by Ajax:
In your function.php

<?php
if(!function_exists('O_updating_data_term_reorder')){ 
    function O_updating_data_term_reorder() {

        check_ajax_referer('o-nonce', 'security');

        global $wpdb;
        $terms_ID = $_POST['each_term'];

        $new_order = 0;

        foreach($terms_ID as $key => $ID):

            $term_ID = $ID['term'];

            $new_order++;

            $result = $wpdb->update($wpdb->prefix.'term_taxonomy', array('term_order' => $new_order), array('term_id' => $term_ID));

            if(is_wp_error($result)) {
                echo json_encode(array('save'=>false, 'message'=> 'ERROR: '.$result->get_error_message()));
                exit();
            }

        endforeach;

        echo json_encode(array('save'=>true, 'count'=>$new_order));

        die();
    }
    add_action( 'wp_ajax_o_update_term_reorder', 'O_updating_data_term_reorder' );
}
?>

draggable-ajax.js:

(function($) {
    "use strict";

     //jQuery Sortable is required http://api.jqueryui.com/sortable/
     //Call "wp_enqueue_script('jquery-ui-sortable');" in your wp_enqueue_scripts
     $('.js-draggable-items > .draggable-column').sortable({
        connectWith: '.draggable-column',
        items: '.draggable-item',
        dropOnEmpty: true,
        opacity: .75,
        handle: '.draggable-handler',
        placeholder: 'draggable-placeholder',
        tolerance: 'pointer',
        start: function(e, ui){
            ui.placeholder.css({
                'height': ui.item.outerHeight(),
                'margin-bottom': ui.item.css('margin-bottom')
            });
        }
    });

    $(document).on( "click","#btn-reorder-terms", function() {
        var each_term_list = [];
        $('.draggable-item').each(function(){
            each_term_list.push({
                term: $(this).attr('data-draggable-id'),
            });
        });
        var data = {
            'action' : 'o_update_term_reorder',
            'each_term' : each_term_list,
            'security' : actions_vars.nonce
        };
        $.ajax({
            type: 'POST',
            dataType: 'json',
            cache: false,
            url: actions_vars.ajaxurl,
            data: data,
            success: function(data) {
                if(data.save === true){
                    //success message here
                    alert('Done');
                }else{
                    //error message here
                    alert('Something wrong');
                }
            },
            error: function(errorThrown) {
               console.log(data);
            }
        });
    });

})(jQuery);

In your front-end listing try that:

<?php
$terms = O_taxonomy_filter_by_order('your_taxonomy');
?>
<button type="button" class="btn btn-primary" id="btn-reorder-terms">Save</button>

<div class="col-md-12 js-draggable-items">
    <div class="row draggable-column">
    <?php
    foreach($terms as $key => $term):
        $ID = $term['term_id'];
        $name = $term['name'];
        $order = $term['term_order'];
        $description = $term['description'];
        $link = get_category_link($ID);
        ?>
        <div class="draggable-item" data-draggable-id="<?php echo $ID;?>" data-term-order="<?php echo $order;?>">
            <div class="col-md-2 draggable-handler"><i class="si si-cursor-move"></i></div>
            <div class="col-md-10"><?php echo $name;?></div>
        </div>
        <?php
    endforeach;
    ?>
    <div>
<div>

wp_term_query doesn't accept term_order as one of the orderby parameters. So you have to forcefully add that option to term query variable by adding below snippets to your functions.php.

function wpcf_filter_terms_order( $orderby, $query_vars, $taxonomies ) {
    return $query_vars['orderby'] == 'term_order' ? 'term_order' : $orderby;
}

add_filter( 'get_terms_orderby', 'wpcf_filter_terms_order', 10, 3 );

After adding above snippets if you will execute something like below it will give you the terms order by term_order.

$terms = get_terms('category', array(
                    'orderby'  => 'term_order',
                    'order'    => 'ASC'
                ));

Second approach, If you want to get the terms by term_order globally and all over the wordpress then just add below snippets to functions.php.

function uc_get_terms_orderby($orderby, $args){
        return 't.term_order';
}
add_filter('get_terms_orderby', 'uc_get_terms_orderby', 10, 2);

As this is one of the top results on Google and none of the above quite worked for me, this did seem to return terms in an order matching their display in the admin...

 get_terms([
  'taxonomy'   => 'whatever_you_want',
  'meta_key'   => 'order',
  'orderby'    => 'meta_value_num'
]);

In case it helps anyone, WP stores a value in the termmeta table under the key order which represents its display position in the admin menus. Obviously it would need to be adapted if you wanted to query on other meta at the same time.

Many years later,

You can add this code in your functions.php :

function wpcf_filter_terms_order( $orderby, $query_vars, $taxonomies ) {
    return $query_vars['orderby'] == 'term_order' ? 'term_order' : $orderby;
}

add_filter( 'get_terms_orderby', 'wpcf_filter_terms_order', 10, 3 );

This code force WP to use the orderby => term_order argument in your term query.

Instead of menu_order try term_order.

According to documentation on the Codex, wp_get_object_terms supports:

• name
• count
• slug
• term_group
• term_order and
• term_id