How do I get the trackback count of a post in wordpress without writing an SQL query?

You can specify what kind of comment you want to retrieve using get_comments().

// Assumes you've set $post_id somewhere.
$args = array(
    'type' => 'trackback',
    'post_id' => $post_id,
);
$trackbacks = get_comments( $args );
$trackback_count = count( $trackbacks );

Edited to add: As Sally CJ points out in the comments, you can do this in just one step:

// Assumes you've set $post_id somewhere.
$args = array(
    'type'    => 'trackback',
    'post_id' => $post_id,
    'count'   => true,
);
$trackback_count = get_comments( $args );

This code is untested but it should provide you a starting place.

As far as being inside/outside The Loop: The above code should work just fine either way, as long as you've got a way to get the post ID (get_the_ID() should work in The Loop; if you've got a WP_Post object called $my_post, then $my_post->ID should get it for you).

References

• get_comments()
• Allowed comment query arguments