sanitize vimeo embed code?

You need to add a custom validation/sanitization callback, and hook it into publish_post (and/or draft_post and/or future_post, as applicable). For example:

<?php
function wpse_44807_update_custom_post_meta() {
    // Globalize the $post object
    global $post;
    // If our custom post meta key is set, sanitize it;
    // otherwise, return false
    $my_post_custom = ( isset( $_POST['_my_post_custom'] ? wp_filter_nohtml_kses( $_POST['_my_post_custom'] ? false );
    // Now, delete or update our custom post meta key
    if ( false == $my_post_custom ) {
        delete_post_meta( $post->ID, '_my_post_custom' );
    } else {
        update_post_meta( $post->ID, '_my_post_custom', $my_post_custom );
    }
}
add_action( 'publish_post', 'wpse_44807_update_custom_post_meta' );
add_action( 'draft_post', 'wpse_44807_update_custom_post_meta' );
?>

Note that I'm sanitizing using the wp_filter_nohtml_kses() filter, which would be appropriate if you are expecting, say, a video ID or something similarly alpha-numeric. Your choice of sanitization will change, depending on the type of expected input.

Also: I'm using an underscore-prefixed custom post meta key, which is appropriate if you're defining a custom post meta box for your custom post meta key. (The underscore prefix hides this meta key from the generic "custom field" meta box drop-down.)

update_post_meta() will sanitize it for database insertion for you. What you really need to be watching for is malicious HTML and such that would affect the output. For that you will need a regex or some other means of comparison to a known/desired format. To handle that I would recommend only accepting the video ID or something of that nature where you can strictly control how the output is done, rather than allowing the user to supply you with the iframe and such.