Best How To :

The first issue I see is that your are placing your video as (hidden) inline content. Most dynamic inline or multimedia content, as is the case of audio, videos and iframes, causes other problems like audio/video still playing in the background after fancybox is closed, etc... not mentioning they may also affect your page load and performance.

I wouldn't advice you to use static HTML inline content as you have it but to take another approach and build the HTML content to show in fancybox on-the-fly.

First, target your MP4 media file directly in your link's href like :

<a id="btnForm" href="http://vjs.zencdn.net/v/oceans.mp4" title="" class="fancy_image"><img src="/getVideoIcon.png" alt=""></a>

Notice that with this approach, you could have different links to several videos (using classes instead of ID in your selector) and you only need to set the corresponding href without having to create a new hidden inline content for each new video.

Then, in your fancybox custom initialization script, use callbacks to :

• pre-build the <video> content and insert the href from the link (beforeLoad)
• initialize video.js (beforeShow)
• remove the video.js instance after you are done (afterClose).

Here is the code:

var player = {}; // global object (so we don't have to initialize several variables)
jQuery(document).ready(function ($) {
    $("#btnForm").fancybox({
        fitToView : false, // keeps fixed video dimensions regardless screen size
        width : 640, // matches size in <video> tag
        height : 264,
        type : "html", // this is important
        autoSize: false, // fixes intermediate state
        beforeLoad : function () {
            // create a random ID :
            player.id = "video-player" + new Date().getTime();
            // pre-build HTML video content :
            player.content =
                '<video id="' + player.id + '" src="' + this.href + '" class="video-js vjs-default-skin" controls preload="auto" width="640" height="264" poster="http://www.ephotobay.com/image/birds-wallpaper-8-chai3721post.jpg" data-setup="{}">' +
                '<p class="vjs-no-js">To view this video please enable JavaScript, and consider upgrading to a web browser that <a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a></p>' +
                '</video>';
            // load video.js flash fallback :
            videojs.options.flash.swf = "{path}/video-js.swf";
            // set fancybox content :
            this.content = player.content;
        },
        beforeShow : function () {
            player.js = videojs(player.id); // initialize video.js
        },
        afterClose : function () {
            $(player.js).remove(); // remove video.js instances
        }
    });
}); // ready

Notice we need the combination of fitToView : false and autoSize: false to eliminate the "intermediate state", but it's important to set dimensions to avoid unexpected results.

And see the DEMO (feel free to explore the source code)