You are squaring the term too early. The (') symbol means you need to take the complex conjugate of the term before multiplying it by the inverse of Q and then the term again. I believe you are trying to calculate this in which case the term you want to...

opencv,image-processing,histogram,covariance

The common point here is probability density map. Compute probability density maps for each of your methods, then sum ( or multipy) values, normalize, and you'll have combined probability density map for your object.