I am having trouble using fminsearch [...] Stop right there and think some more about the function you're trying to minimize. Numerical optimization (which is what fminsearch does) is unnecessary, here. Your function is a quadratic function of vector x; in other words, its value at x can be...

matlab,correlation,matrix-inverse,determinants,fminsearch

How about sqrt(det(Gamma)) for the sqrt-determinant and inv(Gamma) for inverse? But if you do not want to implement it yourself you can look at yulewalkerarestimator UPD: For estimation of autocovariance matrix use xcov also, this topic is a bit more explained here...

matlab,function,parameters,fminsearch

You should call your function like [estimates, ~, exitflag, output] = fminsearch(...) to diagnose your function call. exitflag should be 1 if the tolerance was reached, otherwise something wrong happened. output is a structure that you can consult to see the run summary of your fminsearch() call. Also, your function...