I don't know an explicit formula (and if one exists, it's probably not easy to digest), but it's easy to get the center as a result of solving a small linear system -- without optimization algorithms. Given points P1, P2, P3, P4, apply translation so that P1 becomes the origin....
c,computational-geometry,tetrahedra
USING TRIGONOMETRY Compute the lengths of the altitudes HA and HD in their respective face from the sides. Compute the angle AHD by the cosine formula. USING ANALYTICAL GEOMETRY Project A orthogonally onto BC to get H: BH = ((AB.BC)/BC²).BC (bold are vectors) Compute the angle from cos AHD =...