As a kind of navigation equipment

magnetic compass was widely applied in the navigation systems of flight vehicles

but it was always disturbed by ferromagnetic material to produce the magnetic deviation. The traditional magnetic compass deviation calibration generally adapted the least squares ellipse fitting calibration algorithm

but the coefficient matrix of this algorithm may be singular which caused the algorithm unstable question. By analyzing the singularity of the coefficient matrix

an improved least squares ellipse fitting calibration algorithm was proposed to solve the unstable question of conventional algorithm. Simultaneously this algorithm transformed six dimensional eigenvector's solution to the three dimensional eigenvector's solution

reduced the computation load while keeping the precise. The experimental results shown the rapidity and validity of this algorithm and the magnetic compass precision improved from 0.8° to 0.4°

meeting the requirements of the navigation system after calibration.