To compensate the nonlinear errors of heterodyne interferometers in nanometer measurement
an experimental investigation is conducted. Based on the polarization properties of a coated corner-cube retro-reflector
a model is derived for describing the effect of the axial rotation of a measuring cube-corner retro-reflector in the motion direction on the first-harmonic nonlinearity when a laser emits the elliptical polarized light. The simulation results indicate that the first-harmonic nonlinearity can be reduced by the axial rotation of the cube-corner retro reflector. The experimental result shows that the first harmonic nonlinearity is reduced from 3.48 to 1.39 nm when the axial orientation rotation angle of the measuring cube-corner retro reflector increases from 0° to 100°
which is 40% that of the original one. The method simplies the complexes of light path and circuit systems for nonlinear error compensation approaches.