A vision-based low altitude tracking system mounted on an unmanned aerial vehicle is established. The system is composed of ground station and on-board module

and constructed two parallel control loops: automatic tracking control loop and manual intervening control loop. The multi-threshold segmentation algorithm with adaptive tolerance based on gray-level histogram is presented

and a object dynamic clustering algorithm with double sub-windows is analyzed. The pan and tilt head are controlled based on the miss distance of the object centroid. The system can switch smoothly from one control loop to another as the miss distance is replaced by the coordinates of the mouse relative to the center of the monitor view. The motion is segmented even in the shade by several appended thresholds for varying illunination. The object centroid position is obtained correctly through 2~3 iterations in the process of dynamic clustering

and the disturbance of other objects is removed. The system can produce good tracking results at a frame rate of 15 frame/s. The results show that the algorithms and methods are available

and the system has a certain extent practicability.