To realize tracking targets automatically in the environments of complex background, color confusion and target deformation, the characteristics of Camshift algorithm sensitive to the hue component are extended to saturation, brightness and other components, and a convex combination of three-component color model is set up. The multi-objective programming optimal solution is used to get the best combination of adaptive color recognition, and combining with the center of mass, inclination to identify and Kalman tracking motion information prediction, the accurate tracking results are obtained in complex scenes of color target rotated, block and rate changes. The improved Camshift combination algorithm is applied in real-time adaptive tracking systems, and the recognition speed and tracking capability of systems are improved.