In view of template selection problems which affected the LUT inverse halftone image quality, puts forward a solution to an optimal LUT inverse halftone template based on improved simulated annealing algorithm. The algorithm can be used for grayscale image inverse halftoning and is extended to color image. As for color image halftoning, the correlation between 3 channels of R, G, B color image is considered, and the template is extended from single to triple. The experimental result shows that the proposed algorithm finds template with global optimality, and compares with traditional template selection algorithm, the result picture adopting the optimal template for grayscale and color image inverse halftone restoration gets better effect in human vision system(HSV) and the average peak signal to noise ratio is higher.