Abstract: To improve the resolution and accuracy of a gaze tracking system, a Sub-pixel edge detection algorithm was proposed in order to locate the center of the pupil accurately in IR image of the eye. With our proposed edge operator, the edge points of the pupil can be obtained by a 1D line search. Furthermore, utilizing the local information of the image in the gradient direction of the edge points, the algorithm can locate the edge points of the pupil in subpixel accuracy. And the algorithm can accurately determine the center and shape parameters of a human’s pupil with an ellipse-fitting method. The experimental results indicate that the proposed algorithm achieved sub-pixel accuracy, which will significantly enhance the performance of the gaze tracking system. We have applied the algorithm to a real-time gaze tracking system.