Abstract: Aim To present a robust method for tracking feature and evaluating the tracking quality of an image sequence. Methods First, a structural feature was selected to contain several patches. The topology among the patches in the structural feature was assumed to be invariant. Then, the gradient based and correlation based algorithm is used to find out the correspondences of the patches between previous frame and current frame. Thirdly, the evaluation of correspondences under the constraint of invariant structure feature during motion was achieved. Results and Conclusion The approach was applied to several testing images and showed better tracking results than non structural feature algorithm.