Abstract: In dynamic tracking system, object motion estimation is necessary. The measurement locations of the object are combined with background and foreground object movement. A new background compensation algorithm consisting of Harris corners and sparse optical flow is presented to extract the real movement of the object to predict its moving state. Harris-affine detector is used to find out the interested points, and then the corresponding optical vectors are obtained by calculating the optimal matched region of each interested pointed point. Finally the background movement is available according to the distribution of optical flow vectors orientation. And then Kalman filter is used here to predict the objects location in the coming frame with the "real locations" before. Experimental results show that the error between the predicted location and the measurement is within 10 pixels. The time cost of a single approved mean shift tracking unit is about 10ms. Compared with the conventional tracker, the iteration times decreases and the tracker performs better in real time.