Abstract: In formation flying system, the relative position and attitude of spacecraft can be measured by the radio frequency (RF) relative measurement sensor.In order to verify its measurement precision, a calibration method of RF relative measurement sensor calibration was proposed based on laser tracker. In the process of frame building, methods of fitting sphere and circle to determine coordinate origin and axis based on the least squares were proposed,to solve the problem of larger fitting error. The inherent relationship of coordinate system was taken to design a fast conversion method of relative coordinate. The simulation experiment results show that origin fitting error <i<σ</i<<sub<RMS</sub<=0.120 764 mm, axis fitting error <i<σ</i<<sub<RMS</sub<=0.207 093 mm, the scope of coordinate frame conversion error 0.30~0.75 mm, they can satisfy the requirement of RF relative sensors calibration.