Abstract: A method was proposed to optimize panorama stitching by using photos from an aperture-coded camera. This method generated an optimized panorama, which not only saved the good features of classical panorama, but also can remove the foreground of scene and the blur of image to generate a perfect scene with a large field of view. This paper combined depth camera and panorama photography for the first time. The proposed method used an aperture-coded camera to capture high-resolution photos, extracted the depth of the scene by using which to remove the foreground objects and obtained a picture with interested scenes. Meanwhile, based on the depth information, the image blur can be removed to generate an all-focus panorama. Compared to the classical method of panorama stitching which directly stitch source photos, the proposed method can avoid contradiction between close and far range object in panorama stitching, can avoid unrelated objects mixing with interested scene, and can generate a panorama sharp everywhere. Many experiments are performed, including cylindrical, spherical and multi-viewpoint panorama stitching. The experiment results are shown in this paper, and they prove the proposed method effective.