Stereo matching algorithm plays an important role in binocular vision. Census stereo matching algorithm based on nonparametric transformation has been widely used in recent years, but in the process of obtaining disparity map, Census stereo matching algorithm relies too much on the gray value of the central pixel. In view of the insufficient matching accuracy caused by the traditional Census algorithm relying too much on the center pixel, this paper proposes a Census stereo matching algorithm based on cross based aggregation. In order to improve the accuracy of parallax image, this algorithm finds the pixels around the reference pixels which are similar to their colors through the cross based aggregation, and uses the pixels in the cross based aggregation to optimize the generation value of the reference pixels. The experimental results show that the cross-based Census stereo matching algorithm improves the parallax accuracy of the image, and under the condition of ensuring the matching accuracy, the cost aggregation time of the algorithm is greatly reduced compared with the four path cost aggregation Census algorithm.