In order to solve the problems in traditional convolutional neural network -based handwritten digits recognition, such as the computation overflow and high demand of computer hardware caused by the exponential function operation of Softmax, a support vector machine (SVM) handwritten digital recognition method based on the feature extraction of convolutional neural network is proposed. And, in order to improve the recognition accuracy of handwritten digits, a particle swarm optimization method based on the fitness function in the sense of K -CV is designed. Experimental simulation based on Semeion and MNIET handwritten digits shows that the method proposed in this paper can achieve higher recognition rate than the traditional methods.