Concrete crack detection is difficult to accurately classify, segment and locate due to the complexity of multiple classification. To solve the above problems, this paper proposes a steel fiber reinforced concrete (SFRC) crack detection model based on improved Mask R-CNN. In order to improve the detection rate and accuracy, a distraction module is added to the backbone network of the scheme model to span the feature graph group and improve the capability of feature learning. Based on intersection of union(IoU), the distance between the target and anchor frame, the overlap rate, the scale and the penalty term were increased to improve the regression accuracy, and compared with the original Mask R-CNN model. The simulation results show that the mean average precision of crack and number classification, segmentation and positioning is 96.09%, the model can accurately locate cracks and make pixel-level segmentation and the single image takes 198 ms. The proposed model increases the accuracy and reduces the image processing delay. Compared with the original Mask R-CNN model, the mean average precision and image processing rate are increased by 6.2% and 5.7% respectively. Experimental results show that the proposed model has strong robustness and generalization ability.