Aiming at deflection or loss of high-speed railway fasteners caused by the loose fasteners in the ballastless track of high-speed railway, this paper proposes a high-speed railway fastener detection algorithm based on improved Faster R-CNN. Deformable convolution was introduced in the feature extraction network to build deformable residual convolution block(DRCB), which makes the feature extraction process more focused on the fastener region and achieves the accurate extraction of fastener state; and Alpha-IoU was used as the target regression loss function to improve the regression accuracy of high-speed railway fasteners. The experimental results show that the algorithm proposed improves the detection accuracy of high-speed railway fasteners and can perform fastener localization and state detection more accurately than other algorithms.