Access to the bird nest is a serious threat to the safe and stable operation of overhead catenary system. An improved YOLOv3 algorithm was proposed to detect and identify the adverse effects of bird nests on railway operation. Firstly, the bird nest image of overhead catenary system is preprocessed in the early stage. The denoising and other operations can enhance the extraction of the essential features of the bird nest, and the data enhancement can avoid the over-fitting phenomenon of the neural network to a certain extent. The spatial pyramid pooling module is added into the network structure, and the feature map is pooled at different scales and then stitched together to get the output of fixed size, and the multi-scale features of the nest are extracted. Finally, the GIoU, which measures the distance between the prediction box and the real box, is used as the bounding box loss function, and the model optimizes the loss and the overlap degree between the real box and the prediction box. The experimental results show that the average accuracy of this method in the overhead contact netnest detection reaches 95.1%, which has a high detection accuracy in the overhead contact net nest detection field, and can better identify and detect the nest under the complex overhead contact net background.