In view of the susceptibility of high-speed railway insulator detection to climatic and environmental factors, as well as the deficiencies in accuracy and efficiency, this study proposes an improved algorithm ABFPYOLOv8. This method first constructs a C2f-AFE module to strengthen the extraction ability of global contextual features and suppress complex background interference. Subsequently, the neck network is replaced with a BCNeck structure to enhance the ability to capture target details. Furthermore, the Powerful-IoU loss function is employed to optimize the localization performance and reduce the false detection rate. Additionally, a 160 × 160 small target detection head is added to enhance the recognition ability of small-sized insulators. Experimental results indicate that, compared with the original YOLOv8n model, the ABFP-YOLOv8 model achieves improvement in mAP@50 and mAP@50-95, increase in inference speed, despite reduction in the number of parameters. This suggests that the algorithm is highly suitable for deployment in mobile detection terminals and scenarios with complex and variable detection environments.