Addressing the challenges of highway pavement defect, such as diverse categories, significant scale variations, and high background complexity, an improved YOLOv7-based highway pavement defect detection algorithm is proposed. Firstly, an explicit vision center module is integrated into the neck network to comprehensively capture both global and local information of input features, thereby enhancing the feature extraction capability for small targets. Secondly, a feature fusion module RFECSP, is designed to mitigate the issue of low detection accuracy caused by the loss of detail information and interference from irrelevant background regions, by reinforcing the feature fusion for multi-class and multi-scale defect. Finally, the MPDIoU loss function is employed to further improve the network's convergence speed and detection accuracy. The results demonstrate that the algorithm exhibits excellent effectiveness in detecting roadway pavement defect, effectively meeting the requirements for detecting cracks or potholes in highway pavements.