Due to the long-term repeated loading, surface defects occur in high-speed railway steel rails. In order to improve the accuracy and speed of surface defect detection for multiple classes and scales of steel rails in complex scenarios, a steel rail surface defect segmentation network based on multimodal fusion (DAFNet) is designed. Firstly, a steel rail surface defect dataset containing visible light and infrared channels is constructed, and an improved dual-branch network architecture is adopted to increase segmentation speed. Simultaneously, a bimodal adaptive fusion module (BAFM) is designed to achieve adaptive feature fusion, improving the segmentation accuracy of steel rail surface defects in complex scenarios. Additionally, a spatial detail extraction module (SDEM) and a key information enhancement module (KIEM) are designed to further enhance the perception of defect edges and address the low contrast between defects and backgrounds in complex scenarios. Experiments show that the accuracy and mIoU of the designed network segmentation reach 68.13% and 59.96% respectively, which are significantly better than other mainstream networks. Moreover, FLOPs, parameter quantity, and model size are 17.41 GFLOPs, 1.38 M, and 5.67 MB respectively, which are better than most mainstream networks. The designed network significantly improves the segmentation accuracy of steel rail surface defects and has a high segmentation speed, which is of great significance for ensuring the safe operation of high-speed railways.