In the daily safety inspection of railway infrastructure based on Unmanned Aerial Vehicles (UAVs), the perception of encroaching obstacles, such as gravel, is crucial for ensuring operational safety. However, the highly irregular edge morphology of such obstacles, coupled with the strict computational resource constraints in practical engineering deployment, makes it difficult for existing models to balance feature extraction efficiency with shape-adaptive perception. This paper proposes a customized YOLO11 detection architecture trained entirely from scratch (with zero pre-trained weights). In the backbone network, a Lightweight Efficient Pyramid Adapter (LEPA) is designed to decouple features and significantly strip away redundant parameters. During the multi-scale fusion stage, a Deformable Attentional Multi-scale Fusion (DAMF) module is introduced, which combines spatial attention masks with deformable offsets to adaptively and precisely conform to the physical contours of irregular targets. Experimental validation demonstrates that, compared to the baseline YOLO11, the proposed method boosts the mAP@0.5 and mAP@[0.5:0.95] by 1.72 and 5.12 percentage points, respectively. Benefiting from feature decoupling, the parameter count and theoretical computational complexity are significantly reduced by 5.41% and 11.11%, respectively. This study effectively overcomes the detection challenges caused by limited computational resources and variable target shapes, providing a lightweight and highly efficient technical solution for high-precision intelligent perception in complex railway environments.