Deep learning technology offers advantages in vibration signal recognition with high accuracy and precision. However, acquiring a large number of labeled data for polygonal wheel detection is challenging, which fails to meet the training requirements of conventional neural network models. Existing methods to address the issue of small sample sizes often convert time-domain data into frequency-domain data, but this can re-sult in the loss of certain data features during the time-frequency conversion. To address this issue, a polygonal wheel detection method based on the 1DResAE deep neural network model is proposed. This model completes the detection of polygonal train wheels by unsupervised learning, feature extraction, and supervised learning of time-domain signals without the need for time-frequency conversion of vibration signals. By integrating one-dimensional convolution, residual networks, and autoencoders, a one-dimensional deep neural network is formed, capable of extracting and learning complex one-dimensional vibration signal features. Based on the features extracted and learned by the encoder in the autoencoder, the classifier performs supervised learning with a small amount of labeled data to achieve pattern recognition of polygonal train wheels. Experimental verification using data collected from a small-scale wheel-rail rolling test bench demonstrated that the detection accuracy of this method is 98.971%, with low error and outstanding classification performance. For the task of polygonal wheel detection, the 1DResAE model effectively detects the polygonal order of wheels and has practical applicability.