In order to consider the nonlinear characteristics of rock and soil materials in slope stability analysis, a rigorous upper bound variational analysis method for slope stability based on the deep-seated failure mechanism is proposed. Based on the upper bound theorem of limit analysis, a rotational failure mechanism of homogeneous slope was established. The sliding surface and its stress under limit state were obtained according to the Euler equation of variational principle, variational transversality conditions and boundary conditions. The implicit function for the critical slope height was established via the energy balance equation, and the critical height of slope was optimized by particle swarm optimization algorithm. The slope factor of safety (FOS) was obtained. By comparing , it is found that the potential sliding surface and the distribution of principal stress on the slope obtained by the proposed method are in good agreement with the calculation results of OPTUM G2 and FLAC 3D, and the error of safety factor is small. The results show that the variational analysis method can effectively evaluate the stability of homogeneous slope with sliding below the slope toe.