Abstract:In a typical optomechanical system, the decay rate of an optical cavity must be smaller than the frequency of a mechanical oscillator (resolved sideband regime) in order to generate strong mechanical squeezing generally. This is difficult to implement in experiments. A scheme is proposed to generate strong mechanical squeezing and steady state entanglement in a hybrid atom-optomechanical system in the highly unresolved sideband regime (the decay rate of the optical cavity is much larger than the frequency of the mechanical oscillator). Two two-level atomic ensembles and two lasers with different amplitudes can be used to control the optomechanical system and the resolved sideband limit can be relaxed. Our results are as follows: if two atomic ensembles are put into the optomechanical system, the strong mechanical squeezing beyond 3dB is achieved even in the highly unresolved sideband regime; the steady state entanglement between the cavity and mechanical resonator can also be obtained.