In many interesting systems, the fermion determinant becomes complex and its phase plays a crucial role in the determination of the vacuum. For instance, in finite density QCD at low temperature and high density, exotic fermion condensates are conjectured to form due to such effects. When one applies the complex Langevin method to such a complex action system naively, one cannot obtain the correct results because of the singular-drift problem associated with the appearance of small eigenvalues of the Dirac operator. Here we propose to avoid this problem by adding a fermion bilinear term to the action and extrapolating its coefficient to zero. We test this idea in an SO(4)-invariant matrix model with a Gaussian action and a complex fermion determinant, whose phase is expected to induce the spontaneous breaking of the SO(4) symmetry. Our results agree well with the previous results obtained by the Gaussian expansion method.