The complex Langevin method (CLM) provides a promising way to perform the path integral with a complex action using a stochastic equation for complexified dynamical variables. It is known, however, that the method gives wrong results in some cases, while it works, for instance, in finite density QCD in the deconfinement phase or in the heavy dense limit. Here we revisit the argument for justification of the CLM and point out a subtlety in using the time-evolved observables, which play a crucial role in the argument. This subtlety requires that the probability distribution of the drift term should fall off exponentially or faster at large magnitude. We demonstrate our claim in some examples such as chiral Random Matrix Theory and show that our criterion is indeed useful in judging whether the results obtained by the CLM are trustable or not.