We investigate the potential of the ratios $R^{\tau \mu}$ in $B \to K \ell \ell$ and $B \to K^* \ell \ell$ decays ($\ell=\mu,\,\tau$) to probe new physics in the $\tau-\mu$ sector. We find that this ratio deviates from their SM prediction even for universal couplings. This implies that the bare deviation of these ratios from their SM predictions cannot confirm the nature of possible new physics. For this, we need to compare the allowed range of $R^{\tau \mu}$ for a class of solutions with only universal couplings to leptons and solutions having both universal and non-universal components. By comparing the predictions of $R^{\tau \mu}_K$ and $R^{\tau \mu}_{K^*}$ for the two class of solutions using the current data, we find that the solutions with only universal couplings and solutions having both universal and non-universal couplings can be discriminated if the measured value of $R^{\tau \mu}_{K^*}$ is greater than the SM prediction.