Processes involving the Flavour Changing Neutral Current transitions in b-hadron decays are suppressed in the Standard Model and are sensitive to New Physics. We present a search for the rare decays $B^0_s \to \mu^+\mu^-$ and $B^0 \to \mu^+\mu^-$ using $25~fb^{-1}$ of data collected by the ATLAS experiment at 7 and 8 TeV proton-proton collision energy. For the $B^0$ dimuon decay, an upper limit on the branching fraction is set at ${\cal B}(B^0 \to \mu^+\mu^-) < 4.2 \times 10^{-10}$ at 95% confidence level. For $B^0_s$, the branching fraction ${\cal B}(B^0_s \to \mu^+\mu^-) = \left( 0.9^{+1.1}_{-0.8} \right) \times 10^{-9}$ is obtained. The results are consistent with the Standard Model expectation and with the combined result of the CMS and LHCb experiments.