The Standard Model predicts several rare Higgs boson processes, such as decays into a pair of muons or into a Z boson and a photon. Observing these rare decays would offer new and complementary insights into the Higgs boson's coupling structure beyond the more commonly studied channels. In addition, searches for lepton-flavor-violating decays of the Higgs boson are performed, where any observation would provide unambiguous evidence of physics beyond the Standard Model. Several recent results are presented from the ATLAS experiment based on proton-proton collision data collected in Run 2 at $\sqrt{s} = 13$ TeV, with the inclusion of available Run 3 results at $\sqrt{s} = 13.6$ TeV where relevant.