A search for Lepton Flavor Violating (LFV) decays of the Higgs boson to a muon and a tau lepton, and an electron and a tau lepton is presented. The data set used to perform the search has a total luminosity of 35.9 fb$^{-1}$ of proton-proton collisions collected by the CMS detector in 2016, at a center-of-mass energy of 13 TeV. Good agreement with the standard model background expectation is obtained and we set the observed (expected) upper limits on the LFV branching fractions of the Higgs boson to be B(H$\rightarrow$$\mu$$\tau$) < 0.25$\%$ (0.25$\%$) and B(H$\rightarrow$e$\tau$) < 0.61$\%$ (0.37$\%$) at 95$\%$ confidence level (CL).