Lattice calculation of $B\to D^{(*)}\ell\bar\nu$ decay form factors is now available from multiple lattice collaborations for both zero-recoil limit and non-zero recoil kinematics. Yet, there are a number of challenges we face to better control the systematic errors and to extend the application of lattice calculations to related quantities. The subjects are chosen from my own perspective and not meant to be conprehensive.