We analyze the phase structure of $SU(\infty)$ gauge theory at finite temperature using matrix models. Our basic assumption is that the effective potential is dominated by double-trace terms for the Polyakov loops. As a function of the temperature, a background field for the Polyakov loop, and a quartic coupling, it exhibits a universal structure: in the large portion of the parameter space, there is a continuous phase transition analogous to the third-order phase transition of Gross, Witten and Wadia, but the order of phase transition can be higher than third. We show that different confining potentials give rise to drastically different behavior of the eigenvalue density and the free energy. Therefore lattice simulations at large $N$ could probe the order of phase transition and test our results.