The canonical tensor model, which is a tensor model in the Hamilton formalism, can be straightforwardly quantized and has an exactly solved physical state. The state is expressed by a wave function with a generalized form of the Airy function. The simplest observable on the state can be expressed by a matrix model which contains non-pairwise index contractions. This matrix model has the same form as the one that appears when the replica trick is applied to the spherical $p$-spin model for spin glasses, but our case has different ranges of variables and parameters from the spin glass case. We analyze the matrix model analytically and numerically. We show some evidences for the presence of a continuous phase transition at the location required by a consistency condition of the canonical tensor model. We also show that there are dimensional transitions of configurations around the transition region. Implications of the results to the canonical tensor model are discussed.