Summary: We consider the linear time-dependent Schrödinger equation with a time-dependent smooth potential on an unbounded domain. A Galerkin spectral method with a tensor-product Hermite basis is used as a discretization in space. Discretizing the resulting ODE for the Hermite expansion coefficients involves the computation of the action of the Galerkin matrix on a vector in each time step. We propose a fast algorithm for the direct computation of this matrix-vector product without actually assembling the matrix itself. The costs scale linearly in the size of the basis. Together with the application of a hyperbolically reduced basis, this reduces the computational effort considerably and helps cope with the infamous curse of dimensionality. The application of the fast algorithm is limited to the case of the potential being significantly smoother than the solution. The error analysis is based on a binary tree representation of the three-term recurrence relation for the one-dimensional Hermite functions. The fast algorithm constitutes an efficient tool for schemes involving the action of a matrix due to spectral discretization on a vector, and it is also applicable in the context of spectral approximations for linear problems other than the Schrödinger equation.