Matching users to the right items at the right time is a fundamental task in recommender systems. As users interact with different items over time, users' and items' feature may drift, evolve and co-evolve over time. Traditional models based on static latent features or discretizing time into epochs can become ineffective for capturing the fine-grained temporal dynamics in the user-item interactions. We propose a coevolutionary latent feature process model that accurately captures the coevolving nature of users' and items' feature. We use a recurrent neural network to automatically learn a representation of influences from drift, evolution and co-evolution of user and item features. We develop an efficient stochastic gradient algorithm for learning the model parameters which can readily scale up to millions of events. Experiments on diverse real-world datasets demonstrate significant improvements in user behavior prediction compared to state-of-the-arts.