With the development of Web-based social networks, recommender systems have become prevalent and integral to how users are interacting with the Internet. They filter out redundant information and personalize relevant and interesting items to online users. However, the positive reinforcement effect of recommender systems narrows users’ information experiences and cause filter bubble problems. How to provide relevant and diversified items for online users are becoming a challenging issue. In this study, we develop a novel cross domain matrix factorization model with adaptive diversity regularization to tackle the above challenges. We leverage the social tags and adaptive diversity regularization to im-prove recommendation performance. We conducted a comprehensive experiment on a real social media site to verify the effectiveness of the proposed method. The results show that the proposed method is able to achieve a decent balance between the accuracy and diversity of recommendation.