Enhancing Performance of Operationalized Machine Learning Models by Analyzing User Feedback

Machine learning (ML) models that have been put into production must be actively monitored and maintained to ensure that the models continue to satisfy performance quality requirements. User feedback is often a very good indicator of whether the model is performing as per user expectations. We contend that monitoring ML models through user feedback, and promptly addressing the issues mentioned in the feedback will help to ensure long-term success of deployed ML models. However, the problem of incorporating user feedback for maintaining ML models has not been adequately studied. In this paper, we first motivate the importance of this problem by highlighting the benefits of incorporating user feedback. The paper discusses the challenges of effectively harnessing user feedback for enhancing performance of deployed ML models. Furthermore, we present a novel approach for analyzing user feedback to identify incompleteness of training datasets or differences between distributions of training datasets and actual workload. Our approach compares the user feedback and training data of deployed ML models to find the above mentioned deficiencies in the training dataset. And we determine a priority order to fix the issues. We also present experiments to demonstrate the effectiveness of the proposed approach.