The wide adoption of end-to-end encryption is drastically limiting the visibility Internet Service Providers (ISPs) have on the performance of the services consumed by their customers. In times of strong competition, where customer experience plays a key role in churn management, ISPs require novel solutions enabling network-wide Quality of Experience (QoE) monitoring. To this end, we present DeepQoE, a deeplearning based approach to infer the QoE of web services and mobile applications from the ISP perspective, relying exclusively on the analysis of encrypted network traffic. Using raw features derived from the encrypted stream of bytes as input to deep Convolutional Neural Networks (CNNs), DeepQoE infers the Speed Index of web browsing sessions and general mobile apps with unprecedented accuracy, improving the state of the art by more than 25%, and reducing the QoE inference error in terms of mean opinion scores by nearly 40%. DeepQoE implements a web fingerprinting solution to identify individual web browsing sessions within concurrent web pages traffic, enabling highly detailed, per web page QoE inference in practical deployments. Extensive evaluations over a large and heterogeneous dataset composed of web and app measurements, using different device types and for top-popular websites and apps, confirm the outperformance of DeepQoE over previously used shallow-learning models, as well as the deep-model generalization to different devices, web pages, apps, and network setups.