-
MTrainS: Improving DLRM training efficiency using heterogeneous memories
Authors:
Hiwot Tadese Kassa,
Paul Johnson,
Jason Akers,
Mrinmoy Ghosh,
Andrew Tulloch,
Dheevatsa Mudigere,
Jongsoo Park,
Xing Liu,
Ronald Dreslinski,
Ehsan K. Ardestani
Abstract:
Recommendation models are very large, requiring terabytes (TB) of memory during training. In pursuit of better quality, the model size and complexity grow over time, which requires additional training data to avoid overfitting. This model growth demands a large number of resources in data centers. Hence, training efficiency is becoming considerably more important to keep the data center power dema…
▽ More
Recommendation models are very large, requiring terabytes (TB) of memory during training. In pursuit of better quality, the model size and complexity grow over time, which requires additional training data to avoid overfitting. This model growth demands a large number of resources in data centers. Hence, training efficiency is becoming considerably more important to keep the data center power demand manageable. In Deep Learning Recommendation Models (DLRM), sparse features capturing categorical inputs through embedding tables are the major contributors to model size and require high memory bandwidth. In this paper, we study the bandwidth requirement and locality of embedding tables in real-world deployed models. We observe that the bandwidth requirement is not uniform across different tables and that embedding tables show high temporal locality. We then design MTrainS, which leverages heterogeneous memory, including byte and block addressable Storage Class Memory for DLRM hierarchically. MTrainS allows for higher memory capacity per node and increases training efficiency by lowering the need to scale out to multiple hosts in memory capacity bound use cases. By optimizing the platform memory hierarchy, we reduce the number of nodes for training by 4-8X, saving power and cost of training while meeting our target training performance.
△ Less
Submitted 19 April, 2023;
originally announced May 2023.
-
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models
Authors:
Dheevatsa Mudigere,
Yuchen Hao,
Jianyu Huang,
Zhihao Jia,
Andrew Tulloch,
Srinivas Sridharan,
Xing Liu,
Mustafa Ozdal,
Jade Nie,
Jongsoo Park,
Liang Luo,
Jie Amy Yang,
Leon Gao,
Dmytro Ivchenko,
Aarti Basant,
Yuxi Hu,
Jiyan Yang,
Ehsan K. Ardestani,
Xiaodong Wang,
Rakesh Komuravelli,
Ching-Hsiang Chu,
Serhat Yilmaz,
Huayu Li,
Jiyuan Qian,
Zhuobo Feng
, et al. (28 additional authors not shown)
Abstract:
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pa…
▽ More
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pair it with the new evolution of Zion platform, namely ZionEX. We demonstrate the capability to train very large DLRMs with up to 12 Trillion parameters and show that we can attain 40X speedup in terms of time to solution over previous systems. We achieve this by (i) designing the ZionEX platform with dedicated scale-out network, provisioned with high bandwidth, optimal topology and efficient transport (ii) implementing an optimized PyTorch-based training stack supporting both model and data parallelism (iii) developing sharding algorithms capable of hierarchical partitioning of the embedding tables along row, column dimensions and load balancing them across multiple workers; (iv) adding high-performance core operators while retaining flexibility to support optimizers with fully deterministic updates (v) leveraging reduced precision communications, multi-level memory hierarchy (HBM+DDR+SSD) and pipelining. Furthermore, we develop and briefly comment on distributed data ingestion and other supporting services that are required for the robust and efficient end-to-end training in production environments.
△ Less
Submitted 26 February, 2023; v1 submitted 11 April, 2021;
originally announced April 2021.
-
Mixed-Precision Embedding Using a Cache
Authors:
Jie Amy Yang,
Jianyu Huang,
Jongsoo Park,
Ping Tak Peter Tang,
Andrew Tulloch
Abstract:
In recommendation systems, practitioners observed that increase in the number of embedding tables and their sizes often leads to significant improvement in model performances. Given this and the business importance of these models to major internet companies, embedding tables for personalization tasks have grown to terabyte scale and continue to grow at a significant rate. Meanwhile, these large-s…
▽ More
In recommendation systems, practitioners observed that increase in the number of embedding tables and their sizes often leads to significant improvement in model performances. Given this and the business importance of these models to major internet companies, embedding tables for personalization tasks have grown to terabyte scale and continue to grow at a significant rate. Meanwhile, these large-scale models are often trained with GPUs where high-performance memory is a scarce resource, thus motivating numerous work on embedding table compression during training. We propose a novel change to embedding tables using a cache memory architecture, where the majority of rows in an embedding is trained in low precision, and the most frequently or recently accessed rows cached and trained in full precision. The proposed architectural change works in conjunction with standard precision reduction and computer arithmetic techniques such as quantization and stochastic rounding. For an open source deep learning recommendation model (DLRM) running with Criteo-Kaggle dataset, we achieve 3x memory reduction with INT8 precision embedding tables and full-precision cache whose size are 5% of the embedding tables, while maintaining accuracy. For an industrial scale model and dataset, we achieve even higher >7x memory reduction with INT4 precision and cache size 1% of embedding tables, while maintaining accuracy, and 16% end-to-end training speedup by reducing GPU-to-host data transfers.
△ Less
Submitted 22 October, 2020; v1 submitted 21 October, 2020;
originally announced October 2020.
-
Hybrid Composition with IdleBlock: More Efficient Networks for Image Recognition
Authors:
Bing Xu,
Andrew Tulloch,
Yunpeng Chen,
Xiaomeng Yang,
Lin Qiao
Abstract:
We propose a new building block, IdleBlock, which naturally prunes connections within the block. To fully utilize the IdleBlock we break the tradition of monotonic design in state-of-the-art networks, and introducing hybrid composition with IdleBlock. We study hybrid composition on MobileNet v3 and EfficientNet-B0, two of the most efficient networks. Without any neural architecture search, the dee…
▽ More
We propose a new building block, IdleBlock, which naturally prunes connections within the block. To fully utilize the IdleBlock we break the tradition of monotonic design in state-of-the-art networks, and introducing hybrid composition with IdleBlock. We study hybrid composition on MobileNet v3 and EfficientNet-B0, two of the most efficient networks. Without any neural architecture search, the deeper "MobileNet v3" with hybrid composition design surpasses possibly all state-of-the-art image recognition network designed by human experts or neural architecture search algorithms. Similarly, the hybridized EfficientNet-B0 networks are more efficient than previous state-of-the-art networks with similar computation budgets. These results suggest a new simpler and more efficient direction for network design and neural architecture search.
△ Less
Submitted 19 November, 2019;
originally announced November 2019.
-
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
Authors:
Jongsoo Park,
Maxim Naumov,
Protonu Basu,
Summer Deng,
Aravind Kalaiah,
Daya Khudia,
James Law,
Parth Malani,
Andrey Malevich,
Satish Nadathur,
Juan Pino,
Martin Schatz,
Alexander Sidorov,
Viswanath Sivakumar,
Andrew Tulloch,
Xiaodong Wang,
Yiming Wu,
Hector Yuen,
Utku Diril,
Dmytro Dzhulgakov,
Kim Hazelwood,
Bill Jia,
Yangqing Jia,
Lin Qiao,
Vijay Rao
, et al. (3 additional authors not shown)
Abstract:
The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions…
▽ More
The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers.
△ Less
Submitted 29 November, 2018; v1 submitted 24 November, 2018;
originally announced November 2018.
-
On Periodic Functions as Regularizers for Quantization of Neural Networks
Authors:
Maxim Naumov,
Utku Diril,
Jongsoo Park,
Benjamin Ray,
Jedrzej Jablonski,
Andrew Tulloch
Abstract:
Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply th…
▽ More
Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply these functions component-wise and add the sum over the model parameters as a regularizer to the model loss during training. The frequency and amplitude hyper-parameters of these functions can be adjusted during training. The regularization pushes the weights into discrete points that can be encoded as integers. We show that using this technique the resulting quantized models exhibit the same accuracy as the original ones on CIFAR-10 and ImageNet datasets.
△ Less
Submitted 24 November, 2018;
originally announced November 2018.
-
High performance ultra-low-precision convolutions on mobile devices
Authors:
Andrew Tulloch,
Yangqing Jia
Abstract:
Many applications of mobile deep learning, especially real-time computer vision workloads, are constrained by computation power. This is particularly true for workloads running on older consumer phones, where a typical device might be powered by a single- or dual-core ARMv7 CPU. We provide an open-source implementation and a comprehensive analysis of (to our knowledge) the state of the art ultra-l…
▽ More
Many applications of mobile deep learning, especially real-time computer vision workloads, are constrained by computation power. This is particularly true for workloads running on older consumer phones, where a typical device might be powered by a single- or dual-core ARMv7 CPU. We provide an open-source implementation and a comprehensive analysis of (to our knowledge) the state of the art ultra-low-precision (<4 bit precision) implementation of the core primitives required for modern deep learning workloads on ARMv7 devices, and demonstrate speedups of 4x-20x over our additional state-of-the-art float32 and int8 baselines.
△ Less
Submitted 6 December, 2017;
originally announced December 2017.
-
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Authors:
Priya Goyal,
Piotr Dollár,
Ross Girshick,
Pieter Noordhuis,
Lukasz Wesolowski,
Aapo Kyrola,
Andrew Tulloch,
Yangqing Jia,
Kaiming He
Abstract:
Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large,…
▽ More
Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ~90% scaling efficiency when moving from 8 to 256 GPUs. Our findings enable training visual recognition models on internet-scale data with high efficiency.
△ Less
Submitted 30 April, 2018; v1 submitted 8 June, 2017;
originally announced June 2017.
-
Accurate principal component analysis via a few iterations of alternating least squares
Authors:
Arthur Szlam,
Andrew Tulloch,
Mark Tygert
Abstract:
A few iterations of alternating least squares with a random starting point provably suffice to produce nearly optimal spectral- and Frobenius-norm accuracies of low-rank approximations to a matrix; iterating to convergence is unnecessary. Thus, software implementing alternating least squares can be retrofitted via appropriate setting of parameters to calculate nearly optimally accurate low-rank ap…
▽ More
A few iterations of alternating least squares with a random starting point provably suffice to produce nearly optimal spectral- and Frobenius-norm accuracies of low-rank approximations to a matrix; iterating to convergence is unnecessary. Thus, software implementing alternating least squares can be retrofitted via appropriate setting of parameters to calculate nearly optimally accurate low-rank approximations highly efficiently, with no need for convergence.
△ Less
Submitted 5 March, 2016;
originally announced March 2016.
