research-article

DiLOS: adding performance to paging-based memory disaggregation

Authors:

Youngjin KwonAuthors Info & Claims

APSys '21: Proceedings of the 12th ACM SIGOPS Asia-Pacific Workshop on Systems

Pages 70 - 78

https://doi.org/10.1145/3476886.3477507

Published: 24 August 2021 Publication History

Abstract

Memory disaggregation places computing and memory in physically separate nodes and achieves improved memory utilization in datacenters. Kernel-based approaches for memory disaggregation offer transparent virtual memory by using paging schemes but suffer from expensive page fault handling. As an alternative, library-based approaches incorporate application semantics to memory disaggregation and can even eliminate page fault handling on its data path. However, its lack of compatibility harms generality and obstruct wide adoption.

This paper revisits the paging-based approaches and challenges their performance. We posit that the page fault overhead is not a fundamental limitation. We propose DiLOS, a new memory disaggregating unikernel, that delivers both performance and generality. The key insight of DiLOS to overcome performance drawbacks while maintaining generality lies in the design of a fast, lightweight page fault handler on top of the unikernel's simple execution model. Since each unikernel serves a single application, it also opens room for extra optimization via app-aware prefetching. DiLOS outperforms a recent library-based system (AIFM) by 1.52× and 1.31× when the cache size is 12.5% and 100% of the total working set, respectively. Compared to the state-of-the-art paging-based system (Fastswap), DiLOS with a general-purpose prefetcher achieves up to 2.2× higher performance in real-world workload. An app-aware prefetcher further improves the throughput of Redis in-memory database up to 27%.

References

[1]

Mellanox Accelio. nbdX. https://github.com/accelio/NBDX.

[2]

Emmanuel Amaro, Christopher Branner-Augmon, Zhihong Luo, Amy Ousterhout, Marcos K. Aguilera, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. Can Far Memory Improve Job Throughput? In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys ’20, New York, NY, USA, 2020. Association for Computing Machinery.

Digital Library

[3]

Fran. J. Ballesteros. Clive. https://lsub.org/clive/.

[4]

Mike Battersby and Michael Kerrisk. sigaction. http://man7.org/linux/man-pages/man2/sigaction.2.html.

[5]

Andrew Baumann, Marcus Peinado, and Galen Hunt. Shielding applications from an untrusted cloud with Haven. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 267–283, Broomfield, CO, October 2014. USENIX Association.

[6]

Scott Beamer, Krste Asanović, and David Patterson. The GAP Benchmark Suite, 2017.

[7]

Adam Belay, Andrea Bittau, Ali Mashtizadeh, David Terei, David Mazières, and Christos Kozyrakis. Dune: Safe User-level Access to Privileged CPU Features. In 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12), pages 335–348, Hollywood, CA, October 2012. USENIX Association.

[8]

A. Bratterud, A. Walla, H. Haugerud, P. E. Engelstad, and K. Begnum. IncludeOS: A Minimal, Resource Efficient Unikernel for Cloud Services. In 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom), pages 250–257, 2015.

[9]

Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, page 143–154, New York, NY, USA, 2010. Association for Computing Machinery.

[10]

IBM Corporation. and Michael Kerrisk. userfaultfd. http://man7.org/linux/man-pages/man2/userfaultfd.2.html.

[11]

Galois, Inc. The Haskell Lightweight Virtual Machine. https://github.com/GaloisInc/HaLVM.

[12]

Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin. Efficient Memory Disaggregation with Infiniswap. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 649–667, Boston, MA, March 2017. USENIX Association.

[13]

Jing Guo, Zihao Chang, Sa Wang, Haiyang Ding, Yihui Feng, Liang Mao, and Yungang Bao. Who Limits the Resource Efficiency of My Datacenter: An Analysis of Alibaba Datacenter Traces. In Proceedings of the International Symposium on Quality of Service, IWQoS ’19, New York, NY, USA, 2019. Association for Computing Machinery.

[14]

Ying Huang. mm, swap: VMA based swap readahead. https://lwn.net/Articles/716296/.

[15]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. Design Guidelines for High Performance RDMA Systems. In 2016 USENIX Annual Technical Conference (USENIX ATC 16), pages 437–450, Denver, CO, June 2016. USENIX Association.

[16]

Kartik Kannapur. NYC Taxi Trips - Exploratory Data Analysis. https://www.kaggle.com/kartikkannapur/nyc-taxi-trips-exploratory-data-analysis/notebook.

[17]

Avi Kivity, Dor Laor, Glauber Costa, Pekka Enberg, Nadav HarEl, Don Marti, and Vlad Zolotarov. OSvOptimizing the Operating System for Virtual Machines. In 2014 USENIX Annual Technical Conference (USENIX ATC 14), pages 61–72, Philadelphia, PA, June 2014. USENIX Association.

[18]

Waldemar Kozaczuk. OSv Linux ABI Compatibility. https://github.com/cloudius-systems/osv/wiki/OSv-Linux-ABI-Compatibility.

[19]

Simon Kuenzer, Vlad-Andrei Bădoiu, Hugo Lefeuvre, Sharan Santhanam, Alexander Jung, Gaulthier Gain, Cyril Soldani, Costin Lupu, Ştefan Teodorescu, Costi Răducanu, Cristian Banu, Laurent Mathy, Răzvan Deaconescu, Costin Raiciu, and Felipe Huici. Unikraft: Fast, Specialized Unikernels the Easy Way. In Proceedings of the Sixteenth European Conference on Computer Systems, EuroSys '21, page 376–394, New York, NY, USA, 2021. Association for Computing Machinery.

Digital Library

[20]

Simon Kuenzer, Anton Ivanov, Filipe Manco, Jose Mendes, Yuri Volchkov, Florian Schmidt, Kenichi Yasukata, Michio Honda, and Felipe Huici. Unikernels Everywhere: The Case for Elastic CDNs. In Proceedings of the 13th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE ’17, page 15–29, New York, NY, USA, 2017. Association for Computing Machinery.

[21]

Hsuan-Chi Kuo, Dan Williams, Ricardo Koller, and Sibin Mohan. A Linux in Unikernel Clothing. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys '20, New York, NY, USA, 2020. Association for Computing Machinery.

[22]

Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. What is Twitter, a Social Network or a News Media? In Proceedings of the 19th International Conference on World Wide Web, WWW '10, page 591–600, New York, NY, USA, 2010. Association for Computing Machinery.

Digital Library

[23]

Redis Labs. How fast is Redis? https://redis.io/topics/benchmarks.

[24]

Redis Labs. LRANGE - Redis. https://redis.io/commands/lrange.

[25]

Andres Lagar-Cavilla, Junwhan Ahn, Suleiman Souhlal, Neha Agarwal, Radoslaw Burny, Shakeel Butt, Jichuan Chang, Ashwin Chaugule, Nan Deng, Junaid Shahid, and et al. Software-Defined Far Memory in Warehouse-Scale Computers. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’19, page 317–330, New York, NY, USA, 2019. Association for Computing Machinery.

Digital Library

[26]

Stefan Lankes, Simon Pickartz, and Jens Breitbart. HermitCore: A Unikernel for Extreme Scale Computing. In Proceedings of the 6th International Workshop on Runtime and Operating Systems for Supercomputers, ROSS ’16, New York, NY, USA, 2016. Association for Computing Machinery.

[27]

Jiuxing Liu, Wei Huang, Bülent Abali, and Dhabaleswar K. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In Atul Adya and Erich M. Nahum, editors, Proceedings of the 2006 USENIX Annual Technical Conference, Boston, MA, USA, May 30 - June 3, 2006, pages 29–42. USENIX, 2006.

[28]

Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft. Unikernels: Library Operating Systems for the Cloud. SIGARCH Comput. Archit. News, 41(1):461–472, March 2013.

[29]

Anil Madhavapeddy and David J. Scott. Unikernels: Rise of the Virtual Library Operating System. Queue, 11(11):30–44, December 2013.

[30]

Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. My VM is Lighter (and Safer) than Your Container. In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, page 218–233, New York, NY, USA, 2017. Association for Computing Machinery.

Digital Library

[31]

Joao Martins, Mohamed Ahmed, Costin Raiciu, Vladimir Olteanu, Michio Honda, Roberto Bifulco, and Felipe Huici. ClickOS and the Art of Network Function Virtualization. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 459–473, Seattle, WA, April 2014. USENIX Association.

[32]

Hasan Al Maruf and Mosharaf Chowdhury. Effectively Prefetching Remote Memory with Leap. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 843–857. USENIX Association, July 2020.

[33]

Hossein Moein. DataFrame. https://github.com/hosseinmoein/DataFrame.

[34]

Pierre Olivier, Daniel Chiba, Stefan Lankes, Changwoo Min, and Binoy Ravindran. A Binary-Compatible Unikernel. In Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE 2019, page 59–73, New York, NY, USA, 2019. Association for Computing Machinery.

[35]

Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. Shenango: Achieving high cpu efficiency for latency-sensitive datacenter workloads. In Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation, NSDI'19, page 361–377, USA, 2019. USENIX Association.

[36]

Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Ionnis Koltsidas, and Thomas R. Gross. A Hybrid I/O Virtualization Framework for RDMA-Capable Network Interfaces. SIGPLAN Not., 50(7):17–30, March 2015.

[37]

QEMU. Paravirtualized RDMA Device (PVRDMA). https://github.com/qemu/qemu/blob/master/docs/pvrdma.txt.

[38]

Ali Raza, Parul Sohal, James Cadden, Jonathan Appavoo, Ulrich Drepper, Richard Jones, Orran Krieger, Renato Mancuso, and Larry Woodman. Unikernels: The Next Stage of Linux’s Dominance. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS ’19, page 7–13, New York, NY, USA, 2019. Association for Computing Machinery.

Digital Library

[39]

Zhenyuan Ruan. AIFM. https://github.com/aifm-sys/aifm.

[40]

Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay. AIFM: High-Performance, Application-Integrated Far Memory. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 315–332. USENIX Association, November 2020.

[41]

Salvatore Sanfilippo. Redis. https://redis.io.

[42]

ScaleGrid. Top Redis Use Cases by Core Data Structure Types. https://scalegrid.io/blog/top-redis-use-cases-by-core-data-structure-types/.

[43]

Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 69–87, Carlsbad, CA, October 2018. USENIX Association.

[44]

Matt Stancliff. Redis Quicklist - From a More Civilized Age. https://matt.sh/redis-quicklist.

[45]

Shelby Thomas, Geoffrey M. Voelker, and George Porter. CacheCloud: Towards Speed-of-light Datacenter Communication. In 10th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 18), Boston, MA, July 2018. USENIX Association.

[46]

Muhammad Tirmazi, Adam Barker, Nan Deng, Md E. Haque, Zhijing Gene Qin, Steven Hand, Mor Harchol-Balter, and John Wilkes. Borg: The next Generation. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys '20, New York, NY, USA, 2020. Association for Computing Machinery.

Digital Library

[47]

Chenxi Wang, Haoran Ma, Shi Liu, Yuanqi Li, Zhenyuan Ruan, Khanh Nguyen, Michael D. Bond, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu. Semeru: A Memory-Disaggregated Managed Runtime. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 261–280. USENIX Association, November 2020.

[48]

Dan Williams and Ricardo Koller. Unikernel Monitors: Extending Minimalism Outside of the Box. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16), Denver, CO, June 2016. USENIX Association.

[49]

Jian Yang, Joseph Izraelevitz, and Steven Swanson. FileMR: Rethinking RDMA Networking for Scalable Persistent Memory. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 111–125, Santa Clara, CA, February 2020. USENIX Association.

Cited By

Wang CShan YZuo PCui H(2023)Reinvent Cloud Software Stacks for Resource DisaggregationJournal of Computer Science and Technology10.1007/s11390-023-3272-038:5(949-969)Online publication date: 30-Sep-2023
https://dl.acm.org/doi/10.1007/s11390-023-3272-0
Wang QLu YShu JIves ZBonifati AEl Abbadi A(2022)Sherman: A Write-Optimized Distributed B+Tree Index on Disaggregated MemoryProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3517824(1033-1048)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3517824

Index Terms

DiLOS: adding performance to paging-based memory disaggregation
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing

Recommendations

DiLOS: Do Not Trade Compatibility for Performance in Memory Disaggregation
EuroSys '23: Proceedings of the Eighteenth European Conference on Computer Systems

Memory disaggregation has replaced the landscape of dat-acenters by physically separating compute and memory nodes, achieving improved utilization. As early efforts, kernel paging-based approaches offer transparent virtual memory abstraction for remote ...
UCat: heterogeneous memory management for unikernels
Abstract
Unikernels provide an efficient and lightweight way to deploy cloud computing services in application-specialized and single-address-space virtual machines (VMs). They can efficiently deploy hundreds of unikernel-based VMs in a single physical ...
Rcmp: Reconstructing RDMA-Based Memory Disaggregation via CXL
Memory disaggregation is a promising architecture for modern datacenters that separates compute and memory resources into independent pools connected by ultra-fast networks, which can improve memory utilization, reduce cost, and enable elastic scaling of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

APSys '21: Proceedings of the 12th ACM SIGOPS Asia-Pacific Workshop on Systems

August 2021

159 pages

ISBN:9781450386982

DOI:10.1145/3476886

Program Chairs:
Haryadi Gunawi
University of Chicago
,
Xiaosong Ma
Qatar Computing Research Inst., HBKU

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Research Foundation of Korea

Conference

APSys '21

Sponsor:

SIGOPS

APSys '21: 12th ACM SIGOPS Asia-Pacific Workshop on Systems

August 24 - 25, 2021

Hong Kong, China

Acceptance Rates

APSys '21 Paper Acceptance Rate 19 of 43 submissions, 44%;

Overall Acceptance Rate 169 of 430 submissions, 39%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
556
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)7

Reflects downloads up to 23 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang CShan YZuo PCui H(2023)Reinvent Cloud Software Stacks for Resource DisaggregationJournal of Computer Science and Technology10.1007/s11390-023-3272-038:5(949-969)Online publication date: 30-Sep-2023
https://dl.acm.org/doi/10.1007/s11390-023-3272-0
Wang QLu YShu JIves ZBonifati AEl Abbadi A(2022)Sherman: A Write-Optimized Distributed B+Tree Index on Disaggregated MemoryProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3517824(1033-1048)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3517824

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents