Serverless workflows with durable functions and netherite

S Burckhardt, C Gillum, D Justo, K Kallas…�- arXiv preprint arXiv�…, 2021 - arxiv.org
S Burckhardt, C Gillum, D Justo, K Kallas, C McMahon, CS Meiklejohn
arXiv preprint arXiv:2103.00033, 2021arxiv.org
Serverless is an increasingly popular choice for service architects because it can provide
elasticity and load-based billing with minimal developer effort. A common and important use
case is to compose serverless functions and cloud storage into reliable workflows. However,
existing solutions for authoring workflows provide a rudimentary experience compared to
writing standard code in a modern programming language. Furthermore, executing
workflows reliably in an elastic serverless environment poses significant performance�…
Serverless is an increasingly popular choice for service architects because it can provide elasticity and load-based billing with minimal developer effort. A common and important use case is to compose serverless functions and cloud storage into reliable workflows. However, existing solutions for authoring workflows provide a rudimentary experience compared to writing standard code in a modern programming language. Furthermore, executing workflows reliably in an elastic serverless environment poses significant performance challenges. To address these, we propose Durable Functions, a programming model for serverless workflows, and Netherite, a distributed execution engine to execute them efficiently. Workflows in Durable Functions are expressed as task-parallel code in a host language of choice. Internally, the workflows are translated to fine-grained stateful communicating processes, which are load-balanced over an elastic cluster. The main challenge is to minimize the cost of reliably persisting progress to storage while supporting elastic scale. Netherite solves this by introducing partitioning, recovery logs, asynchronous snapshots, and speculative communication. Our results show that Durable Functions simplifies the expression of complex workflows, and that Netherite achieves lower latency and higher throughput than the prevailing approaches for serverless workflows in Azure and AWS, by orders of magnitude in some cases.
arxiv.org
Showing the best result for this search. See all results