Towards a Serverless Benchmark
Excerpt reproduced with permission from SPEC RG Newsletter Issue 7
Following a durable trend of miniaturization and commodification of software services, there is an emergence of an architecture (model) for cloud-based software that focuses on executing arbitrary functions without much server- and resource-management burden put on the cloud developer or customer. This model, which is now commonly associated with emerging terms such as serverless and Function-as-a-Service (FaaS), addresses needs for which PaaS and SaaS leave many aspects unanswered or insufficiently addressed.
There are several reasons for the rapid emergence of serverless computing. The cloud user is unburdened of the need to manage resources, designing autoscaling policies, and other operational logic. This allows them to focus on their business logic; decreasing the costs of development, removing the need for extensive distributed systems expertise, and improving the time-to-market for applications. On the other hand, the cloud provider has more control (controlling the entire lifecycle) and insight into the users workload. This provides the provider with options to improve scheduling and resource management, minimizing the resource costs of workloads.
In May 2017, the SPEC RG Cloud formed a new activity, lead by Erwin van Eyk and Alexandru Iosup, to investigate this emerging paradigm. The activity initially focused on answering two primary questions: (1) What challenges and issues exist in the current state of serverless computing? (2) What role can the SPEC RG Cloud have in answering (some of) these challenges? Following an initial period of pitching the activity, an initial team was formed, lead by Erwin van Eyk and Alexandru Iosup. After these initial nine months, the inter-disciplinary team has grown to span both industry and academia, with active participants form the TU Delft, VU Amsterdam, University of Wurzburg, ESPOL (Equador), IBM, and SAP.
To answer these questions, we first surveyed the landscape in serverless computing, formulating a consistent terminology (e.g., what is serverless? What is Functionas-a-Service?), learning about the current state-of-the-art in both industry and academia, and determining the current perspectives and challenges [1]. With this clear view of the serverless field, we found that the SPEC RG CLOUD should focus primarily on performance evaluation, creating a much-needed benchmark of serverless platforms. To, potentially, other related performance (engineering) challenges, for which we formed a separate, more specific vision [2].
Our main goal for the coming year is to design and release an initial version of the ’serverless benchmark’; a benchmark of the numerous open-source and closed-source FaaS platforms. As we are progressing towards a serverless benchmark, we are currently working on a reference architecture for FaaS platforms to help identify common components that can be compared across platforms and providers and provide a terminology for the internals of these platforms. To this end, we invite the SPEC Newsletter audience to join this effort; your input, insight, workloads, and knowledge of closed-source implementations could help us shape this field.
[1] E. van Eyk, A. Iosup, S. Seif, and M. Thommes. The SPEC Cloud Group’s Research Vision on FaaS and Serverless Architectures, International Workshop on Serverless Computing, 1–4. ACM, 2017.
[2] E. van Eyk, A. Iosup, C. Abad, S. Eismann, J. Grohmann. A SPEC RG Cloud Group’s Vision on the Performance Challenges of FaaS Cloud Architectures, ACM/SPEC ICPE 2018. ACM.